Image processing device

Abstract

To easily mask only a region corresponding to a user designated item in image data which are obtained by reading a document.SOLUTION: An image processing device comprises an image reading section and a control section. The control section performs first region detection processing for detecting a first region which is predicted to individually include either a region of an item name or a region of an item value from image data, second region detection processing for detecting a second region which is predicted to include both regions of an item name and an item value corresponding to the same item from the image data, processing for detecting the second region including the first region of a processing target, and processing for associating a plurality of first regions included in the same second region with each other. In the first region detection processing and the second region detection processing, a printed character region where characters in the region are printed character is detected and the control section further performs processing for detecting a handwriting region where characters being present in the region are hand-written characters.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device. [Background technology]

[0002] Conventionally, there has been known an image processing device that reads an identification card as a document and masks an area of ​​personal information in the image data obtained by the reading. Such an image processing device is disclosed, for example, in Patent Document 1. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-197682 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, personal information written on an ID card is classified into multiple categories, such as name and address. With such an ID card, it may be desirable to mask only the areas of the image data obtained by scanning that correspond to specific categories. Note that the areas that need to be masked vary from user to user.

[0005] In order to detect an area corresponding to a specified item from the image data obtained by scanning, a method using a language model such as BERT (Bidirectional Encoder Representations from Transformers) may be used. However, this method requires a large model size and a huge amount of calculation. For this reason, for example, image data of an ID card is transferred to the cloud and processed on the cloud.

[0006] If image data of an ID card is transferred to the cloud, there is a risk of personal information being leaked, so transferring image data of an ID card to the cloud is not recommended.

[0007] The present invention has been made to solve the above-mentioned problems, and aims to provide an image processing device that can easily mask only the areas of image data obtained by reading a document that correspond to items specified by the user. [Means for solving the problem]

[0008] To achieve the above object, an image processing device according to one aspect of the present invention includes an image reading unit that reads a document containing multiple sets of item names and item values ​​for personal information items, and a control unit that recognizes target items and performs a masking process on image data obtained by reading the document with the image reading unit to generate masked data in which at least a portion of the target item area present in the image data is masked. The control unit performs a first region detection process that detects, from the image data, first regions that are predicted to individually include either an item name area or an item value area; a second region detection process that detects, from the image data, second regions that are predicted to include both an item name area and an item value area corresponding to the same item; an inclusion region detection process that is performed for each of the multiple first regions and detects second regions that include the target first region by detecting second regions that have image portions whose image similarity with the target first region is equal to or greater than a threshold; and a region linking process that links multiple first regions that are included in the same second region to each other. The first region detection process and the second region detection process detect typed regions where characters within the region are typed. The control unit further performs handwritten area detection processing to detect, from the image data, a handwritten area in which characters present within the area are handwritten characters. [Effects of the Invention]

[0009] In the configuration of the present invention, it is possible to easily mask only the areas of image data obtained by scanning an original that correspond to items designated by the user. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram of a multifunction peripheral according to an embodiment. [Figure 2] FIG. 1 is a block diagram of a multifunction peripheral according to an embodiment. [Figure 3] 1 is a diagram illustrating an example of an identification card that can be read by a multifunction peripheral according to an embodiment; [Figure 4] 10 is a flowchart illustrating the flow of a masking job executed by the multifunction peripheral of an embodiment. [Figure 5] 4 is a diagram showing a first region in image data obtained by reading the identification card shown in FIG. 3. FIG. [Figure 6] 4 is a diagram showing a second area in image data obtained by reading the identification card shown in FIG. 3. FIG. [Figure 7] 4 is a diagram showing masked data generated by a masking job for the identification card shown in FIG. 3. FIG. [Figure 8] FIG. 10 is a conceptual diagram of dictionary data used in a masking job by the multifunction peripheral of an embodiment. [Figure 9] 10 is a flowchart illustrating a flow of an area determination process performed in the multifunction peripheral according to an embodiment. [Figure 10] 10 is a flowchart showing the flow of a first abnormality countermeasure process performed in the multifunction peripheral of an embodiment. [Figure 11] 10 is a flowchart showing the flow of second and third abnormality countermeasure processes performed in the multifunction peripheral of an embodiment. [Figure 12] 10 is a flowchart showing the flow of a second abnormality countermeasure process performed in the multifunction peripheral of an embodiment. [Figure 13] 10 is a flowchart illustrating a flow of a third abnormality countermeasure process performed in the multifunction peripheral of an embodiment. [Figure 14]10 is a flowchart illustrating a process flow performed when a masking job setting is accepted by the multifunction peripheral of an embodiment. [Figure 15] FIG. 10 illustrates a reception screen displayed by the multifunction peripheral according to an embodiment. [Figure 16] FIG. 10 illustrates a registration screen displayed by the multifunction peripheral according to an embodiment. [Figure 17] 1 is a diagram illustrating an example of an identification card (including a handwritten signature) that can be read by a multifunction peripheral according to an embodiment. FIG. [Figure 18] 10 is a flowchart showing the flow of a masking job (a masking job in which handwritten area detection processing is additionally performed) executed by the multifunction peripheral of an embodiment. [Figure 19] 18 is a diagram showing a first region and a handwritten region in image data obtained by reading the identification card shown in FIG. 17. FIG. [Figure 20] 18 is a diagram showing a second region and a handwritten region in image data obtained by reading the identification card shown in FIG. 17. FIG. [Figure 21] 18 is a diagram showing masked data generated by a masking job for the identification card shown in FIG. 17. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0011] <Configuration of the multifunction device> An image processing apparatus according to an embodiment of the present invention will be described below with reference to FIGS. 1 to 21, taking as an example a multifunction peripheral 100 having multiple functions such as a scanning function, a printing function, and a transmission function.

[0012] As shown in FIG. 1, the multifunction device 100 (corresponding to an "image processing device") includes a printing unit 1. The printing unit 1 constitutes the main body of the multifunction device 100. The printing unit 1 prints an image on a sheet S. The printing method of the printing unit 1 is an electrophotographic method. However, this is not limited to this. The printing method of the printing unit 1 may also be an inkjet method.

[0013] The printing unit 1 forms an image based on image data input to the multifunction device 100. The printing unit 1 also transports a sheet S along a sheet transport path. The printing unit 1 prints an image on the sheet S while it is being transported. In FIG. 1, the sheet transport path is indicated by a dashed line.

[0014] The printing unit 1 includes a paper feed roller 11. The paper feed roller 11 comes into contact with a sheet S stored in a sheet cassette CA and rotates in this state, thereby feeding the sheet S from the sheet cassette CA to a sheet transport path.

[0015] The printing unit 1 includes an image forming unit 12. The image forming unit 12 includes a photosensitive drum 12a and a transfer roller 12b. The photosensitive drum 12a carries a toner image on its peripheral surface. The transfer roller 12b is in pressure contact with the photosensitive drum 12a, forming a transfer nip between the photosensitive drum 12a and the transfer roller 12b. The transfer roller 12b rotates together with the photosensitive drum 12a. The image forming unit 12 transfers the toner image onto the sheet S while transporting the sheet S that has entered the transfer nip.

[0016] Although not shown, the image forming unit 12 further includes a charging device, an exposure device, and a developing device. The charging device charges the circumferential surface of the photosensitive drum 12a. The exposure device forms an electrostatic latent image on the circumferential surface of the photosensitive drum 12a. The developing device develops the electrostatic latent image on the circumferential surface of the photosensitive drum 12a into a toner image.

[0017] The printing unit 1 includes a fixing unit 13. The fixing unit 13 includes a heating roller 13a and a pressure roller 13b. The heating roller 13a has a built-in heater (not shown). The pressure roller 13b is pressed against the heating roller 13a, forming a fixing nip between the heating roller 13a and the pressure roller 13b. The pressure roller 13b rotates together with the heating roller 13a. The fixing unit 13 fixes the toner image transferred onto the sheet S onto the sheet S while transporting the sheet S that has entered the fixing nip. The sheet S that has exited the fixing nip is discharged onto an ejection tray ET.

[0018] The multifunction device 100 also includes an image reading unit 2. The image reading unit 2 is disposed on top of the main body of the multifunction device 100. In a job involving reading an original D, the original D is set in the image reading unit 2. The image reading unit 2 reads the original D set in the image reading unit 2 and generates image data of the read original D.

[0019] The image reading unit 2 includes contact glasses G1 and G2. The contact glasses G1 and G2 are installed in a housing RH of the image reading unit 2. The housing RH has an opening on the top surface. The contact glasses G1 and G2 are attached to the opening on the top surface of the housing RH.

[0020] The image reading unit 2 includes a document transport device DP. The document transport device DP is attached to the housing RH. When viewed from the front of the multifunction device 100, the document transport device DP pivots around its rear portion as if swinging its front portion up and down. The document transport device DP opens and closes relative to the top surface of the housing RH.

[0021] The document transport device DP has a set tray ST on which a document D is set. The document transport device DP transports the document D set on the set tray ST onto the contact glass G1.

[0022] In the conveying reading mode, the user sets the document D on the set tray ST. Then, the document D is automatically conveyed onto the contact glass G1 by the document conveying device DP (in other words, the document D passing over the contact glass G1) and is read. On the other hand, in the placed reading mode, the user sets the document D on the contact glass G2, and the document D on the contact glass G2 is read.

[0023] The image reading unit 2 includes a light source 21, an image sensor 22, a mirror 23, and a lens 24. The light source 21, the image sensor 22, the mirror 23, and the lens 24 are installed inside the housing RH. The image reading unit 2 performs a scanning operation in which light is irradiated from the light source 21 toward the contact glass G1 or G2 and photoelectrically converted by the image sensor 22.

[0024] The light source 21 has a plurality of LED elements. The plurality of LED elements are arranged in a line in the main scanning direction (a direction perpendicular to the paper surface of FIG. 1). The image sensor 22 has a plurality of photoelectric conversion elements arranged in the main scanning direction. The mirror 23 reflects light toward the lens 24. The lens 24 collects the light reflected by the mirror 23 and guides it to the image sensor 22.

[0025] The light source 21 and the mirror 23 are mounted on a carriage 25 that is movable in a sub-scanning direction (left and right direction in FIG. 1) that is perpendicular to the main scanning direction. As the carriage 25 moves in the sub-scanning direction, the reading line of the image reading unit 2 moves in the sub-scanning direction.

[0026] As shown in FIG. 2, the multifunction device 100 also includes an operation display unit 3. The operation display unit 3 is an operation panel having a touch screen. The operation display unit 3 displays software buttons, messages, and the like on the touch screen. The operation display unit 3 is also provided with a plurality of hardware buttons. The operation display unit 3 accepts operations from the user. The user can configure the multifunction device 100 via the operation display unit 3 with respect to various jobs, such as a masking job, which will be described later.

[0027] The multifunction peripheral 100 includes a control unit 10. The control unit 10 includes a CPU, an ASIC, a memory, and the like. The control unit 10 also includes an image processing circuit. The control unit 10 performs various image processing operations on image data. The control unit 10 also controls the printing of an image onto a sheet S by the printing unit 1, and the reading of an original D by the image reading unit 2.

[0028] The control unit 10 also controls the operation and display unit 3. Specifically, the control unit 10 controls the display operation of the touch screen. The control unit 10 detects operations on software buttons and hardware buttons. The control unit 10 sets jobs based on operations received by the operation and display unit 3 from the user.

[0029] The multifunction peripheral 100 includes a storage unit 101. The storage unit 101 is a non-volatile storage device. An HDD, an SSD, or the like may be used as the storage unit 101. The storage unit 101 is connected to the control unit 10. The control unit 10 writes information to the storage unit 101 and reads information from the storage unit 101.

[0030] The storage unit 101 stores predetermined information in advance. For example, the storage unit 101 stores a character recognition program in advance. The control unit 10 performs character recognition processing such as OCR (Optical Character Recognition) processing based on the character recognition program. The control unit 10 subjects image data obtained by reading the document D with the image reading unit 2 to the character recognition processing.

[0031] The multifunction peripheral 100 includes a communication unit 102. The communication unit 102 is an interface for communicatively connecting an external device to the multifunction peripheral 100. The communication unit 102 includes a communication circuit, a communication memory, a communication connector, and the like. The communication unit 102 is connected to the control unit 10. The control unit 10 uses the communication unit 102 to send and receive data to and from the external device.

[0032] The communication unit 102 is communicably connected to an external device via a network NT such as a LAN or the Internet. Although not shown, the communication unit 102 may be directly connected to an external device via a communication cable. An external device connected to the communication unit 102 is, for example, a personal computer 1000 (hereinafter referred to as PC 1000) used by a user of the multifunction peripheral 100. An external device other than the PC 1000 may be communicably connected to the multifunction peripheral 100. By connecting the PC 1000 to the multifunction peripheral 100, image data of the original D obtained by reading the original D with the image reading unit 2 can be transmitted to the PC 1000. This allows the image data of the original D to be saved in the PC 1000.

[0033] <Masking job overview> When entering into a contract for goods or services, the person wishing to enter into the contract may be required to register personal information. This is just one example, and there are various other cases where registration of personal information is required. Examples of personal information include name, address, and date of birth.

[0034] The user of the multifunction device 100 registers personal information. When registering personal information, the user uses the scanning function of the multifunction device 100 to scan their identification card. There are various types of identification cards that can be used, and these vary depending on the country and field. Examples of identification cards include driver's licenses, insurance cards, student ID cards, and passports.

[0035] An example of an ID card is shown schematically in Figure 3. Personal information is classified into multiple items and written on the ID card. In other words, the ID card lists multiple sets of item names and item values ​​for items related to personal information. In the example shown in Figure 3, the character string "Name" corresponds to the item name, and the character string "aaaa" corresponds to the item value of the item corresponding to the item name "Name." The character string "Address" corresponds to the item name, and the character string "bbbb" corresponds to the item value of the item corresponding to the item name "Address." The character string "Date of Birth" corresponds to the item name, and the character string "cccc" corresponds to the item value of the item corresponding to the item name "Date of Birth."

[0036] When the ID card shown in Figure 3 is read, the image data obtained by reading the ID card will contain a character region containing the string "Name" indicating the item name and a character region containing the string "aaaa" indicating the item value. Also, a character region containing the string "Address" indicating the item name and a character region containing the string "bbbb" indicating the item value will appear. Also, a character region containing the string "Date of Birth" indicating the item name and a character region containing the string "cccc" indicating the item value will appear.

[0037] In the following explanation, when it is necessary to distinguish between multiple character areas, the character area for the item name "Name" is assigned the code C11, and the character area for the item value "aaaa" is assigned the code C12. The character area for the item name "Address" is assigned the code C21, and the character area for the item value "bbbb" is assigned the code C22. The character area for the item name "Date of Birth" is assigned the code C31, and the character area for the item value "cccc" is assigned the code C32.

[0038] After the multifunction device 100 reads the ID, an image based on the image data of the ID obtained by the reading (i.e., the personal information written on the ID) is printed on a sheet S. Then, the sheet S on which the personal information is printed is stored. In this manner, the personal information is registered. Note that the image data (i.e., electronic data) of the ID itself may also be stored.

[0039] Here, in some cases, it may be necessary to mask some personal information. For example, suppose an identification card has a field for date of birth, but the date of birth is not required in contracts for goods and services. In this example, it may be necessary to mask personal information related to the date of birth. Also, for example, in some countries, identification cards have a field for religion, but it is stipulated that personal information related to religion should not be recorded. In this example, it may be necessary to mask personal information related to religion.

[0040] For example, it has been common practice to manually black out personal information to conceal it, but manually masking some personal information is a hassle for users.

[0041] For this reason, a masking function is installed in the multifunction device 100. In other words, the multifunction device 100 is capable of executing a job related to the masking function (hereinafter simply referred to as a "masking job").

[0042] By using the masking function, it is possible to obtain masked data, which is image data in which some of the personal information on the ID card has been masked. An image based on the masked data can be printed on sheet S, or the masked data can be sent to PC 1000 and saved there.

[0043] When a masking job is executed, the masking job is configured. In the masking job configuration, personal information to be masked can be arbitrarily specified. The masking job configuration will be explained in detail later.

[0044] After setting the masking job, the user places an identification card as the document D in the image reading unit 2. In this state, the user performs a start operation for the masking job on the operation display unit 3. When the control unit 10 detects that the start operation has been performed on the operation display unit 3, it starts the masking job.

[0045] The flow of a masking job will be described below with reference to the flowchart shown in Fig. 4. The flow shown in Fig. 4 starts when the control unit 10 detects an operation to start a masking job on the operation and display unit 3.

[0046] Before starting a masking job, the user sets up the masking job. In setting up the masking job, target items, which are items corresponding to the item values ​​to be masked, are specified. When executing the masking job, the control unit 10 recognizes the target items specified by the user.

[0047] In step #1, the control unit 10 causes the image reading unit 2 to read an identification card as the document D. The image reading unit 2 reads the identification card and generates image data of the read identification card (i.e., scan data of the identification card). The control unit 10 acquires the image data of the identification card obtained by the image reading unit 2 reading the identification card.

[0048] In step #2, the control unit 10 performs a first region detection process using a first learning model obtained by machine learning. The control unit 10 performs the first region detection process by detecting a first region that is predicted to individually include either an item name region or an item value region from the image data of the identification card.

[0049] The first learning model for the first area detection process is a learning model that has been trained to detect the first area from image data obtained by reading an identification card using the image reading unit 2. The first learning model is a trained model and is stored in advance in the memory unit 101.

[0050] An identification card usually has multiple pairs of item names and item values. Therefore, multiple first areas are detected in the first area detection process. For this reason, if no first area is detected or only one first area is detected as a result of the first area detection process, the masking job may be stopped and a message may be displayed on the operation display unit 3 prompting the user to check the document D set in the image reading unit 2.

[0051] In the ID card shown in Figure 3, the character area C11 of the item name "Name," the character area C12 of the item value "aaaa," the character area C21 of the item name "Address," the character area C22 of the item value "bbbb," the character area C31 of the item name "Date of Birth," and the character area C32 of the item value "cccc" are each individually detected as the first area. The first areas are clearly shown in Figure 5.

[0052] In step #3, the control unit 10 performs a second region detection process using a second learning model obtained by machine learning. The control unit 10 performs the second region detection process by detecting a second region that is predicted to include both the area of ​​the item name and the area of ​​the item value that correspond to the same item from the image data of the identification card.

[0053] The second learning model for the second area detection process is a learning model that has been trained to detect the second area from image data obtained by reading an identification card using the image reading unit 2. The second learning model is a trained model and is stored in advance in the memory unit 101.

[0054] If an identification card has been read, at least one second area will be detected as a result of the second area detection process. If no second area is detected in the second area detection process, it is possible that something other than an identification card has been read. Therefore, if no second area is detected in the second area detection process, the masking job may be stopped, and a message prompting the user to check the document D set in the image reading unit 2 may be displayed on the operation display unit 3.

[0055] In the example shown in Fig. 5, regions C10, C20, and C3 shown in Fig. 6 are each detected as the second region. Specifically, a single region C10 that includes a character region C11 for the item name "Name" and a character region C12 for the item value "aaaa" is detected as the second region. A single region C20 that includes a character region C21 for the item name "Address" and a character region C22 for the item value "bbbb" is detected as the second region. A single region C30 that includes a character region C31 for the item name "Date of Birth" and a character region C32 for the item value "cccc" is detected as the second region.

[0056] In step #4, control unit 10 performs inclusion region detection processing for each of the multiple first regions detected in the first region detection processing. Each of the multiple first regions is subjected to the inclusion region detection processing once. Control unit 10 performs the inclusion region detection processing on one of the first regions as the processing target, and when that inclusion region detection processing is complete, it performs the inclusion region detection processing on another first region that has not yet been processed as the new processing target.

[0057] The control unit 10 performs an inclusion region detection process to detect a second region that encompasses the first region to be processed. The control unit 10 detects a second region that encompasses the first region to be processed by detecting a second region having an image portion whose image similarity with the first region to be processed is equal to or greater than the threshold for the inclusion region detection process. The control unit 10 determines that the first region to be processed is encompassed by a second region having an image portion whose image similarity with the first region to be processed is equal to or greater than the threshold for the inclusion region detection process. Note that character recognition processing has not been performed at this point. In other words, the image similarity refers to the similarity between image data, not the similarity between character strings (i.e., between text data). The control unit 10 performs the inclusion region detection process using, for example, a known pattern matching technique. The threshold for the inclusion region detection process is not particularly limited, but is 70% or greater.

[0058] In the examples shown in FIGS. 5 and 6, the second region C10 has an image portion where the image similarity with the first region C11 (region with the item name "Name") is equal to or greater than a threshold, and also has an image portion where the image similarity with the first region C12 (region with the item value "aaaa") is equal to or greater than a threshold. The second region C20 has an image portion where the image similarity with the first region C21 (region with the item name "Address") is equal to or greater than a threshold, and also has an image portion where the image similarity with the first region C22 (region with the item value "bbbb") is equal to or greater than a threshold. The second region C30 has an image portion where the image similarity with the first region C31 (region with the item name "Date of Birth") is equal to or greater than a threshold, and also has a portion where the image similarity with the first region C32 (region with the item value "cccc") is equal to or greater than a threshold. As a result, the first regions C11 and C12 are determined to be included in the second region C10, the first regions C21 and C22 are determined to be included in the second region C20, and the first regions C31 and C32 are determined to be included in the second region C30.

[0059] In step #5, the control unit 10 performs a region linking process. Specifically, the control unit 10 links a plurality of first regions included in the same second region to each other.

[0060] 5 and 6, first regions C11 and C12 included in second region C10 are linked to each other. First regions C21 and C22 included in second region C20 are linked to each other. First regions C31 and C32 included in second region C30 are linked to each other.

[0061] In step #6, the control unit 10 performs an area determination process. By performing the area determination process, the control unit 10 recognizes which of the multiple first areas linked to each other is an item name area and which is an item value area. In other words, by performing the area determination process, the control unit 10 distinguishes between an item name area and an item value area.

[0062] 5 and 6, in the second area C10, the first area C11 is determined to be the area for item names, and the first area C12 is determined to be the area for item values. In the second area C20, the first area C21 is determined to be the area for item names, and the first area C22 is determined to be the area for item values. In the second area C30, the first area C31 is determined to be the area for item names, and the first area C32 is determined to be the area for item values.

[0063] The control unit 10 recognizes the area of ​​the target item by performing an area determination process. The area of ​​the target item is a second area that includes both a first area corresponding to the item name of the target item and a first area corresponding to the item value of the target item. The area determination process will be described in detail later. Note that the area of ​​the target item may be recognized using other methods.

[0064] In step #7, the control unit 10 performs a masking process to mask at least a portion of the area of ​​the target item present in the image data of the identification card. By performing the masking process, the control unit 10 generates masked data in which at least a portion of the area of ​​the target item present in the image data of the identification card is masked.

[0065] The control unit 10 masks the first area associated with the area of ​​the item name of the target item (i.e., the area of ​​the item value of the target item). Alternatively, the control unit 10 masks both the area of ​​the item name and the area of ​​the item value of the target item. Substantially the entire area of ​​the second area corresponding to the target item may be masked.

[0066] An example of masked data is shown in Figure 7. The masked data shown in Figure 7 is generated when the target item is date of birth. That is, at least the item value "cccc" corresponding to the item name "Date of Birth" is masked. Although not shown, both the item name "Date of Birth" and the item value "cccc" may be masked.

[0067] In step #8, the control unit 10 causes the output unit to perform output processing of the masked data. For example, in the masking job settings, the output method of the masked data can be selected. The output methods include printing and transmission.

[0068] When printing is selected as the output method, the control unit 10 causes the printing unit 1 to print (in other words, output) an image based on the masked data onto a sheet S. In this case, the printing unit 1 corresponds to the "output unit," and the output destination is the sheet S.

[0069] When transmission is selected as the output method, the control unit 10 causes the communication unit 102 to transmit (in other words, output) the masked data to the PC 1000. The masked data may be converted into PDF data and transmitted to the PC 1000. By transmitting the masked data to the PC 1000, the masked data can be saved in the PC 1000. In this case, the communication unit 102 corresponds to the "output unit," and the output destination is the PC 1000.

[0070] In this embodiment, a first region detection process, a second region detection process, an inclusion region detection process, and a region linking process are performed, which allows each region of the image data of the identification card that has an item name and an item value corresponding to the same item to be linked with high accuracy.

[0071] By linking the fields of the item name and the field value corresponding to the same item in the image data of the ID card to each other, it is possible to easily mask at least a portion of the target item. Specifically, it is sufficient to mask the field linked to the field of the item name of the target item. This allows the field value of the target item to be masked. With this configuration, it is possible to easily mask only the fields corresponding to the user-specified items in the image data obtained by scanning the document D (ID card) without transferring the image data of the ID card to a processing device on the cloud.

[0072] If the item value area in the image data of an ID card can be masked, the leakage of personal information can be suppressed. Even if the item name area of ​​a target item is masked, if the item value area of ​​the target item is not masked, personal information will be leaked. Therefore, it is important to accurately perform a process of linking each item name and item value area corresponding to the same item in the image data of an ID card.

[0073] In this embodiment, the first area detection process is performed using the first learning model, and the second area detection process is performed using the second learning model. Here, machine learning requires a huge amount of computation and a huge amount of memory capacity. For this reason, machine learning processes are often performed by a processing device on the cloud. However, this requires transferring image data of the identification card to the processing device on the cloud, which is undesirable because it may result in the leakage of personal information.

[0074] Therefore, in this embodiment, a trained first learning model and a trained second learning model are used. The first learning model and the second learning model are each stored in advance in the storage unit 101. This allows the first area detection process and the second area detection process to be performed within the multifunction device 100 without increasing the memory capacity within the multifunction device 100. In other words, there is no need to transfer image data of the identification card to a processing device on the cloud. As a result, leakage of personal information can be suppressed.

[0075] <Area determination process> The control unit 10 performs an area determination process to determine whether each of a plurality of first areas (i.e., character areas containing character strings) present in the image data of the identification card is an area for an item name or an area for an item value. The area determination process uses dictionary data DD. The dictionary data DD is stored in advance in the storage unit 101 (see FIG. 2).

[0076] A conceptual diagram of the dictionary data DD is shown in Figure 8. The dictionary data DD is data in which multiple different similar item names are predefined for one type of item. The dictionary data DD is created in advance for each of the multiple types of items. The storage unit 101 stores in advance multiple dictionary data DD corresponding to each of the multiple types of items. The multiple dictionary data DD are created in advance by the manufacturer of the multifunction device 100 and pre-stored in the storage unit 101. In Figure 8, items are indicated by A, B... Similar item names corresponding to item A are indicated by a1, a2, a3... Similar item names corresponding to item B are indicated by b1, b2, b3...

[0077] Taking an identification number as an example of an item, one ID card may use the character string "ID" as the item name, another ID card may use the character string "Number" as the item name, and yet another ID card may use the character string "Num" as the item name. Although the character strings listed here are different from one another, they all correspond to the same item. Therefore, in the dictionary data DD corresponding to the identification number as an item, the character string "ID," the character string "Number," and the character string "Num" are predefined as similar item names.

[0078] The flow of the area determination process will be described below with reference to the flowchart shown in Fig. 9. The flow shown in Fig. 9 starts when the area linking process by the control unit 10 is completed.

[0079] In step #11, the control unit 10 performs character recognition processing on the image data of the ID card obtained by the image reading unit 2 reading the ID card (i.e., the processing of step #1 shown in FIG. 4). The control unit 10 performs OCR processing on the image data of the ID card. As a result, the control unit 10 extracts character strings from each of the multiple first areas present in the image data of the ID card. The control unit 10 recognizes the character strings in each of the multiple first areas. Note that the first areas contain character strings indicating item names or character strings indicating item values.

[0080] In step #12, the control unit 10 performs preprocessing on the multiple character strings extracted by the character recognition process. The type of preprocessing is not particularly limited. For example, the control unit 10 performs preprocessing by unifying uppercase and lowercase letters in the character string. This process makes the character string indistinguishable from uppercase and lowercase letters. Furthermore, for example, the control unit 10 performs preprocessing by removing spaces and predetermined symbols.

[0081] After the processing of step #12, the control unit 10 performs an item name area determination process. When performing the item name area determination process, the control unit 10 sets one of the multiple first areas detected in the first area detection process as a processing target.

[0082] Then, the control unit 10 performs a similarity calculation process (the process of step #13) as one process of the item name region determination process. In step #13, the control unit 10 calculates the similarity between each character string of all similar item names defined in the multiple dictionary data DD and the character string of the first region to be processed. Hereinafter, the similarity between character strings will be referred to as character string similarity, and will be distinguished from image similarity.

[0083] When performing the similarity calculation process, the control unit 10 selects one of the plurality of dictionary data DD (here, the selected dictionary data DD is referred to as target dictionary data DD). The control unit 10 also selects one of the plurality of similar item names defined in the target dictionary data DD (here, the selected similar item name is referred to as target similar item name).

[0084] Then, the control unit 10 calculates the similarity between the character string in the first region to be processed and the character string of the target similar item name. For example, the control unit 10 can calculate the similarity between the character string in the first region to be processed and the character string of the target similar item name by using the following formula (1).

number

[0085] In formula (1), S1 is one character string and S2 is the other character string. For example, S1 is the character string of the similar item name defined in the dictionary data DD, and S2 is the character string of the first region extracted from the image data of an ID card. len(S1) is the number of characters in the character string S1, and len(S2) is the number of characters in the character string S2. M is the number of characters that match between the character string S1 and the character string S2.

[0086] For example, suppose that character string S1 is "Number" and character string S2 is "Nombern." Even if "Number" is written on the identification card, it may be recognized as "Nombern" due to a misrecognition in the character recognition process.

[0087] In this example, len(S1) = 6 and len(S2) = 7. Furthermore, between string S1 and string S2, counting from the first character, the first character "N", the third character "m", the fourth character "b", the fifth character "e", and the sixth character "r" match each other. As a result, M = 5.

[0088] In this example, the similarity value is 0.76923. If the character strings S1 and S2 contain completely different characters (i.e., M=0), the similarity value will be 0, and if the character strings S1 and S2 contain exactly the same characters (i.e., M=len(S1)=len(S2)), the similarity value will be 1. That is, in calculating the similarity between the character string in the first region to be processed and the character string of the target similar item name, the higher the similarity, the larger the similarity value, and the lower the similarity, the smaller the similarity value.

[0089] After determining the string similarity between the character string in the first region to be processed and the character string of the target similar item name, the control unit 10 newly selects an unselected similar item name from among the multiple similar item names defined in the target dictionary data DD. The control unit 10 switches the target similar item name and determines the string similarity between the character string in the first region to be processed and the character string of the new target similar item name. The control unit 10 determines the string similarity between the character string in the first region to be processed and all of the multiple similar item names defined in the target dictionary data DD. The control unit 10 then stores the similar item name that has the highest string similarity with the character string in the first region to be processed from among the multiple similar item names defined in the target dictionary data DD, in association with the value of the string similarity.

[0090] The control unit 10 also calculates the string similarity between all similar item names defined in the multiple dictionary data DD and the character string in the first domain to be processed. That is, the control unit 10 newly selects an unselected dictionary data DD from the multiple dictionary data DD. The control unit 10 switches the target dictionary data DD and calculates the string similarity between all similar item names defined in the new target dictionary data DD and the character string in the first domain to be processed. The control unit 10 then stores the similar item name, among the multiple similar item names defined in the target dictionary data DD, that has the highest string similarity with the character string in the first domain to be processed, in association with its string similarity value. As a result, multiple similar item names corresponding to multiple types of items are stored, each associated with a string similarity value. In other words, multiple similarity values ​​(the similarity values ​​are string similarity values) corresponding to multiple types of items are stored.

[0091] The control unit 10 performs a maximum value detection process (the process of step #14) as one process of the item name area determination process. In step #14, the control unit 10 compares the multiple similarity values ​​(the similarity values ​​are values ​​of character string similarity) calculated and stored in the similarity calculation process with each other. In other words, the control unit 10 compares the multiple similarity values ​​corresponding to the multiple types of items with each other.

[0092] Then, the control unit 10 detects the highest value of the character string similarity (i.e., the highest similarity value). That is, the control unit 10 calculates the character string similarity between the character string in the first region to be processed and each character string of all similar item names, and detects the highest value among the calculated similarity values.

[0093] The control unit 10 performs a threshold comparison process (processing of step #15) as part of the item name region determination process. In step #15, the control unit 10 compares a predetermined threshold for the item name region determination process with the highest value of the string similarity.

[0094] For example, to set a threshold value for the item name area determination process, an identification card is actually read by the multifunction device 100, and the result of a similarity calculation process for character strings present in the item name area of ​​the image data obtained by the read (here, the resulting value is referred to as the similarity value of the item name area) is obtained. Also, the result of a similarity calculation process for character strings present in an area of ​​the image data other than the item name area (here, the resulting value is referred to as the similarity value of the other area) is obtained. Note that multiple similarity values ​​for the item name area and multiple similarity values ​​for the other areas are obtained.

[0095] Furthermore, a first average value, which is the average value of the similarity values ​​of the plurality of item name regions, is calculated, and a second average value, which is the average value of the plurality of other regions, is calculated. The average value of the first average value and the second average value is then set as the threshold value for the item name region determination process.

[0096] In step #16, the control unit 10 determines whether the highest value of the character string similarity is equal to or greater than the threshold value for the item name area determination process. In Fig. 9, the threshold value for the item name area determination process is represented as Th. If the control unit 10 determines that the highest value of the character string similarity is equal to or greater than the threshold value for the item name area determination process, the process proceeds to step #17.

[0097] In step #17, the control unit 10 determines that the first region to be processed is an area for an item name. In step #18, the control unit 10 determines an item corresponding to the first region to be processed based on a character string as an item name present in the first region to be processed (i.e., the first region determined to be an area for an item name). Here, the character string in the first region to be processed is defined as a similar item name in one of the dictionary data DDs. Therefore, the control unit 10 recognizes an item corresponding to the dictionary data DD in which the character string in the first region to be processed is defined, and determines that the recognized item is an item corresponding to the first region to be processed.

[0098] In step #16, if the control unit 10 determines that the highest value of the character string similarity is less than the threshold value for the item name area determination process, the process proceeds to step #19. If the process proceeds to step #19, the control unit 10 determines that the first area to be processed is another area (i.e., an area different from the item name area).

[0099] Each of the processes from step #13 to step #19 is performed as one process of the item name area determination process. After completing the item name area determination process for a certain first area to be processed, the control unit 10 determines whether or not there are any first areas remaining for which the item name area determination process has not been performed (here referred to as unprocessed first areas). If there are any unprocessed first areas remaining, the control unit 10 sets the unprocessed first area as a new processing object. In other words, the control unit 10 switches the processing object. Then, the control unit 10 performs the item name area determination process (each of the processes from step #13 to step #19) for the new first area to be processed.

[0100] In this embodiment, dictionary data DD, in which a plurality of different similar item names are predefined for one type of item, is stored in advance in the storage unit 101 for each of a plurality of types of items. Then, the item name area determination process is performed using the dictionary data DD. This prevents an inconvenience that occurs when an item having a plurality of corresponding item names is to be masked, where the masked item is properly masked on an identification card on which one of the plurality of item names is written, but the masked item is not masked on an identification card on which another item name is written (in other words, the masked item is not detected).

[0101] For example, suppose an identification number is an item to be masked. In this case, in this embodiment, the identification number is properly masked on both an identification card with "ID" written as the item name and an identification card with "Num" written as the item name. On the other hand, in a conventional configuration, the identification number is properly masked on an identification card with "ID" written as the item name, but the identification number is not masked on an identification card with "Num" written as the item name, which can be an inconvenience.

[0102] To prevent these problems, a large dictionary is required. Furthermore, when using machine learning models, the model size becomes large. However, because memory capacity is limited, processing is performed on the cloud.

[0103] In this embodiment, by using dictionary data DD, processing on the cloud is not required. In other words, there is no need to transfer image data of the identification card (i.e., personal information) to a processing device on the cloud. This makes it possible to prevent the leakage of personal information. In other words, in this embodiment, it is possible to easily mask only the areas corresponding to the items specified by the user from the image data obtained by scanning the document D (identification card) without transferring the image data of the identification card to a processing device on the cloud.

[0104] Furthermore, in this embodiment, the string similarity between the string in the first region to be processed and each string in all similar item names is calculated, and if the highest value of the calculated string similarity is equal to or greater than a threshold, the first region to be processed is determined to be an item name region. Therefore, even if misrecognition occurs in the character recognition process, the item name region can be detected with high accuracy.

[0105] <Anomaly Countermeasure Processing> First, the first abnormality countermeasure process will be described.

[0106] Typically, the multiple items of personal information listed on an ID card are different from each other. For example, it is highly unlikely that a single ID card will have multiple items such as "name." However, if the accuracy of the character recognition process for the image data of the ID card is low, multiple item names (character strings thereof) that each correspond to the same item may be extracted from the image data of the ID card. If there is dirt or discoloration on part of the ID card, or if the resolution used to read the ID card is low, the accuracy of the character recognition process will decrease.

[0107] To prevent such problems from occurring, the control unit 10 performs a first abnormality countermeasure process. Specifically, when the character strings in multiple first areas determined to be item name areas correspond to the same first item, the control unit 10 determines the first area containing the character string with the highest maximum value as the area corresponding to the first item, and determines the remaining first areas as not areas corresponding to the first item. In other words, the control unit 10 corrects the result of the item name area determination process.

[0108] The flow of the first abnormality countermeasure processing will be described below with reference to the flowchart shown in Fig. 10. The flow shown in Fig. 10 starts when the item name area determination processing by the control unit 10 (i.e., processing according to the flow shown in Fig. 9) is completed. After completing the item name area determination processing, the control unit 10 continues to perform the first abnormality countermeasure processing on the first area to be processed.

[0109] In step #21, the control unit 10 determines whether the first area to be processed is determined to be an area of ​​an item name. If the control unit 10 determines that the first area to be processed is an area of ​​an item name, the process proceeds to step #22. If the control unit 10 determines that the first area to be processed is not an area of ​​an item name, the process ends.

[0110] In step #22, the control unit 10 determines whether the item corresponding to the first region to be processed is the same as the item corresponding to another first region. In other words, the control unit 10 determines whether the character strings in each of the first regions determined to be regions of item names correspond to the same item (first item).

[0111] If the control unit 10 determines in step #22 that the item corresponding to the first area to be processed is the same as the item corresponding to another first area, the process proceeds to step #23. In other words, if the control unit 10 determines that the character strings in each of the multiple first areas determined to be item name areas correspond to the same item (first item), the process proceeds to step #23. If the control unit 10 determines that the item corresponding to the first area to be processed is different from the item corresponding to another first area, the process ends. Hereinafter, another first area that has the same corresponding item as the first area to be processed will be referred to as the first area to be compared.

[0112] In step #23, the control unit 10 recognizes the highest value found in the highest value detection process (i.e., the process of step #14 shown in FIG. 9) for each of the first region to be processed and the first region to be compared. Then, the control unit 10 compares the highest values ​​of the first region to be processed and the first region to be compared with each other. As a result, the highest values ​​of the first region to be processed and the first region to be compared with each other. For example, the control unit 10 determines whether the highest value M1 of the first region to be processed is equal to or greater than the highest value M2 of the first region to be compared.

[0113] In step #23, if the control unit 10 determines that the highest value M1 of the first region to be processed is equal to or greater than the highest value M2 of the first region to be compared, the process proceeds to step #24. If the control unit 10 determines that the highest value M1 of the first region to be processed is less than the highest value M2 of the first region to be compared, the process proceeds to step #25.

[0114] When the process proceeds to step #24, the control unit 10 does not change the determination result that the first region to be processed is the region corresponding to the first item, but changes the determination result for the first region to be compared. Specifically, the control unit 10 recognizes the second item, which is the item corresponding to the similar item name with the second highest string similarity to the string of the first region to be compared, and determines that the first region to be compared is the region corresponding to the second item.

[0115] When the process proceeds to step #25, the control unit 10 does not change the determination result that the first region to be compared is the region corresponding to the first item, but changes the determination result for the first region to be processed. Specifically, the control unit 10 recognizes the second item, which is the item corresponding to the similar item name having the second highest string similarity with the string of the first region to be processed, and determines that the first region to be processed is the region corresponding to the second item.

[0116] In other words, the following processes are performed in steps #23 to #25.

[0117] When the character strings in each of the multiple first regions determined to be regions of an item name correspond to the same first item, the control unit 10 determines, among the multiple first regions, the first region containing the character string with the highest maximum value as the region corresponding to the first item. Furthermore, the control unit 10 recognizes, among the multiple first regions, a second item that is an item corresponding to a similar item name having the second highest character string similarity to the character string in a first region not determined to be a region corresponding to the first item, and determines the first region not determined to be a region corresponding to the first item as a region corresponding to the second item.

[0118] Next, the second abnormality countermeasure processing and the third abnormality countermeasure processing will be described.

[0119] An identification card has multiple item fields. Each item field contains one item name, and it is rare for multiple item names to be written in one item field. In other words, normally, two or more of the multiple first areas contained in the same second area are not determined to be item name areas. Also, normally, one of the multiple first areas contained in the same second area is determined to be an item name area. However, if the accuracy of the character recognition process for the image data of the identification card is low, it is possible that two or more of the multiple first areas contained in the same second area are determined to be item name areas, or none of them are determined to be item name areas.

[0120] Therefore, the control unit 10 performs a second abnormality countermeasure process and a third abnormality countermeasure process. Here, multiple first areas included in the same second area are linked to each other. Therefore, the control unit 10 performs the second abnormality countermeasure process or the third abnormality countermeasure process on the multiple first areas that are linked to each other.

[0121] The following describes the flow of the second abnormality countermeasure processing and the third abnormality countermeasure processing with reference to the flowcharts shown in FIGS.

[0122] First, the control unit 10 performs processing according to the flow shown in Fig. 11. The flow shown in Fig. 11 starts when the area determination processing by the control unit 10 (i.e., processing according to the flow shown in Fig. 9) is completed. After completing the item name area determination processing, the control unit 10 continues to perform processing according to the flow shown in Fig. 11.

[0123] In step #31, the control unit 10 performs a character string acquisition process to acquire a character string in a first region linked to the first region to be processed. Then, in step #32, the control unit 10 determines whether or not a region determination process has been performed on the first region including the character string acquired in the character string acquisition process (i.e., whether or not a determination has been made). If it is determined that a determination has been made, the control unit 10 proceeds to step #33, and if it is determined that a determination has not been made, the control unit 10 ends this flow.

[0124] In step #33, the control unit 10 determines whether the first area to be processed and the first area linked to the first area to be processed are both determined to be areas of item names. In other words, the control unit 10 determines whether two or more of the multiple first areas linked to each other are determined to be areas of item names.

[0125] In step #33, if the control unit 10 determines that two or more of the first areas linked to each other are determined to be areas of item names, the process proceeds to step #34. When the process proceeds to step #34, the control unit 10 performs a second abnormality countermeasure process.

[0126] If the control unit 10 determines in step #33 that two or more of the multiple first areas linked to each other have not been determined to be item name areas, the process proceeds to step #35. In step #35, the control unit 10 determines whether the first area to be processed and the first area linked to the first area to be processed have both been determined to be areas different from item name areas. In other words, the control unit 10 determines whether all of the multiple first areas linked to each other have been determined to be areas different from item name areas.

[0127] In step #35, if the control unit 10 determines that all of the multiple first areas linked to each other are determined to be areas different from the area of ​​the item name, the process proceeds to step #36. When the process proceeds to step #36, the control unit 10 performs a third abnormality countermeasure process.

[0128] In step #35, if the control unit 10 has not determined that all of the multiple first areas linked to each other are areas different from the area of ​​the item name, this flow ends. In other words, if only one of the multiple first areas linked to each other is determined to be the area of ​​the item name, neither the second abnormality countermeasure process nor the third abnormality countermeasure process is performed.

[0129] As the second abnormality countermeasure process, a process is performed according to the flow shown in Fig. 12. That is, when the control unit 10 determines that two or more of the multiple first areas linked to each other (that is, multiple first areas included in the same second area) are areas of item names, a process is performed according to the flow shown in Fig. 12.

[0130] In step #41, the control unit 10 recognizes the highest value found in the highest value detection process (i.e., the process of step #14 in FIG. 9) for each of the multiple first regions linked to one another. In step #42, the control unit 10 recognizes the highest highest value.

[0131] Then, in step #43, the control unit 10 determines that the first area containing the character string with the highest maximum value (herein referred to as the first area with the highest maximum value) is an area of ​​the item name among the multiple first areas linked to each other. Also, in step #44, the control unit 10 determines that the remaining first areas other than the first area with the highest maximum value are areas other than the area of ​​the item name among the multiple first areas linked to each other.

[0132] Here, if a first area present in the image data of the identification card is determined to be an area of ​​an item name, the control unit 10 stores information indicating that the first area corresponds to the item name. In this configuration, when the second abnormality countermeasure process is performed, for a first area that is re-determined to be an area different from the area of ​​the item name, the information indicating that the area corresponds to the item name should be deleted. Therefore, in step #45, the control unit 10 deletes the information indicating that the remaining first areas other than the first area with the highest maximum value correspond to the item name.

[0133] As the third abnormality countermeasure process, a process is performed according to the flow shown in Fig. 13. That is, when the control unit 10 determines that all of the multiple first areas linked to each other (that is, multiple first areas included in the same second area) are areas different from the area of ​​the item name, a process is performed according to the flow shown in Fig. 13.

[0134] In step #51, the control unit 10 recognizes the highest value found in the highest value detection process (i.e., the process of step #14 in FIG. 9) for each of the multiple first regions linked to one another. In step #52, the control unit 10 recognizes the highest highest value.

[0135] Then, in step #53, the control unit 10 determines that the first region containing the character string with the highest maximum value (herein referred to as the first region with the highest maximum value) is the region of the item name among the multiple first regions linked to each other. Also, in step #54, the control unit 10 determines that the remaining first regions other than the first region with the highest maximum value are regions other than the region of the item name among the multiple first regions linked to each other.

[0136] Here, if the first area in the image data of the ID is determined to be an area for an item name, the control unit 10 stores information indicating that the first area corresponds to the item name. In this configuration, when the third abnormality countermeasure process is performed, information indicating that the first area corresponds to the item name should be added to the first area that is re-determined as an area for an item name by that process. Therefore, in step #55, the control unit 10 adds information indicating that the first area with the highest maximum value corresponds to the item name.

[0137] In this embodiment, by performing the first to third abnormality countermeasure processes, it is possible to prevent the occurrence of a problem in which multiple areas with an item name corresponding to the same item are detected, and also to prevent the occurrence of a problem in which an area with an item name is not detected.

[0138] <Masking job settings> The masking job can be set by the user. The masking job is set via the operation display unit 3. For example, the user can specify the item corresponding to the item value to be masked (i.e., the target item).

[0139] Specifically, the operation display unit 3 accepts an item designation operation to designate one of the items. The control unit 10 recognizes the item designated by the item designation operation as the target item. Then, as a masking process, the control unit 10 performs a process of masking at least the area of ​​the item value among the areas of the target item present in the image data of the identification card to generate masked data.

[0140] The flow of processing performed by the control unit 10 when accepting settings for a masking job will be described below with reference to the flowchart shown in Fig. 14. For example, in a masking job, after the image reading unit 2 reads an identification card and before the masking process starts, an item specification operation is accepted by the operation display unit 3. That is, after the process of step #1 shown in Fig. 4 and before the process of step #7, the flow shown in Fig. 14 starts.

[0141] In step #61, the operation display unit 3 displays a reception screen 31 as shown in FIG. 15. Reception buttons RB are arranged on the reception screen 31. The reception buttons RB include a basic settings button B1 and a custom button B2. The operation display unit 3 accepts an operation on any of the reception buttons RB as an item designation operation. The control unit 10 determines the user-designated item (i.e., the target item) based on which reception button RB the item designation operation was performed on.

[0142] Specifically, the storage unit 101 stores a masking list in which at least one item is registered. The reception button RB is associated with the masking list. When there are multiple masking lists, multiple reception buttons RB (specifically, the same number as the number of masking lists) are displayed on the reception screen 31. The multiple reception buttons RB are associated with different masking lists.

[0143] The masking list corresponding to the basic settings button B1 is registered with items preselected by the manufacturer. The masking list corresponding to the custom button B2 is registered with items arbitrarily selected by the user. By creating a new masking list, the custom button B2 (i.e., the reception button RB) corresponding to the newly created masking list will be displayed on the reception screen 31.

[0144] In order to receive a request to create a new masking list from the user, a new creation button NB is displayed on the reception screen 31. The operation display unit 3 receives an operation on the new creation button NB as a request to create a new masking list.

[0145] When a request to create a new masking list is received, the operation display unit 3 displays a registration screen 32 as shown in Fig. 16 and receives a registration operation from the user to select an item to be registered. A plurality of items (the item names) are displayed as options on the registration screen 32.

[0146] The operation display unit 3 accepts an operation on the display field of any of the displayed items as a registration operation. For example, if you want to register the item "height," you can select the item "height" as the item to be registered by operating the display field of the item "height." A check mark is added to the display field of the item selected in the registration operation. By operating the decision button DB in the registration screen 32, you can complete the selection of the item in the registration operation.

[0147] When an operation is performed on the decision button DB, the control unit 10 determines that acceptance of the registration operation has been completed. After accepting the registration operation, the control unit 10 generates a new masking list in which the items selected in the registration operation are registered, and stores the new masking list in the storage unit 101.

[0148] When a new masking list is created and the operation display unit 3 subsequently receives an item designation operation, the control unit 10 causes the operation display unit 3 to display a reception screen 31 on which a reception button RB corresponding to the new masking list is arranged in addition to the reception button RB corresponding to the existing masking list.

[0149] When any of the reception buttons RB is operated, the control unit 10 determines that the operation display unit 3 has received an item designation operation. At this time, the control unit 10 determines that the masking list corresponding to the reception button RB for which the item designation operation was performed has been designated. In other words, the item designation operation is an operation for designating any of the masking lists.

[0150] 14, in step #62, the control unit 10 recognizes the reception button RB on which the item designation operation has been performed, i.e., the masking list designated by the item designation operation. Then, the control unit 10 recognizes the item registered in the masking list designated by the item designation operation as the target item.

[0151] In step #63, the control unit 10 performs a masking process. The process of step #63 is the same as the process of step #7 shown in Fig. 4. Note that the control unit 10 performs the masking process after performing each of the processes of steps #2 to #6 shown in Fig. 4.

[0152] In step #64, the control unit 10 causes the operation display unit 3 to display a preview image (not shown) showing the contents of the masked data, and also causes the operation display unit 3 to accept an editing operation to edit the masked data. An editing operation is an operation to change an item to be masked. In other words, the editing operation includes an operation to add a new item to be masked, and an operation to remove a masked item from the masking target.

[0153] In step #65, the control unit 10 determines whether an editing operation has been performed. If an editing operation has been performed, the process proceeds to step #61. After that, in step #63, the control unit 10 generates new masked data that reflects the edited content of the editing operation.

[0154] If it is determined in step #65 that no editing operation has been performed, the process proceeds to step #66. In step #66, the control unit 10 causes the output unit to perform output processing of the masked data. The processing in step #66 is the same as the processing in step #8 shown in FIG.

[0155] In this embodiment, after scanning the ID card, an item designation operation is received to designate the item to be masked (i.e., the target item). By performing the item designation operation, at least the area of ​​the item value among the areas of the target item present in the image data of the ID card is masked. In other words, it is possible to easily mask only the areas of the image data obtained by scanning the document D (ID card) that correspond to the items designated by the user.

[0156] For example, in a conventional configuration, it is necessary to scan an ID card in advance and register the area to be masked (the position within the image data). In other words, the ID card must be registered in advance. In a conventional configuration, if the ID card is registered in advance, it is possible to mask the desired area within the image data of the ID card.

[0157] However, from the user's perspective, the task of pre-registering ID cards is cumbersome. If there are many types of ID cards that can be used, pre-registration of each type of ID card must be performed, which increases the amount of work. In addition, the masking target cannot be changed flexibly. If a user wants to change the masking target, they must re-register the ID card, which is cumbersome.

[0158] On the other hand, in this embodiment, when a certain identification card is read by performing an item designation operation, the area of ​​the image data of the item designated by the item designation operation is masked, and when another identification card of a different type from the identification card is read, the area of ​​the image data of the item designated by the item designation operation is also masked. Also, for example, when the masking target is item A, if it is necessary to further add item B as a masking target (i.e., it is necessary to change the masking target), it is sufficient to simply designate items A and B by the item designation operation. As a result, work equivalent to pre-registering an identification card is not required, improving user convenience.

[0159] In this embodiment, the item designation operation is an operation for designating a masking list. This eliminates the need to designate each item one by one when masking multiple items. This further improves user convenience.

[0160] In addition, in this embodiment, a new masking list can be created, which is convenient for the user as the masking list can be created as desired.

[0161] Furthermore, in this embodiment, the masked data can be edited, improving user convenience. When editing the masked data, a preview image showing the contents of the masked data is displayed. This allows the masked data to be previewed while being edited, thereby enabling efficient editing of the masked data.

[0162] <Handwritten signature masking> Some identification cards include a handwritten signature. A handwritten signature is a type of personal information. A handwritten signature is a string of handwritten characters only. An example of such a document D (i.e., an identification card including a handwritten signature) is shown in FIG. 17. For convenience, in FIG. 17, the string "abcd" is taken to be the handwritten signature.

[0163] For example, a first learning model for a first region detection process is trained to detect a region containing characters as a first region, but the characters in that region are printed characters. Similarly, a second learning model for a second region detection process is trained to detect a region containing characters as a second region, but the characters in that region are printed characters.

[0164] Therefore, the first region detection process and the second region detection process detect a type region where all the characters within the region are type. That is, the first region detected by the first region detection process and the second region detected by the second region detection process are type regions.

[0165] Here, in some cases, it may be necessary to selectively mask a handwritten signature as personal information. However, the first area detection process and the second area detection process may not be able to detect the area of ​​the handwritten signature. For example, if the handwritten characters constituting the handwritten signature are significantly distorted, the first area detection process and the second area detection process may not be able to detect the area of ​​the handwritten signature. For this reason, if only the first area detection process and the second area detection process are performed as processes for detecting an area that may be subject to masking from image data, the inconvenience of not being able to mask the handwritten signature may occur.

[0166] Therefore, the control unit 10 further performs handwritten area detection processing to detect areas that may be subject to masking from the image data. In the handwritten area detection processing, handwritten areas in which all characters are handwritten are detected from the image data obtained by scanning the identification card.

[0167] Furthermore, when handwritten area detection processing is additionally performed, a handwritten signature (i.e., personal information written in handwritten characters) can be designated as personal information to be masked in the masking job settings. For example, an item "handwritten signature" is added as an option to the registration screen 32 (see FIG. 16). By selecting the item "handwritten signature," an acceptance button RB corresponding to the masking list in which the item "handwritten signature" is registered is displayed on the reception screen 31 (see FIG. 15). When the acceptance button RB is operated, the control unit 10 recognizes the handwritten signature as personal information to be masked. In this configuration, an operation on the acceptance button RB corresponding to the masking list in which the item "handwritten signature" is registered corresponds to an operation of designating the handwritten signature as personal information to be masked. In other words, the operation display unit 3 accepts an operation of designating information to be masked.

[0168] The flow of a masking job in which handwritten area detection processing is additionally performed will be described below with reference to the flowchart shown in FIG.

[0169] 18 starts when a start operation for a masking job is performed on the operation display unit 3. The control unit 10 sequentially performs the processes of steps #71 to #76. Note that the processes of steps #71 to #76 are the same as the processes of steps #1 to #6. Therefore, detailed explanations of the processes of steps #71 to #76 will be omitted, as the explanations of the processes of steps #1 to #6 will be used.

[0170] Next, in step #77, the control unit 10 performs handwritten area detection processing using a third learning model obtained by machine learning. The third learning model for handwritten area detection processing is a learning model that has been trained to detect handwritten areas from image data obtained by reading an identification card with the image reading unit 2. The third learning model is a trained model that is pre-stored in the storage unit 101.

[0171] For example, when the identification card shown in Fig. 17 is read, as shown in Fig. 19, a character area C41 for the item name "Name," a character area C42 for the item value "abcd (printed characters)," and a character area C51 for the item name "Signature" are each detected individually as the first area. Note that the character string "Name" corresponds to the item name, and the character string "abcd (printed characters)" corresponds to the item value of the item corresponding to the item name "Name." Therefore, as shown in Fig. 20, an area C40 including character areas C41 and C42 is detected as the second area.

[0172] In the identification card shown in FIG. 17, character areas C41, C42, and C51 are each a typed character area. However, the character string "abcd" below character area C51 is a handwritten signature. That is, all characters in character area C52 are handwritten. In this case, in the first area detection process, character areas C41, C42, and C51 are each detected as a first area, but character area C52 is not detected as a first area. Similarly, in the second area detection process, character area C52 is not detected as a second area.

[0173] On the other hand, in the handwritten area detection process, character area C52 is detected as a handwritten area. Note that all characters in the other character areas are printed characters. For this reason, the other character areas are not detected as handwritten areas.

[0174] After the process of step #77 (handwritten area detection process), in step #78, the control unit 10 performs a masking process. Then, in step #79, the control unit 10 performs an output process. Note that the processes of steps #78 and #79 are the same as the processes of steps #7 and #8. Therefore, detailed explanations of the processes of steps #78 and #79 will be omitted, as the explanations of the processes of steps #7 and #8 will be used.

[0175] For example, when the operation display unit 3 receives an operation to designate a handwritten signature as personal information to be masked, the control unit 10 masks the handwritten area in the image data of the identification card. Here, the control unit 10 does not perform character recognition processing on the handwritten area. Therefore, when the operation display unit 3 receives an operation to designate a handwritten signature as personal information to be masked, the control unit 10 masks the entire handwritten area regardless of the content indicated by the characters present in the handwritten area.

[0176] As a result, when the identification card shown in Fig. 17 is to be read and the handwritten signature is personal information that should be masked, masked data such as that shown in Fig. 21 is generated. That is, the character area C52, which is a handwritten area, is masked.

[0177] In this embodiment, in addition to the first area detection process and the second area detection process, a handwritten area detection process is performed, thereby enabling accurate detection of handwritten areas present in the image data of the identification card. Here, if a handwritten signature is present on the identification card, the handwritten area detection process is performed on the image data of the identification card, and an area corresponding to the handwritten signature in the image data of the identification card is detected as a handwritten area. Thus, when a handwritten signature is designated as personal information to be masked, masking the handwritten area in the image data of the identification card ensures that the personal information designated by the user is masked.

[0178] In addition, in this embodiment, a trained third learning model is used. The third learning model is stored in advance in the storage unit 101. This allows the handwritten area detection process to be performed within the multifunction device 100 without increasing the memory capacity within the multifunction device 100. In other words, there is no need to transfer image data of the identification card to a processing device on the cloud. As a result, leakage of personal information can be suppressed.

[0179] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims rather than the description of the above embodiments, and further includes all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0180] 1 Printing unit (output unit) 2 Image reading unit 3 Operation display section 10 Control Unit 100 Multifunction printers (image processing devices) 101 Storage section 102 Communication unit (output unit) 1000 PCs (external equipment) D Manuscript S seat

Claims

1. an image reading unit that reads a document on which a plurality of sets of item names and item values ​​of items related to personal information are written; a control unit that recognizes a target item that is an item to be masked, and performs a masking process on image data obtained by reading the document by the image reading unit, thereby generating masked data in which at least a part of an area of ​​the target item present in the image data is masked; The control unit a first region detection process for detecting a first region that is predicted to individually include either an area of ​​the item name or an area of ​​the item value from the image data; a second region detection process for detecting a second region from the image data that is predicted to include regions of both the item name and the item value corresponding to the same item; an inclusion area detection process that is performed for each of the plurality of first areas, and that detects the second area that includes the first area to be processed by detecting the second area having an image portion whose image similarity with the first area to be processed is equal to or greater than a threshold; performing a region linking process for linking the plurality of first regions included in the same second region to each other; In the first area detection process and the second area detection process, a type area in which characters are typed is detected, The control unit further performs a handwritten area detection process for detecting a handwritten area in which characters present within the area are handwritten characters from the image data.

2. A storage unit is provided, The storage unit a trained first learning model that has been machine-trained to detect the first region from the image data; a second learning model that has been trained by machine learning to detect the second region from the image data; a trained third learning model that has been machine-trained to detect the handwritten region from the image data; The control unit performing a process of detecting the first region from the image data using the first learning model as the first region detection process; performing a process of detecting the second region from the image data using the second learning model as the second region detection process; The image processing device according to claim 1 , wherein the handwritten area detection process is a process of detecting the handwritten area from the image data using the third learning model.

3. an operation display unit that displays information and accepts operations; the operation display unit accepts an operation to designate information to be masked, the control unit does not perform character recognition processing on the handwritten area, The image processing device according to claim 1 , wherein when the operation display unit receives an operation to designate a handwritten signature as information to be masked, the control unit masks the handwritten area.

4. an output unit that performs output processing of the masked data, 2. The image processing device according to claim 1, wherein the output unit is at least one of a printing unit that performs the output process of printing an image based on the masked data onto a sheet, and a communication unit that performs the output process of transmitting the masked data to an external device.

Images