Image processing apparatus

Abstract

To easily mask only an area corresponding to an item designated by a user in image data obtained by reading a document.SOLUTION: An image processing apparatus includes an image reading unit, a control unit, and a storage unit. The storage unit stores dictionary data in which a plurality of different similar item names are defined in advance for one type of item. The control unit performs first area detection processing of detecting a first area predicted to individually include either an area of an item name or an area of an item value from the image data, character recognition processing, and item name area determination processing, which is processing performed for each of the plurality of first areas, of obtaining, for all the similar item names, a character string similarity with a partial character string obtained by subdividing a character string of the first area to be processed and determining that the first area to be processed is an area of an item name when the character string similarity between any of the similar item names and the partial character string is equal to or more than a threshold.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to an image processing device. [Background technology]

[0002] Conventionally, there has been known an image processing device that reads an identification card as a document and masks an area of ​​personal information in the image data obtained by the reading. Such an image processing device is disclosed, for example, in Patent Document 1. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-197682 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, personal information written on an ID card is classified into multiple categories, such as name and address. With such an ID card, it may be desirable to mask only the areas of the image data obtained by scanning that correspond to specific categories. Note that the areas that need to be masked vary from user to user.

[0005] In order to detect an area corresponding to a specified item from the image data obtained by scanning, a method using a language model such as BERT (Bidirectional Encoder Representations from Transformers) may be used. However, this method requires a large model size and a huge amount of calculation. For this reason, for example, image data of an ID card is transferred to the cloud and processed on the cloud.

[0006] If image data of an ID card is transferred to the cloud, there is a risk of personal information being leaked, so transferring image data of an ID card to the cloud is not recommended.

[0007] The present invention has been made to solve the above-mentioned problems, and aims to provide an image processing device that can easily mask only the areas of image data obtained by reading a document that correspond to items specified by the user. [Means for solving the problem]

[0008] To achieve the above object, an image processing device according to one aspect of the present invention includes an image reading unit that reads a document containing multiple sets of item names and item values ​​for personal information items; a control unit that recognizes target items that are to be masked and performs a masking process on image data obtained by reading the document with the image reading unit to generate masked data in which at least a portion of the target item area present in the image data is masked; and a storage unit. The storage unit stores dictionary data in which multiple similar item names are predefined for each type of item, for each type of item. The control unit performs a first area detection process that detects first areas that are predicted to individually include either an item name area or an item value area from the image data, a character recognition process that recognizes character strings in the first areas, and an item name area determination process that is performed for each of the multiple first areas, in which the control unit calculates string similarities between all similar item names and substrings obtained by dividing the character string in the first area to be processed, and determines the first area to be processed as an item name area if the string similarity between any similar item name and the substring is equal to or greater than a threshold. [Effects of the Invention]

[0009] In the configuration of the present invention, it is possible to easily mask only the areas of image data obtained by scanning an original that correspond to items designated by the user. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram of a multifunction peripheral according to an embodiment. [Figure 2] FIG. 1 is a block diagram of a multifunction peripheral according to an embodiment. [Figure 3] 1 is a diagram illustrating an example of an identification card that can be read by a multifunction peripheral according to an embodiment; [Figure 4] 10 is a flowchart illustrating the flow of a masking job executed by the multifunction peripheral of an embodiment. [Figure 5] 4 is a diagram showing a first region in image data obtained by reading the identification card shown in FIG. 3. FIG. [Figure 6] 4 is a diagram showing a second area in image data obtained by reading the identification card shown in FIG. 3. FIG. [Figure 7] 4 is a diagram showing masked data generated by a masking job for the identification card shown in FIG. 3. FIG. [Figure 8] FIG. 10 is a conceptual diagram of dictionary data used in a masking job by the multifunction peripheral of an embodiment. [Figure 9] 10 is a flowchart illustrating a flow of an area determination process performed in the multifunction peripheral according to an embodiment. [Figure 10] 10 is a flowchart illustrating a flow of exception processing performed in the multifunction peripheral of an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] <Configuration of the multifunction device> An image processing apparatus according to an embodiment of the present invention will be described below with reference to FIGS. 1 to 10, taking as an example a multifunction peripheral 100 having multiple functions such as a scanning function, a printing function, and a transmission function.

[0012] As shown in FIG. 1, the multifunction device 100 (corresponding to an "image processing device") includes a printing unit 1. The printing unit 1 constitutes the main body of the multifunction device 100. The printing unit 1 prints an image on a sheet S. The printing method of the printing unit 1 is an electrophotographic method. However, this is not limited to this. The printing method of the printing unit 1 may also be an inkjet method.

[0013] The printing unit 1 forms an image based on image data input to the multifunction device 100. The printing unit 1 also transports a sheet S along a sheet transport path. The printing unit 1 prints an image on the sheet S while it is being transported. In FIG. 1, the sheet transport path is indicated by a dashed line.

[0014] The printing unit 1 includes a paper feed roller 11. The paper feed roller 11 comes into contact with a sheet S stored in a sheet cassette CA and rotates in this state, thereby feeding the sheet S from the sheet cassette CA to a sheet transport path.

[0015] The printing unit 1 includes an image forming unit 12. The image forming unit 12 includes a photosensitive drum 12a and a transfer roller 12b. The photosensitive drum 12a carries a toner image on its peripheral surface. The transfer roller 12b is in pressure contact with the photosensitive drum 12a, forming a transfer nip between the photosensitive drum 12a and the transfer roller 12b. The transfer roller 12b rotates together with the photosensitive drum 12a. The image forming unit 12 transfers the toner image onto the sheet S while transporting the sheet S that has entered the transfer nip.

[0016] Although not shown, the image forming unit 12 further includes a charging device, an exposure device, and a developing device. The charging device charges the circumferential surface of the photosensitive drum 12a. The exposure device forms an electrostatic latent image on the circumferential surface of the photosensitive drum 12a. The developing device develops the electrostatic latent image on the circumferential surface of the photosensitive drum 12a into a toner image.

[0017] The printing unit 1 includes a fixing unit 13. The fixing unit 13 includes a heating roller 13a and a pressure roller 13b. The heating roller 13a has a built-in heater (not shown). The pressure roller 13b is pressed against the heating roller 13a, forming a fixing nip between the heating roller 13a and the pressure roller 13b. The pressure roller 13b rotates together with the heating roller 13a. The fixing unit 13 fixes the toner image transferred onto the sheet S onto the sheet S while transporting the sheet S that has entered the fixing nip. The sheet S that has exited the fixing nip is discharged onto an ejection tray ET.

[0018] The multifunction device 100 also includes an image reading unit 2. The image reading unit 2 is disposed on top of the main body of the multifunction device 100. In a job involving reading an original D, the original D is set in the image reading unit 2. The image reading unit 2 reads the original D set in the image reading unit 2 and generates image data of the read original D.

[0019] The image reading unit 2 includes contact glasses G1 and G2. The contact glasses G1 and G2 are installed in a housing RH of the image reading unit 2. The housing RH has an opening on the top surface. The contact glasses G1 and G2 are attached to the opening on the top surface of the housing RH.

[0020] The image reading unit 2 includes a document transport device DP. The document transport device DP is attached to the housing RH. When viewed from the front of the multifunction device 100, the document transport device DP pivots around its rear portion as if swinging its front portion up and down. The document transport device DP opens and closes relative to the top surface of the housing RH.

[0021] The document transport device DP has a set tray ST on which a document D is set. The document transport device DP transports the document D set on the set tray ST onto the contact glass G1.

[0022] In the conveying reading mode, the user sets the document D on the set tray ST. Then, the document D is automatically conveyed onto the contact glass G1 by the document conveying device DP (in other words, the document D passing over the contact glass G1) and is read. On the other hand, in the placed reading mode, the user sets the document D on the contact glass G2, and the document D on the contact glass G2 is read.

[0023] The image reading unit 2 includes a light source 21, an image sensor 22, a mirror 23, and a lens 24. The light source 21, the image sensor 22, the mirror 23, and the lens 24 are installed inside the housing RH. The image reading unit 2 performs a scanning operation in which light is irradiated from the light source 21 toward the contact glass G1 or G2 and photoelectrically converted by the image sensor 22.

[0024] The light source 21 has a plurality of LED elements. The plurality of LED elements are arranged in a line in the main scanning direction (a direction perpendicular to the paper surface of FIG. 1). The image sensor 22 has a plurality of photoelectric conversion elements arranged in the main scanning direction. The mirror 23 reflects light toward the lens 24. The lens 24 collects the light reflected by the mirror 23 and guides it to the image sensor 22.

[0025] The light source 21 and the mirror 23 are mounted on a carriage 25 that is movable in a sub-scanning direction (left and right direction in FIG. 1) that is perpendicular to the main scanning direction. As the carriage 25 moves in the sub-scanning direction, the reading line of the image reading unit 2 moves in the sub-scanning direction.

[0026] As shown in FIG. 2, the multifunction device 100 also includes an operation display unit 3. The operation display unit 3 is an operation panel having a touch screen. The operation display unit 3 displays software buttons, messages, and the like on the touch screen. The operation display unit 3 is also provided with a plurality of hardware buttons. The operation display unit 3 accepts operations from the user. The user can configure the multifunction device 100 via the operation display unit 3 with respect to various jobs, such as a masking job, which will be described later.

[0027] The multifunction peripheral 100 includes a control unit 10. The control unit 10 includes a CPU, an ASIC, a memory, and the like. The control unit 10 also includes an image processing circuit. The control unit 10 performs various image processing operations on image data. The control unit 10 also controls the printing of an image onto a sheet S by the printing unit 1, and the reading of an original D by the image reading unit 2.

[0028] The control unit 10 also controls the operation and display unit 3. Specifically, the control unit 10 controls the display operation of the touch screen. The control unit 10 detects operations on software buttons and hardware buttons. The control unit 10 sets jobs based on operations received by the operation and display unit 3 from the user.

[0029] The multifunction peripheral 100 includes a storage unit 101. The storage unit 101 is a non-volatile storage device. An HDD, an SSD, or the like may be used as the storage unit 101. The storage unit 101 is connected to the control unit 10. The control unit 10 writes information to the storage unit 101 and reads information from the storage unit 101.

[0030] The storage unit 101 stores predetermined information in advance. For example, the storage unit 101 stores a character recognition program in advance. The control unit 10 performs character recognition processing such as OCR (Optical Character Recognition) processing based on the character recognition program. The control unit 10 subjects image data obtained by reading the document D with the image reading unit 2 to the character recognition processing.

[0031] The multifunction peripheral 100 includes a communication unit 102. The communication unit 102 is an interface for communicatively connecting an external device to the multifunction peripheral 100. The communication unit 102 includes a communication circuit, a communication memory, a communication connector, and the like. The communication unit 102 is connected to the control unit 10. The control unit 10 uses the communication unit 102 to send and receive data to and from the external device.

[0032] The communication unit 102 is communicably connected to an external device via a network NT such as a LAN or the Internet. Although not shown, the communication unit 102 may be directly connected to an external device via a communication cable. An external device connected to the communication unit 102 is, for example, a personal computer 1000 (hereinafter referred to as PC 1000) used by a user of the multifunction peripheral 100. An external device other than the PC 1000 may be communicably connected to the multifunction peripheral 100. By connecting the PC 1000 to the multifunction peripheral 100, image data of the original D obtained by reading the original D with the image reading unit 2 can be transmitted to the PC 1000. This allows the image data of the original D to be saved in the PC 1000.

[0033] <Masking job overview> When entering into a contract for goods or services, the person wishing to enter into the contract may be required to register personal information. This is just one example, and there are various other cases where registration of personal information is required. Examples of personal information include name, address, and date of birth.

[0034] The user of the multifunction device 100 registers personal information. When registering personal information, the user uses the scanning function of the multifunction device 100 to scan their identification card. There are various types of identification cards that can be used, and these vary depending on the country and field. Examples of identification cards include driver's licenses, insurance cards, student ID cards, and passports.

[0035] An example of an ID card is shown schematically in Figure 3. Personal information is classified into multiple items and written on the ID card. In other words, the ID card lists multiple sets of item names and item values ​​for items related to personal information. In the example shown in Figure 3, the character string "Name" corresponds to the item name, and the character string "aaaa" corresponds to the item value of the item corresponding to the item name "Name." The character string "Address" corresponds to the item name, and the character string "bbbb" corresponds to the item value of the item corresponding to the item name "Address." The character string "Date of Birth" corresponds to the item name, and the character string "cccc" corresponds to the item value of the item corresponding to the item name "Date of Birth."

[0036] When the ID card shown in Figure 3 is read, the image data obtained by reading the ID card will contain a character region containing the string "Name" indicating the item name and a character region containing the string "aaaa" indicating the item value. Also, a character region containing the string "Address" indicating the item name and a character region containing the string "bbbb" indicating the item value will appear. Also, a character region containing the string "Date of Birth" indicating the item name and a character region containing the string "cccc" indicating the item value will appear.

[0037] In the following explanation, when it is necessary to distinguish between multiple character areas, the character area for the item name "Name" is assigned the code C11, and the character area for the item value "aaaa" is assigned the code C12. The character area for the item name "Address" is assigned the code C21, and the character area for the item value "bbbb" is assigned the code C22. The character area for the item name "Date of Birth" is assigned the code C31, and the character area for the item value "cccc" is assigned the code C32.

[0038] After the multifunction device 100 reads the ID, an image based on the image data of the ID obtained by the reading (i.e., the personal information written on the ID) is printed on a sheet S. Then, the sheet S on which the personal information is printed is stored. In this manner, the personal information is registered. Note that the image data (i.e., electronic data) of the ID itself may also be stored.

[0039] Here, in some cases, it may be necessary to mask some personal information. For example, suppose an identification card has a field for date of birth, but the date of birth is not required in contracts for goods and services. In this example, it may be necessary to mask personal information related to the date of birth. Also, for example, in some countries, identification cards have a field for religion, but it is stipulated that personal information related to religion should not be recorded. In this example, it may be necessary to mask personal information related to religion.

[0040] For example, it has been common practice to manually black out personal information to conceal it, but manually masking some personal information is a hassle for users.

[0041] For this reason, a masking function is installed in the multifunction device 100. In other words, the multifunction device 100 is capable of executing a job related to the masking function (hereinafter simply referred to as a "masking job").

[0042] By using the masking function, it is possible to obtain masked data, which is image data in which some of the personal information on the ID card has been masked. An image based on the masked data can be printed on sheet S, or the masked data can be sent to PC 1000 and saved there.

[0043] When a masking job is executed, the masking job settings are configured, and the personal information to be masked can be specified in the masking job settings.

[0044] After setting the masking job, the user places an identification card as the document D in the image reading unit 2. In this state, the user performs a start operation for the masking job on the operation display unit 3. When the control unit 10 detects that the start operation has been performed on the operation display unit 3, it starts the masking job.

[0045] The flow of a masking job will be described below with reference to the flowchart shown in Fig. 4. The flow shown in Fig. 4 starts when the control unit 10 detects an operation to start a masking job on the operation and display unit 3.

[0046] Before starting a masking job, the user sets up the masking job. In setting up the masking job, target items, which are items corresponding to the item values ​​to be masked, are specified. When executing the masking job, the control unit 10 recognizes the target items specified by the user.

[0047] In step #1, the control unit 10 causes the image reading unit 2 to read an identification card as the document D. The image reading unit 2 reads the identification card and generates image data of the read identification card (i.e., scan data of the identification card). The control unit 10 acquires the image data of the identification card obtained by the image reading unit 2 reading the identification card.

[0048] In step #2, the control unit 10 performs a first region detection process using a first learning model obtained by machine learning. The control unit 10 performs the first region detection process by detecting a first region that is predicted to individually include either an item name region or an item value region from the image data of the identification card.

[0049] The first learning model for the first area detection process is a learning model that has been trained to detect the first area from image data obtained by reading an identification card using the image reading unit 2. The first learning model is a trained model and is stored in advance in the memory unit 101.

[0050] An identification card usually has multiple pairs of item names and item values. Therefore, multiple first areas are detected in the first area detection process. For this reason, if no first area is detected or only one first area is detected as a result of the first area detection process, the masking job may be stopped and a message may be displayed on the operation display unit 3 prompting the user to check the document D set in the image reading unit 2.

[0051] In the ID card shown in Figure 3, the character area C11 of the item name "Name," the character area C12 of the item value "aaaa," the character area C21 of the item name "Address," the character area C22 of the item value "bbbb," the character area C31 of the item name "Date of Birth," and the character area C32 of the item value "cccc" are each individually detected as the first area. The first areas are clearly shown in Figure 5.

[0052] In step #3, the control unit 10 performs a second region detection process using a second learning model obtained by machine learning. The control unit 10 performs the second region detection process by detecting a second region that is predicted to include both the area of ​​the item name and the area of ​​the item value that correspond to the same item from the image data of the identification card.

[0053] The second learning model for the second area detection process is a learning model that has been trained to detect the second area from image data obtained by reading an identification card using the image reading unit 2. The second learning model is a trained model and is stored in advance in the memory unit 101.

[0054] If an identification card has been read, at least one second area will be detected as a result of the second area detection process. If no second area is detected in the second area detection process, it is possible that something other than an identification card has been read. Therefore, if no second area is detected in the second area detection process, the masking job may be stopped, and a message prompting the user to check the document D set in the image reading unit 2 may be displayed on the operation display unit 3.

[0055] In the example shown in Fig. 5, areas C10, C20, and C30 shown in Fig. 6 are detected as second areas. Specifically, a single area C10 that includes a character area C11 for the item name "Name" and a character area C12 for the item value "aaaa" is detected as the second area. A single area C20 that includes a character area C21 for the item name "Address" and a character area C22 for the item value "bbbb" is detected as the second area. A single area C30 that includes a character area C31 for the item name "Date of Birth" and a character area C32 for the item value "cccc" is detected as the second area.

[0056] In step #4, control unit 10 performs inclusion region detection processing for each of the multiple first regions detected in the first region detection processing. Each of the multiple first regions is subjected to the inclusion region detection processing once. Control unit 10 performs the inclusion region detection processing on one of the first regions as the processing target, and when that inclusion region detection processing is complete, it performs the inclusion region detection processing on another first region that has not yet been processed as the new processing target.

[0057] The control unit 10 performs an inclusion region detection process to detect a second region that encompasses the first region to be processed. The control unit 10 detects a second region that encompasses the first region to be processed by detecting a second region having an image portion whose image similarity with the first region to be processed is equal to or greater than the threshold for the inclusion region detection process. The control unit 10 determines that the first region to be processed is encompassed by a second region having an image portion whose image similarity with the first region to be processed is equal to or greater than the threshold for the inclusion region detection process. Note that character recognition processing has not been performed at this point. In other words, the image similarity refers to the similarity between image data, not the similarity between character strings (i.e., between text data). The control unit 10 performs the inclusion region detection process using, for example, a known pattern matching technique. The threshold for the inclusion region detection process is not particularly limited, but is 70% or greater.

[0058] In the examples shown in FIGS. 5 and 6, the second region C10 has an image portion where the image similarity with the first region C11 (region with the item name "Name") is equal to or greater than a threshold, and also has an image portion where the image similarity with the first region C12 (region with the item value "aaaa") is equal to or greater than a threshold. The second region C20 has an image portion where the image similarity with the first region C21 (region with the item name "Address") is equal to or greater than a threshold, and also has an image portion where the image similarity with the first region C22 (region with the item value "bbbb") is equal to or greater than a threshold. The second region C30 has an image portion where the image similarity with the first region C31 (region with the item name "Date of Birth") is equal to or greater than a threshold, and also has a portion where the image similarity with the first region C32 (region with the item value "cccc") is equal to or greater than a threshold. As a result, the first regions C11 and C12 are determined to be included in the second region C10, the first regions C21 and C22 are determined to be included in the second region C20, and the first regions C31 and C32 are determined to be included in the second region C30.

[0059] In step #5, the control unit 10 performs a region linking process. Specifically, the control unit 10 links a plurality of first regions included in the same second region to each other.

[0060] 5 and 6, first regions C11 and C12 included in second region C10 are linked to each other. First regions C21 and C22 included in second region C20 are linked to each other. First regions C31 and C32 included in second region C30 are linked to each other.

[0061] In step #6, the control unit 10 performs an area determination process. By performing the area determination process, the control unit 10 recognizes which of the multiple first areas linked to each other is an item name area and which is an item value area. In other words, by performing the area determination process, the control unit 10 distinguishes between an item name area and an item value area. By performing the area determination process, the control unit 10 determines which of the multiple first areas linked to each other is an item name area, and recognizes the other first area linked to the one first area determined to be an item name area as an item value area.

[0062] 5 and 6, in the second area C10, the first area C11 is determined to be the area for item names, and the first area C12 is determined to be the area for item values. In the second area C20, the first area C21 is determined to be the area for item names, and the first area C22 is determined to be the area for item values. In the second area C30, the first area C31 is determined to be the area for item names, and the first area C32 is determined to be the area for item values.

[0063] The control unit 10 recognizes the area of ​​the target item by performing an area determination process. The area of ​​the target item is a second area that includes both a first area corresponding to the item name of the target item and a first area corresponding to the item value of the target item. The area determination process will be described in detail later. Note that the area of ​​the target item may be recognized using other methods.

[0064] In step #7, the control unit 10 performs a masking process to mask at least a portion of the area of ​​the target item present in the image data of the identification card. By performing the masking process, the control unit 10 generates masked data in which at least a portion of the area of ​​the target item present in the image data of the identification card is masked.

[0065] The control unit 10 masks the first area associated with the area of ​​the item name of the target item (i.e., the area of ​​the item value of the target item). Alternatively, the control unit 10 masks both the area of ​​the item name and the area of ​​the item value of the target item. Substantially the entire area of ​​the second area corresponding to the target item may be masked.

[0066] An example of masked data is shown in Figure 7. The masked data shown in Figure 7 is generated when the target item is date of birth. That is, at least the item value "cccc" corresponding to the item name "Date of Birth" is masked. Although not shown, both the item name "Date of Birth" and the item value "cccc" may be masked.

[0067] In step #8, the control unit 10 causes the output unit to perform output processing of the masked data. For example, in the masking job settings, the output method of the masked data can be selected. The output methods include printing and transmission.

[0068] When printing is selected as the output method, the control unit 10 causes the printing unit 1 to print (in other words, output) an image based on the masked data onto a sheet S. In this case, the printing unit 1 corresponds to the "output unit," and the output destination is the sheet S.

[0069] When transmission is selected as the output method, the control unit 10 causes the communication unit 102 to transmit (in other words, output) the masked data to the PC 1000. The masked data may be converted into PDF data and transmitted to the PC 1000. By transmitting the masked data to the PC 1000, the masked data can be saved in the PC 1000. In this case, the communication unit 102 corresponds to the "output unit," and the output destination is the PC 1000.

[0070] In this embodiment, a first region detection process, a second region detection process, an inclusion region detection process, and a region linking process are performed, which allows each region of the image data of the identification card that has an item name and an item value corresponding to the same item to be linked with high accuracy.

[0071] By linking the fields of the item name and the field value corresponding to the same item in the image data of the ID card to each other, it is possible to easily mask at least a portion of the target item. Specifically, it is sufficient to mask the field linked to the field of the item name of the target item. This allows the field value of the target item to be masked. With this configuration, it is possible to easily mask only the fields corresponding to the user-specified items in the image data obtained by scanning the document D (ID card) without transferring the image data of the ID card to a processing device on the cloud.

[0072] If the item value area in the image data of an ID card can be masked, the leakage of personal information can be suppressed. Even if the item name area of ​​a target item is masked, if the item value area of ​​the target item is not masked, personal information will be leaked. Therefore, it is important to accurately perform a process of linking each item name and item value area corresponding to the same item in the image data of an ID card.

[0073] In this embodiment, the first area detection process is performed using the first learning model, and the second area detection process is performed using the second learning model. Here, machine learning requires a huge amount of computation and a huge amount of memory capacity. For this reason, machine learning processes are often performed by a processing device on the cloud. However, this requires transferring image data of the identification card to the processing device on the cloud, which is undesirable because it may result in the leakage of personal information.

[0074] Therefore, in this embodiment, a trained first learning model and a trained second learning model are used. The first learning model and the second learning model are each stored in advance in the storage unit 101. This allows the first area detection process and the second area detection process to be performed within the multifunction device 100 without increasing the memory capacity within the multifunction device 100. In other words, there is no need to transfer image data of the identification card to a processing device on the cloud. As a result, leakage of personal information can be suppressed.

[0075] <Area determination process> The control unit 10 performs an area determination process to determine whether each of a plurality of first areas (i.e., character areas containing character strings) present in the image data of the identification card is an area for an item name or an area for an item value. The area determination process uses dictionary data DD. The dictionary data DD is stored in advance in the storage unit 101 (see FIG. 2).

[0076] A conceptual diagram of the dictionary data DD is shown in Figure 8. The dictionary data DD is data in which multiple different similar item names are predefined for one type of item. The dictionary data DD is created in advance for each of the multiple types of items. The storage unit 101 stores in advance multiple dictionary data DD corresponding to each of the multiple types of items. The multiple dictionary data DD are created in advance by the manufacturer of the multifunction device 100 and pre-stored in the storage unit 101. In Figure 8, items are indicated by A, B... Similar item names corresponding to item A are indicated by a1, a2, a3... Similar item names corresponding to item B are indicated by b1, b2, b3...

[0077] Taking an identification number as an example of an item, one ID card may use the character string "ID" as the item name, another ID card may use the character string "Number" as the item name, and yet another ID card may use the character string "Num" as the item name. Although the character strings listed here are different from one another, they all correspond to the same item. Therefore, in the dictionary data DD corresponding to the identification number as an item, the character string "ID," the character string "Number," and the character string "Num" are predefined as similar item names.

[0078] The flow of the area determination process will be described below with reference to the flowchart shown in Fig. 9. The flow shown in Fig. 9 starts when the area linking process by the control unit 10 is completed.

[0079] In step #11, the control unit 10 performs character recognition processing on the image data of the ID card obtained by the image reading unit 2 reading the ID card (i.e., the processing of step #1 shown in FIG. 4). The control unit 10 performs OCR processing on the image data of the ID card. As a result, the control unit 10 extracts character strings from each of the multiple first areas present in the image data of the ID card. The control unit 10 recognizes the character strings in each of the multiple first areas. Note that the first areas contain character strings indicating item names or character strings indicating item values.

[0080] In step #12, the control unit 10 performs preprocessing on the multiple character strings extracted by the character recognition process. The type of preprocessing is not particularly limited. For example, the control unit 10 performs preprocessing by unifying uppercase and lowercase letters in the character string. This process makes the character string indistinguishable from uppercase and lowercase letters. Furthermore, for example, the control unit 10 performs preprocessing by removing spaces and predetermined symbols.

[0081] After the processing of step #12, the control unit 10 performs an item name area determination process. When performing the item name area determination process, the control unit 10 sets one of the multiple first areas detected in the first area detection process as a processing target.

[0082] Then, the control unit 10 performs a similarity calculation process (the process of step #13) as one process of the item name region determination process. In step #13, the control unit 10 calculates the similarity between all similar item names (their character strings) defined in the multiple dictionary data DD and the substrings obtained by dividing the character string of the first region to be processed. Hereinafter, the similarity between the similar item names and the substrings of the first region will be referred to as character string similarity, and will be distinguished from image similarity.

[0083] When performing the similarity calculation process, the control unit 10 selects one of the plurality of dictionary data DD (here, the selected dictionary data DD is referred to as target dictionary data DD). The control unit 10 also selects one of the plurality of similar item names defined in the target dictionary data DD (here, the selected similar item name is referred to as target similar item name).

[0084] Then, the control unit 10 calculates the string similarity between the partial string of the first region to be processed and the target similar item name. The method for calculating the string similarity is not particularly limited. The string similarity may be calculated using the first method, or the second method. The string similarity may be calculated using a method different from the first and second methods. The maximum value of the string similarity is "1" and the minimum value is "0". The larger the value of the string similarity, the higher the similarity between the target strings.

[0085] In the similarity calculation process, when the target similar item name matches the partial character string of the first region to be processed, i.e., when the target similar item name is included in the character string of the first region to be processed, the calculated character string similarity is 1. On the other hand, when the target similar item name does not match the partial character string of the first region to be processed, i.e., when the target similar item name is not included in the character string of the first region to be processed, the calculated character string similarity is a value smaller than 1.

[0086] The first and second methods will be explained below, but for ease of understanding, the target similar item name is assumed to be "PC." The cases where the character string in the first region to be processed is "PCabc" and "PoCabc" will be explained.

[0087] 1. First Method In the first method, it is determined whether a part of the character string (i.e., a substring) in the first region to be processed matches the target similar item name. In other words, in the first method, it is determined whether the target similar item name is included in the character string in the first region to be processed. If the character string in the first region to be processed is "PCabc," the substrings would be, for example, "PC," "Ca," "PCa," and "abc." If the character string in the first region to be processed is "PoCabc," the substrings would be, for example, "Po," "oC," "PoC," and "Cabc." Note that the substrings listed here are merely examples.

[0088] The character string "PC" in the partial character string of the first region to be processed matches the character string "PC" in the target similar item name. However, character strings other than the character string "PC" in the partial character string of the first region to be processed do not match the character string "PC" in the target similar item name.

[0089] Therefore, if the character string in the first region to be processed is "PCabc", the subcharacter string in the first region to be processed matches the target similar item name. That is, the character string in the first region to be processed includes the target similar item name. On the other hand, if the character string in the first region to be processed is "PoCabc", the subcharacter string in the first region to be processed does not match the target similar item name. That is, the character string in the first region to be processed does not include the target similar item name.

[0090] In the similarity calculation process using the first method, if any substring in the first region to be processed matches the target similar item name, the string similarity is set to "1." On the other hand, if none of the substrings in the first region to be processed matches the target similar item name, the string similarity is set to "0."

[0091] As a result, in the similarity calculation process using the first method, if the character string in the first region to be processed is "PCabc," the character string similarity will be "1." On the other hand, if the character string in the first region to be processed is "PoCabc," the character string similarity will be "0."

[0092] 2. Second Method In the second method, each character in the string of the first region to be processed is set as a starting character, and the string of characters from the starting character in the string of the first region to be processed, which is the number of characters in the target similar item name, is set as a substring. Furthermore, the number of matching characters N1, which is the number of characters that match between the target similar item name and the substring, is counted. The value obtained by dividing the number of matching characters N1 by the number of characters N2 in the target similar item name (=N1 / N2) is the string similarity.

[0093] In the similarity calculation process using the second method, if the character string in the first region to be processed is "PCabc," the characters "P," "C," "a," and "b" are the starting characters, starting from the first character. The number of characters in the target similar item name "PC" is 2 (=N2). Therefore, the character strings "PC," "Ca," "ab," and "bc" are the substrings.

[0094] Note that the character string "PCabc" in the first area to be processed does not have two characters starting with the character "c," so the character "c" does not become the starting character. If the number of characters in the target similar item name is "3," then in addition to the character "c," the character "b" also does not become the starting character.

[0095] When focusing on the substring "PC" in the first region to be processed, the number of matching characters between it and the target similar item name "PC" is "2 (= N1)". In this case, the string similarity is "1 (= 2 / 2)". When focusing on the substring "Ca" in the first region to be processed, the number of matching characters between it and the target similar item name "PC" is "1 (= N1)". In this case, the string similarity is "0.5 (= 1 / 2)". When focusing on the substrings "ab" and "bc" in the first region to be processed, the number of matching characters between them and the target similar item name "PC" is "0 (= N1)". In this case, the string similarity is "0 (= 0 / 2)". As a result, if the string in the first region to be processed is "PCabc", the string similarity between the substring in the first region to be processed and the target similar item name is "1".

[0096] In the similarity calculation process using the second method, if the character string in the first region to be processed is "PoCabc," the characters "P," "o," "C," "a," and "b" are the starting characters, starting from the first character. The number of characters in the target similar item name "PC" is 2 (=N2). Therefore, the character strings "Po," "oC," "Ca," "ab," and "bc" are the substrings.

[0097] Focusing on the substrings "Po," "oC," and "Ca" in the first region to be processed, the number of matching characters between them and the target similar item name "PC" is "1 (=N1)." In this case, the string similarity is "0.5 (=1 / 2)." Focusing on the substrings "ab" and "bc" in the first region to be processed, the number of matching characters between them and the target similar item name "PC" is "0 (=N1)." In this case, the string similarity is "0 (=0 / 2)." As a result, if the string in the first region to be processed is "PoCabc," the string similarity between the substrings in the first region to be processed and the target similar item name is "0.5."

[0098] After determining the string similarity between the partial character string in the first region to be processed and the target similar item name, the control unit 10 newly selects an unselected similar item name from among the multiple similar item names defined in the target dictionary data DD. The control unit 10 switches the target similar item name and determines the string similarity between the partial character string in the first region to be processed and the new target similar item name. The control unit 10 determines the string similarity between the partial character string in the first region to be processed and all of the multiple similar item names defined in the target dictionary data DD. The control unit 10 then stores the similar item name that has the highest string similarity with the partial character string in the first region to be processed from among the multiple similar item names defined in the target dictionary data DD, in association with the value of the string similarity.

[0099] The control unit 10 also calculates the string similarity between all similar item names defined in the multiple dictionary data DD and the partial string in the first domain to be processed. That is, the control unit 10 newly selects an unselected dictionary data DD from the multiple dictionary data DD. The control unit 10 switches the target dictionary data DD and calculates the string similarity between all similar item names defined in the new target dictionary data DD and the partial string in the first domain to be processed. The control unit 10 then stores the similar item name, among the multiple similar item names defined in the target dictionary data DD, that has the highest string similarity with the partial string in the first domain to be processed, in association with its string similarity value. As a result, multiple similar item names corresponding to multiple types of items are stored, each associated with a string similarity value. In other words, multiple similarity values ​​(the similarity values ​​are string similarity values) corresponding to multiple types of items are stored.

[0100] The control unit 10 performs a maximum value detection process (the process of step #14) as one process of the item name area determination process. In step #14, the control unit 10 compares the multiple similarity values ​​(the similarity values ​​are values ​​of character string similarity) calculated and stored in the similarity calculation process with each other. In other words, the control unit 10 compares the multiple similarity values ​​corresponding to the multiple types of items with each other.

[0101] Then, the control unit 10 detects the highest value of the character string similarity (i.e., the highest similarity value). That is, the control unit 10 calculates the character string similarity between the partial character string of the first region to be processed and all similar item names, and detects the highest value among the calculated similarity values.

[0102] The control unit 10 performs a threshold comparison process (processing of step #15) as part of the item name region determination process. In step #15, the control unit 10 compares a predetermined threshold for the item name region determination process with the highest value of the character string similarity.

[0103] For example, to set a threshold value for the item name area determination process, an identification card is actually read by the multifunction device 100, and the result of a similarity calculation process for character strings present in the item name area of ​​the image data obtained by the read (here, the resulting value is referred to as the similarity value of the item name area) is obtained. Also, the result of a similarity calculation process for character strings present in an area of ​​the image data other than the item name area (here, the resulting value is referred to as the similarity value of the other area) is obtained. Note that multiple similarity values ​​for the item name area and multiple similarity values ​​for the other areas are obtained.

[0104] Furthermore, a first average value, which is the average value of the similarity values ​​of the plurality of item name regions, is calculated, and a second average value, which is the average value of the plurality of other regions, is calculated. The average value of the first average value and the second average value is then set as the threshold value for the item name region determination process. Note that the method for setting the threshold value for the item name region determination process is not particularly limited, and other setting methods may be used.

[0105] In step #16, the control unit 10 determines whether the highest value of the character string similarity is equal to or greater than the threshold value for the item name area determination process. In Fig. 9, the threshold value for the item name area determination process is represented as Th. If the control unit 10 determines that the highest value of the character string similarity is equal to or greater than the threshold value for the item name area determination process, the process proceeds to step #17.

[0106] In step #17, the control unit 10 determines that the first region to be processed is an area of ​​an item name. That is, the control unit 10 calculates the string similarity between all similar item names defined in the multiple dictionary data DD and the substrings obtained by dividing the string in the first region to be processed. Then, if the string similarity between any of the similar item names and the substrings in the first region to be processed is equal to or greater than a threshold, the control unit 10 determines that the first region to be processed is an area of ​​an item name.

[0107] In step #18, the control unit 10 determines the item corresponding to the first region to be processed. Specifically, the control unit 10 determines, as a candidate item, the item corresponding to the similar item name that has the highest string similarity with the partial string of the first region to be processed among all similar item names defined in the multiple dictionary data DD. The control unit 10 then determines that the candidate item is the item corresponding to the first region to be processed.

[0108] In step #16, if the control unit 10 determines that the highest value of the character string similarity is less than the threshold value for the item name area determination process, the process proceeds to step #19. If the process proceeds to step #19, the control unit 10 determines that the first area to be processed is another area (i.e., an area different from the item name area).

[0109] Each of the processes from step #13 to step #19 is performed as one process of the item name area determination process. After completing the item name area determination process for a certain first area to be processed, the control unit 10 determines whether or not there are any first areas remaining for which the item name area determination process has not been performed (here referred to as unprocessed first areas). If there are any unprocessed first areas remaining, the control unit 10 sets the unprocessed first area as a new processing object. In other words, the control unit 10 switches the processing object. Then, the control unit 10 performs the item name area determination process (each of the processes from step #13 to step #19) for the new first area to be processed.

[0110] In this embodiment, dictionary data DD, in which a plurality of different similar item names are predefined for one type of item, is stored in advance in the storage unit 101 for each of a plurality of types of items. Then, the item name area determination process is performed using the dictionary data DD. This prevents an inconvenience that occurs when an item having a plurality of corresponding item names is to be masked, where the masked item is properly masked on an identification card on which one of the plurality of item names is written, but the masked item is not masked on an identification card on which another item name is written (in other words, the masked item is not detected).

[0111] For example, suppose an identification number is an item to be masked. In this case, in this embodiment, the identification number is correctly masked on both an identification card with "ID" written as the item name and an identification card with "Num" written as the item name. On the other hand, in a conventional configuration, the identification number is correctly masked on an identification card with "ID" written as the item name, but the identification number is not masked on an identification card with "Num" written as the item name, which can be an inconvenience.

[0112] To prevent these problems, a large dictionary is required. Furthermore, when using machine learning models, the model size becomes large. However, because memory capacity is limited, processing is performed on the cloud.

[0113] In this embodiment, by using dictionary data DD, processing on the cloud is not required. In other words, there is no need to transfer image data of the identification card (i.e., personal information) to a processing device on the cloud. This makes it possible to prevent the leakage of personal information. In other words, in this embodiment, it is possible to easily mask only the areas corresponding to the items specified by the user from the image data obtained by scanning the document D (identification card) without transferring the image data of the identification card to a processing device on the cloud.

[0114] In this embodiment, the similarity calculation process (i.e., the process of step #13 shown in FIG. 9) involves calculating the character string similarity between the partial character strings obtained by dividing the character string in the first region to be processed and the similar item name. This provides the following effects.

[0115] For example, on an identification card such as a passport, the name of a single item is written in two or more languages. Therefore, in the first region detection process of a masking job targeting the identification card (i.e., the process of step #2 shown in FIG. 4), a region of character strings containing two or more languages ​​is detected as the first region. For example, a region of character strings containing Japanese character strings and English character strings is detected as the first region.

[0116] In this case, if the character string similarity to the similar item name is calculated for the entire character string in the first region, the value of the character string similarity will be small even if the character string in the first region indicates a similar item name, and as a result, the item name region determination process cannot be performed accurately.

[0117] On the other hand, in this embodiment, a character string consisting only of Japanese characters is used as a substring, and the character string similarity between the Japanese substring and a similar item name is calculated. Alternatively, a character string consisting only of English characters is used as a substring, and the character string similarity between the English substring and a similar item name is calculated. This improves the accuracy of calculating the character string similarity. In other words, the item name region determination process can be performed with high accuracy.

[0118] This allows for accurate detection of areas corresponding to user-specified items. As a result, areas corresponding to user-specified items can be accurately masked. That is, it is possible to prevent the problem of areas corresponding to user-specified items not being masked. It is also possible to prevent the problem of areas other than areas corresponding to user-specified items being masked.

[0119] <Exception handling> When the first region to be processed is determined to be an area of ​​an item name, the control unit 10 selects, as a candidate item, an item corresponding to a similar item name that has the highest string similarity with the partial string of the first region to be processed among all similar item names defined in the multiple dictionary data DD. If there is one candidate item, the control unit 10 determines that this candidate item is the item corresponding to the first region to be processed. This makes it easy to determine the item corresponding to the first region to be processed.

[0120] However, in some cases, multiple candidate items may appear. For example, suppose that the string "address" is defined as a similar item name to item A, and the string "dd" is defined as a similar item name to item B, which is different from item A. Also, suppose that the ID card that is the target of the masking job has the string "cardholder address" written as an item name.

[0121] When such an ID card is the target of a masking job, the area containing the character string "cardholder address" becomes the first area. When calculating the string similarity between the similar item name "address" of item A and the partial string of the first area, at least the character string "address" of the character string "cardholder address" becomes a partial string, so the obtained string similarity is "1." When calculating the string similarity between the similar item name "dd" of item B and the partial string of the first area, at least the character string "dd" of the character string "cardholder address" becomes a partial string, so the obtained string similarity is "1." Therefore, in this example, there are two candidate items.

[0122] In this way, when there are multiple candidate items, the control unit 10 performs exception processing. By performing exception processing, the control unit 10 narrows down the items corresponding to the first area to be processed to one.

[0123] The flow of exception handling will be described below with reference to the flow shown in Fig. 10. Exception handling is performed as part of the item determination process performed in step #18 shown in Fig. 9.

[0124] In step #21, the control unit 10 determines whether there are multiple candidate items. If the control unit 10 determines that there is only one candidate item, the process proceeds to step #22. In step #22, the control unit 10 determines that the candidate item is an item that corresponds to the first area to be processed.

[0125] If the control unit 10 determines in step #21 that there are multiple candidate items, the process proceeds to step #23. In step #23, the control unit 10 recognizes the priority (in other words, the importance) of each of the multiple candidate items based on priority data PD that defines the priority of each of the multiple items. The priority data PD is predetermined by the manufacturer of the multifunction peripheral 100 and is stored in the storage unit 101 (see FIG. 2). When recognizing the priority of each of the multiple candidate items, the control unit 10 refers to the priority data PD.

[0126] Thereafter, the process proceeds to step #24. In step #24, the control unit 10 determines that, of the multiple candidate items, the candidate item with the higher priority defined by the priority data PD is the item corresponding to the first area to be processed.

[0127] Here, the importance of personal information varies depending on the type of personal information. For items with a high degree of importance (herein referred to as high importance items), it is necessary to reliably prevent information leakage.

[0128] However, there are cases where a region other than the region of the high importance item is mistakenly determined to be the region of the high importance item, and there are also cases where a region of the high importance item is mistakenly determined to be the other region.

[0129] If an area other than the area of ​​the high importance item is mistakenly determined to be the area of ​​the high importance item, the area other than the area of ​​the high importance item may be unnecessarily masked. This unnecessary masking is not a problem from the viewpoint of information protection. On the other hand, if the area of ​​the high importance item is mistakenly determined to be the other area, an inconvenience may occur in that the area of ​​the high importance item is not masked even though it is a target for masking. In other words, highly important personal information may be leaked.

[0130] Therefore, the priority of each of the multiple items is determined based on the importance of each of the multiple items. As a result, when there are multiple candidate items, the candidate item for which prevention of information leakage is more important is determined to be the item corresponding to the first area to be processed. As a result, it is possible to prevent the leakage of personal information with high importance.

[0131] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims rather than the description of the above embodiments, and further includes all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0132] 1 Printing unit (output unit) 2 Image reading unit 10 Control Unit 100 Multifunction printers (image processing devices) 101 Storage section 102 Communication unit (output unit) 1000 PCs (external equipment) D Manuscript DD dictionary data S seat

Claims

1. an image reading unit that reads a document on which a plurality of sets of item names and item values ​​of items related to personal information are written; a control unit that recognizes a target item that is an item to be masked, and performs a masking process on image data obtained by reading the document by the image reading unit, thereby generating masked data in which at least a portion of an area of ​​the target item present in the image data is masked; a storage unit, the storage unit stores dictionary data in which a plurality of different similar item names are predefined for one type of the item, for each of the plurality of types of the items; The control unit a first region detection process for detecting a first region that is predicted to individually include either an area of ​​the item name or an area of ​​the item value from the image data; a character recognition process for recognizing a character string in the first region; an item name area determination process, which is a process performed for each of the plurality of first areas, in which for all the similar item names, a string similarity is calculated between the string of the first area being processed and a substring obtained by dividing the string, and if the string similarity between any of the similar item names and the substring is equal to or greater than a threshold, the first area being processed is determined to be an area of ​​the item name.

2. When the first area to be processed is determined to be an area of ​​the item name, the control unit sets the item corresponding to the similar item name having the highest character string similarity with the partial character string of the first area to be processed as a candidate item, The image processing device according to claim 1 , wherein when there is one candidate item, the control unit determines that the candidate item is the item corresponding to the first region to be processed.

3. 3. The image processing device according to claim 2, wherein when there are multiple candidate items, the control unit determines that the candidate item having a higher predetermined priority among the multiple candidate items is the item corresponding to the first area to be processed.

4. The control unit a second region detection process for detecting a second region from the image data that is predicted to include regions of both the item name and the item value corresponding to the same item; performing a region linking process for linking the plurality of first regions included in the same second region to each other; The image processing device according to claim 1 , wherein the control unit recognizes one of the first areas, which is determined to be the area of ​​the item name, as the area of ​​the item value, the other of the first areas linked to the one of the first areas.

5. an output unit that performs output processing of the masked data, 2. The image processing device according to claim 1, wherein the output unit is at least one of a printing unit that performs the output process of printing an image based on the masked data onto a sheet, and a communication unit that performs the output process of transmitting the masked data to an external device.

Images