Image processing device

Abstract

An image processing device (100) comprises an image reading unit (2), a control unit (10), and an operation display unit (3). The operation display unit (3) receives an item designation operation. The control unit (10) recognizes an item designated through the item designation operation as a target item, and performs, as masking processing, processing for masking at least an region of an item value from a region of the target item present in image data to generate masked data.

Description

Image Processing Device

[0001] The present invention relates to an image processing device.

[0002] Conventionally, there has been known an image processing device that reads an identification card as a document and masks an area of ​​personal information in the image data obtained by the reading. Such an image processing device is disclosed, for example, in Japanese Patent Application Laid-Open No. 2003-222294.

[0003] Japanese Patent Application Laid-Open No. 2021-197682

[0004] For example, personal information written on an ID card is classified into multiple categories, such as name and address. With such an ID card, it may be desirable to mask only the areas of the image data obtained by scanning that correspond to specific categories. Note that the areas that need to be masked vary from user to user.

[0005] If only a specific type of ID card is to be targeted, the area to be masked (in other words, the position within the image data of the ID card) can be set in advance. As a result, if the scanned ID card is of the specific type, only the area of ​​the image data of the ID card that corresponds to the specified item will be masked.

[0006] However, when scanning and masking a type of identification card different from the predetermined type, it is necessary to set the area to be masked each time, which is troublesome and inconvenient for the user.

[0007] The present invention has been made to solve the above-mentioned problems, and aims to provide an image processing device that can easily mask only the areas of image data obtained by reading a document that correspond to items specified by the user.

[0008] To achieve the above object, an image processing device according to a first aspect of the present invention includes an image reading unit that reads a document containing multiple sets of item names and item values ​​for items related to personal information, a control unit that recognizes target items that are to be masked and performs a masking process on image data obtained by reading the document with the image reading unit to generate masked data in which at least a portion of the area of ​​the target items present in the image data is masked, and an operation display unit that displays information and accepts operations. The operation display unit accepts an item designation operation to designate one of the items. The control unit recognizes the item designated by the item designation operation as the target item and performs the masking process by masking at least the area of ​​the item value of the area of ​​the target item present in the image data to generate masked data.

[0009] According to a second aspect of the present invention, an image processing device includes an image reading unit that reads a document containing multiple sets of item names and item values ​​for personal information items, and a control unit that recognizes target items that are to be masked and performs a masking process on image data obtained by reading the document with the image reading unit, thereby generating masked data in which at least a portion of the target item area present in the image data is masked. The control unit performs a first area detection process that detects, from the image data, a first area that is predicted to include either an item name area or an item value area, a second area detection process that detects, from the image data, a second area that is predicted to include both an item name area and an item value area corresponding to the same item, an inclusion area detection process that is performed for each of the multiple first areas, and that detects a second area that includes the first area to be processed by detecting a second area having an image portion whose image similarity with the first area to be processed is equal to or greater than a threshold, and an area linking process that links multiple first areas that are included in the same second area to each other.

[0010] According to a third aspect of the present invention, an image processing device includes an image reading unit that reads a document containing multiple sets of item names and item values ​​for personal information items; a control unit that recognizes target items that are to be masked and performs a masking process on image data obtained by reading the document with the image reading unit, thereby generating masked data in which at least a portion of the target item area present in the image data is masked; and a storage unit. The storage unit stores dictionary data in which multiple similar item names are predefined for each type of item, for each type of item. The control unit performs a first area detection process that detects, from the image data, first areas that are predicted to individually include either an item name area or an item value area; a character recognition process that recognizes character strings in the first areas; and an item name area determination process that is performed for each of the multiple first areas, in which character string similarities between the character string in the target first area and each of the character strings of all similar item names are calculated and, if the highest calculated character string similarity is equal to or greater than a threshold, the target first area is determined to be an item name area.

[0011] In the configuration of the present invention, it is possible to easily mask only the areas of image data obtained by scanning an original that correspond to items designated by the user.

[0012] FIG. 1 is a schematic diagram of a multifunction peripheral of an embodiment; FIG. 2 is a block diagram of the multifunction peripheral of an embodiment; FIG. 3 is a diagram schematically illustrating an example of an identification card that can be read by the multifunction peripheral of an embodiment; FIG. 4 is a flowchart illustrating the flow of a masking job executed by the multifunction peripheral of an embodiment; FIG. 5 is a diagram illustrating a first region within image data obtained by reading an identification card by the multifunction peripheral of an embodiment; FIG. 6 is a diagram illustrating a second region within image data obtained by reading an identification card by the multifunction peripheral of an embodiment; FIG. 7 is a diagram illustrating masked data generated in a masking job by the multifunction peripheral of an embodiment; FIG. 8 is a conceptual diagram of dictionary data used in a masking job by the multifunction peripheral of an embodiment; FIG. 9 is a flowchart illustrating the flow of a region determination process executed by the multifunction peripheral of an embodiment; FIG. 10 is a flowchart illustrating the flow of a first abnormality countermeasure process executed by the multifunction peripheral of an embodiment; FIG. 11 is a flowchart illustrating the flow of a second and third abnormality countermeasure process executed by the multifunction peripheral of an embodiment; FIG. 12 is a flowchart illustrating the flow of a second abnormality countermeasure process executed by the multifunction peripheral of an embodiment;

[0013] <Configuration of Multifunction Device> An image processing apparatus according to an embodiment of the present invention will be described below with reference to FIGS. 1 to 16, taking as an example a multifunction device 100 having multiple functions such as a scanning function, a printing function, and a transmission function.

[0014] As shown in FIG. 1, the multifunction device 100 (corresponding to an "image processing device") includes a printing unit 1. The printing unit 1 constitutes the main body of the multifunction device 100. The printing unit 1 prints an image on a sheet S. The printing method of the printing unit 1 is an electrophotographic method. However, the printing method of the printing unit 1 is not limited to this. The printing method of the printing unit 1 may also be an inkjet method.

[0015] The printing unit 1 forms an image based on image data input to the multifunction device 100. The printing unit 1 also transports a sheet S along a sheet transport path. The printing unit 1 prints an image on the sheet S while it is being transported. In Fig. 1, the sheet transport path is indicated by a dashed line.

[0016] The printing unit 1 includes a paper feed roller 11. The paper feed roller 11 comes into contact with the sheet S stored in the sheet cassette CA and rotates in this state, thereby feeding the sheet S from the sheet cassette CA to a sheet transport path.

[0017] The printing unit 1 includes an image forming unit 12. The image forming unit 12 includes a photosensitive drum 12a and a transfer roller 12b. The photosensitive drum 12a carries a toner image on its peripheral surface. The transfer roller 12b is in pressure contact with the photosensitive drum 12a, forming a transfer nip between the photosensitive drum 12a and the transfer roller 12b. The transfer roller 12b rotates together with the photosensitive drum 12a. The image forming unit 12 transfers the toner image onto the sheet S while transporting the sheet S that has entered the transfer nip.

[0018] Although not shown, the image forming unit 12 further includes a charging device, an exposure device, and a developing device. The charging device charges the circumferential surface of the photosensitive drum 12a. The exposure device forms an electrostatic latent image on the circumferential surface of the photosensitive drum 12a. The developing device develops the electrostatic latent image on the circumferential surface of the photosensitive drum 12a into a toner image.

[0019] The printing unit 1 includes a fixing unit 13. The fixing unit 13 includes a heating roller 13a and a pressure roller 13b. The heating roller 13a has a built-in heater (not shown). The pressure roller 13b is pressed against the heating roller 13a, forming a fixing nip between the heating roller 13a and the pressure roller 13b. The pressure roller 13b rotates together with the heating roller 13a. The fixing unit 13 fixes the toner image transferred onto the sheet S to the sheet S while transporting the sheet S that has entered the fixing nip. The sheet S that has exited the fixing nip is discharged onto an ejection tray ET.

[0020] The multifunction device 100 also includes an image reading unit 2. The image reading unit 2 is disposed on the top of the main body of the multifunction device 100. In a job involving reading an original D, the original D is set in the image reading unit 2. The image reading unit 2 reads the original D set in the image reading unit 2 and generates image data of the read original D.

[0021] The image reading unit 2 includes contact glasses G1 and G2. The contact glasses G1 and G2 are installed in a housing RH of the image reading unit 2. The housing RH has an opening on the top surface. The contact glasses G1 and G2 are attached to the opening on the top surface of the housing RH.

[0022] The image reading unit 2 includes a document transport device DP. The document transport device DP is attached to the housing RH. When viewed from the front of the multifunction device 100, the document transport device DP pivots around its rear portion as if swinging its front portion up and down. The document transport device DP opens and closes relative to the top surface of the housing RH.

[0023] The document transport device DP has a set tray ST on which the document D is set. The document transport device DP transports the document D set on the set tray ST onto the contact glass G1.

[0024] In the conveying reading mode, the user sets the document D on the set tray ST. Then, the document D is automatically conveyed onto the contact glass G1 by the document conveying device DP (in other words, the document D passing over the contact glass G1) and is read. On the other hand, in the placing reading mode, the user sets the document D on the contact glass G2, and the document D on the contact glass G2 is read.

[0025] The image reading unit 2 includes a light source 21, an image sensor 22, a mirror 23, and a lens 24. The light source 21, the image sensor 22, the mirror 23, and the lens 24 are installed inside a housing RH. The image reading unit 2 performs a scanning operation in which light is irradiated from the light source 21 toward the contact glass G1 or G2 and photoelectrically converted by the image sensor 22.

[0026] The light source 21 has a plurality of LED elements. The plurality of LED elements are arranged in a line in the main scanning direction (a direction perpendicular to the plane of FIG. 1 ). The image sensor 22 has a plurality of photoelectric conversion elements arranged in the main scanning direction. The mirror 23 reflects light toward the lens 24. The lens 24 collects the light reflected by the mirror 23 and guides it to the image sensor 22.

[0027] The light source 21 and the mirror 23 are mounted on a carriage 25 that is movable in a sub-scanning direction (left and right direction in FIG. 1 ) that is perpendicular to the main scanning direction. As the carriage 25 moves in the sub-scanning direction, the reading line of the image reading unit 2 moves in the sub-scanning direction.

[0028] As shown in FIG. 2 , the multifunction device 100 also includes an operation display unit 3. The operation display unit 3 is an operation panel having a touch screen. The operation display unit 3 displays software buttons, messages, and the like on the touch screen. The operation display unit 3 is also provided with a plurality of hardware buttons. The operation display unit 3 accepts operations from the user. The user can configure the multifunction device 100 via the operation display unit 3 with respect to various jobs, such as a masking job, which will be described later.

[0029] The multifunction device 100 includes a control unit 10. The control unit 10 includes a CPU, an ASIC, a memory, and the like. The control unit 10 also includes an image processing circuit. The control unit 10 performs various image processing operations on image data. The control unit 10 also controls the printing of an image on a sheet S by the printing unit 1 and the reading of an original D by the image reading unit 2.

[0030] The control unit 10 also controls the operation and display unit 3. Specifically, the control unit 10 controls the display operation of the touch screen. The control unit 10 detects operations on software buttons and hardware buttons. The control unit 10 sets jobs based on operations received by the operation and display unit 3 from the user.

[0031] The multifunction peripheral 100 includes a storage unit 101. The storage unit 101 is a non-volatile storage device. An HDD, an SSD, or the like may be used as the storage unit 101. The storage unit 101 is connected to the control unit 10. The control unit 10 writes information to the storage unit 101 and reads information from the storage unit 101.

[0032] The storage unit 101 stores predetermined information in advance. For example, the storage unit 101 stores a character recognition program in advance. The control unit 10 performs character recognition processing such as OCR (Optical Character Recognition) processing based on the character recognition program. The control unit 10 subjects the image data obtained by reading the document D with the image reading unit 2 to the character recognition processing.

[0033] The multifunction device 100 includes a communication unit 102. The communication unit 102 is an interface for communicatively connecting an external device to the multifunction device 100. The communication unit 102 includes a communication circuit, a communication memory, a communication connector, and the like. The communication unit 102 is connected to the control unit 10. The control unit 10 uses the communication unit 102 to send and receive data to and from the external device.

[0034] The communication unit 102 is communicably connected to external devices via a network NT such as a LAN or the Internet. Although not shown, the communication unit 102 may be directly connected to external devices via a communication cable. An example of an external device connected to the communication unit 102 is a personal computer 1000 (hereinafter referred to as PC 1000) used by a user of the multifunction device 100. An external device other than the PC 1000 may be communicably connected to the multifunction device 100. By connecting the PC 1000 to the multifunction device 100, image data of the original D obtained by scanning the original D with the image scanning unit 2 can be transmitted to the PC 1000. This allows the image data of the original D to be stored in the PC 1000.

[0035] <Masking Job Overview> When entering into a contract for a product or service, the person wishing to enter into the contract may be required to register personal information. This is just one example, and there are many other cases where personal information must be registered. Examples of personal information include name, address, and date of birth.

[0036] The user of the multifunction device 100 registers personal information. When registering personal information, the user uses the scanning function of the multifunction device 100 to scan their identification card. There are various types of identification cards that can be used, and these vary depending on the country and field. Examples of identification cards include a driver's license, insurance card, student ID card, and passport.

[0037] An example of an identification card is shown schematically in FIG. 3. Personal information is classified into multiple items and written on the identification card. That is, the identification card lists multiple sets of item names and item values ​​for items related to personal information. In the example shown in FIG. 3, the character string "Name" corresponds to the item name, and the character string "aaaa" corresponds to the item value of the item corresponding to the item name "Name". The character string "Address" corresponds to the item name, and the character string "bbbb" corresponds to the item value of the item corresponding to the item name "Address". The character string "Date of Birth" corresponds to the item name, and the character string "cccc" corresponds to the item value of the item corresponding to the item name "Date of Birth".

[0038] When the identification card shown in Figure 3 is read, the image data obtained by reading the identification card includes a character area including the character string "Name" indicating the item name and a character area including the character string "aaaa" indicating the item value. A character area including the character string "Address" indicating the item name and a character area including the character string "bbbb" indicating the item value also appears. A character area including the character string "Date of Birth" indicating the item name and a character string including the character string "cccc" indicating the item value also appears.

[0039] In the following description, when it is necessary to distinguish between multiple character areas, the character area of ​​the item name "Name" is assigned the code C11, and the character area of ​​the item value "aaaa" is assigned the code C12. The character area of ​​the item name "Address" is assigned the code C21, and the character area of ​​the item value "bbbb" is assigned the code C22. The character area of ​​the item name "Date of Birth" is assigned the code C31, and the character area of ​​the item value "cccc" is assigned the code C32.

[0040] After the multifunction device 100 reads the ID, an image based on the image data of the ID obtained by the reading (i.e., the personal information written on the ID) is printed on a sheet S. The sheet S with the printed personal information is then stored. In this manner, the personal information is registered. Note that the image data (i.e., electronic data) of the ID itself may also be stored.

[0041] Here, in some cases, it may be necessary to mask some personal information. For example, suppose an identification card has a field for date of birth, but the date of birth is not required in contracts for goods and services. In this example, it may be necessary to mask personal information related to the date of birth. Also, for example, in some countries, identification cards have a field for religion, but it is stipulated that personal information related to religion should not be recorded. In this example, it may be necessary to mask personal information related to religion.

[0042] For example, it has been common practice to manually black out personal information to conceal it, but manually masking some personal information is a hassle for users.

[0043] For this reason, a masking function is installed in the multifunction device 100. In other words, the multifunction device 100 is capable of executing a job related to the masking function (hereinafter simply referred to as a "masking job").

[0044] By using the masking function, it is possible to obtain masked data, which is image data in which some personal information on the identification card has been masked. An image based on the masked data can be printed on a sheet S, or the masked data can be sent to the PC 1000 and saved in the PC 1000.

[0045] When executing a masking job, the masking job is configured. In the masking job configuration, personal information to be masked can be arbitrarily specified. The masking job configuration will be explained in detail later.

[0046] After setting the masking job, the user places an identification card as the document D in the image reading unit 2. In this state, the user performs a start operation for the masking job on the operation display unit 3. When the control unit 10 detects that the start operation has been performed on the operation display unit 3, it starts the masking job.

[0047] The flow of a masking job will be described below with reference to the flowchart shown in Fig. 4. The flow shown in Fig. 4 starts when the control unit 10 detects a start operation of a masking job on the operation and display unit 3.

[0048] Before starting a masking job, the user configures the masking job. In the masking job configuration, target items are specified, which are items corresponding to the item values ​​to be masked. When executing the masking job, the control unit 10 recognizes the target items specified by the user.

[0049] In step #1, the control unit 10 causes the image reading unit 2 to read an identification card as the document D. The image reading unit 2 reads the identification card and generates image data of the read identification card (i.e., scanned data of the identification card). The control unit 10 acquires the image data of the identification card obtained by the image reading unit 2 reading the identification card.

[0050] In step #2, the control unit 10 performs a first region detection process using the first learning model obtained by machine learning. The control unit 10 performs the first region detection process by detecting, from the image data of the identification card, a first region that is predicted to individually include either an item name region or an item value region.

[0051] The first learning model for the first area detection process is a learning model that has been trained to detect the first area from image data obtained by reading an identification card using the image reading unit 2. The first learning model is a trained model and is stored in advance in the memory unit 101.

[0052] An identification card usually has multiple pairs of item names and item values. Therefore, multiple first areas are detected in the first area detection process. For this reason, if no first areas are detected or only one first area is detected as a result of the first area detection process, the masking job may be stopped and a message prompting the user to check the document D set in the image reading unit 2 may be displayed on the operation display unit 3.

[0053] In the identification card shown in Fig. 3, the character area C11 of the item name "Name," the character area C12 of the item value "aaaa," the character area C21 of the item name "Address," the character area C22 of the item value "bbbb," the character area C31 of the item name "Date of Birth," and the character area C32 of the item value "cccc" are each individually detected as the first area. The first areas are clearly shown in Fig. 5.

[0054] In step #3, the control unit 10 performs a second region detection process using the second learning model obtained by machine learning. The control unit 10 performs the second region detection process by detecting, from the image data of the identification card, a second region that is predicted to include both the region of the item name and the region of the item value that correspond to the same item.

[0055] The second learning model for the second area detection process is a learning model that has been trained to detect the second area from image data obtained by reading an identification card using the image reading unit 2. The second learning model is a trained model and is stored in advance in the memory unit 101.

[0056] If an identification card has been read, at least one second area will be detected as a result of the second area detection process. If no second area is detected in the second area detection process, it is possible that something other than an identification card has been read. Therefore, if no second area is detected in the second area detection process, the masking job may be stopped, and a message prompting the user to check the document D set in the image reading unit 2 may be displayed on the operation display unit 3.

[0057] In the example shown in Fig. 5, areas C10, C20, and C30 shown in Fig. 6 are detected as second areas. Specifically, a single area C10 including a character area C11 of the item name "Name" and a character area C12 of the item value "aaaa" is detected as the second area. A single area C20 including a character area C21 of the item name "Address" and a character area C22 of the item value "bbbb" is detected as the second area. A single area C30 including a character area C31 of the item name "Date of Birth" and a character area C32 of the item value "cccc" is detected as the second area.

[0058] In step #4, control unit 10 performs inclusion region detection processing for each of the multiple first regions detected in the first region detection processing. Each of the multiple first regions is subjected to the inclusion region detection processing once. Control unit 10 performs the inclusion region detection processing on one of the first regions as the processing target, and when that inclusion region detection processing is complete, it performs the inclusion region detection processing on another first region that has not yet been processed as the new processing target.

[0059] The control unit 10 performs an inclusion region detection process to detect a second region that encompasses the first region to be processed. The control unit 10 detects a second region that encompasses the first region to be processed by detecting a second region having an image portion whose image similarity with the first region to be processed is equal to or greater than the threshold for the inclusion region detection process. The control unit 10 determines that the first region to be processed is encompassed by a second region having an image portion whose image similarity with the first region to be processed is equal to or greater than the threshold for the inclusion region detection process. Note that character recognition processing has not been performed at this point. In other words, image similarity refers to the similarity between image data, not the similarity between character strings (i.e., between text data). The control unit 10 performs the inclusion region detection process using, for example, a known pattern matching technique. The threshold for the inclusion region detection process is not particularly limited, but is 70% or greater.

[0060] 5 and 6 , the second region C10 has an image portion where the image similarity with the first region C11 (region with the item name “Name”) is equal to or greater than a threshold, and also has an image portion where the image similarity with the first region C12 (region with the item value “aaaa”) is equal to or greater than a threshold. The second region C20 has an image portion where the image similarity with the first region C21 (region with the item name “Address”) is equal to or greater than a threshold, and also has an image portion where the image similarity with the first region C22 (region with the item value “bbbb”) is equal to or greater than a threshold. The second region C30 has an image portion where the image similarity with the first region C31 (region with the item name “Date of Birth”) is equal to or greater than a threshold, and also has a portion where the image similarity with the first region C32 (region with the item value “cccc”) is equal to or greater than a threshold. As a result, the first regions C11 and C12 are determined to be included in the second region C10, the first regions C21 and C22 are determined to be included in the second region C20, and the first regions C31 and C32 are determined to be included in the second region C30.

[0061] In step #5, the control unit 10 performs a region linking process. Specifically, the control unit 10 links a plurality of first regions included in the same second region to each other.

[0062] 5 and 6, first regions C11 and C12 included in second region C10 are linked to each other. First regions C21 and C22 included in second region C20 are linked to each other. First regions C31 and C32 included in second region C30 are linked to each other.

[0063] In step #6, the control unit 10 performs an area determination process. By performing the area determination process, the control unit 10 recognizes which of the multiple first areas linked to each other is an area for item names and which is an area for item values. In other words, by performing the area determination process, the control unit 10 distinguishes between the area for item names and the area for item values.

[0064] 5 and 6, in the second area C10, the first area C11 is determined to be the area for the item name, and the first area C12 is determined to be the area for the item value. In the second area C20, the first area C21 is determined to be the area for the item name, and the first area C22 is determined to be the area for the item value. In the second area C30, the first area C31 is determined to be the area for the item name, and the first area C32 is determined to be the area for the item value.

[0065] The control unit 10 recognizes the area of ​​the target item by performing an area determination process. The area of ​​the target item is a second area that includes both a first area corresponding to the item name of the target item and a first area corresponding to the item value of the target item. The area determination process will be described in detail later. Note that the area of ​​the target item may be recognized using other methods.

[0066] In step #7, the control unit 10 performs a masking process to mask at least a portion of the area of ​​the target item present in the image data of the identification card. By performing the masking process, the control unit 10 generates masked data in which at least a portion of the area of ​​the target item present in the image data of the identification card is masked.

[0067] The control unit 10 masks the first region associated with the region of the item name of the target item (i.e., the region of the item value of the target item). Alternatively, the control unit 10 masks both the region of the item name and the region of the item value of the target item. Substantially the entire region of the second region corresponding to the target item may be masked.

[0068] An example of masked data is shown in Figure 7. The masked data shown in Figure 7 is generated when the target item is date of birth. That is, at least the item value "cccc" corresponding to the item name "Date of Birth" is masked. Although not shown, both the item name "Date of Birth" and the item value "cccc" may be masked.

[0069] In step #8, the control unit 10 causes the output unit to perform output processing of the masked data. For example, in the masking job settings, the output method of the masked data can be selected. The output methods include printing and transmission.

[0070] When printing is selected as the output method, the control unit 10 causes the printing unit 1 to print (in other words, output) an image based on the masked data onto a sheet S. In this case, the printing unit 1 corresponds to the "output unit," and the output destination is the sheet S.

[0071] When transmission is selected as the output method, the control unit 10 causes the communication unit 102 to transmit (in other words, output) the masked data to the PC 1000. The masked data may be converted into PDF data and transmitted to the PC 1000. By transmitting the masked data to the PC 1000, the masked data can be saved in the PC 1000. In this case, the communication unit 102 corresponds to the "output unit," and the output destination is the PC 1000.

[0072] In this embodiment, a first region detection process, a second region detection process, an inclusion region detection process, and a region linking process are performed, which allows each region of the image data of the identification card that has an item name and an item value corresponding to the same item to be linked with high accuracy.

[0073] By linking the fields of the item name and the field value corresponding to the same item in the image data of the ID card to each other, it is possible to easily mask at least a portion of the target item. Specifically, it is sufficient to mask the field linked to the field of the item name of the target item. This allows the field value of the target item to be masked. With this configuration, it is possible to easily mask only the fields corresponding to the user-specified items in the image data obtained by scanning the document D (ID card) without transferring the image data of the ID card to a processing device on the cloud.

[0074] If the item value area in the image data of an ID card can be masked, the leakage of personal information can be suppressed. Even if the item name area of ​​a target item is masked, if the item value area of ​​the target item is not masked, personal information will be leaked. Therefore, it is important to accurately perform a process of linking each item name and item value area corresponding to the same item in the image data of an ID card.

[0075] In this embodiment, the first area detection process is performed using the first learning model, and the second area detection process is performed using the second learning model. Here, machine learning requires a huge amount of computation and a huge amount of memory capacity. For this reason, machine learning processes are often performed by a processing device on the cloud. However, this requires transferring image data of the ID card to the processing device on the cloud, which is undesirable because it may result in the leakage of personal information.

[0076] Therefore, in this embodiment, a trained first learning model and a trained second learning model are used. The first learning model and the second learning model are each pre-stored in the storage unit 101. This allows the first area detection process and the second area detection process to be performed within the multifunction device 100 without increasing the memory capacity within the multifunction device 100. In other words, there is no need to transfer image data of the identification card to a processing device on the cloud. As a result, leakage of personal information can be suppressed.

[0077] <Area Determination Process> The control unit 10 performs an area determination process to determine whether each of a plurality of first areas (i.e., character areas containing character strings) present in the image data of the identification card is an area for an item name or an area for an item value. The area determination process uses dictionary data DD. The dictionary data DD is pre-stored in the storage unit 101 (see FIG. 2).

[0078] A conceptual diagram of the dictionary data DD is shown in FIG. 8. The dictionary data DD is data in which a plurality of different similar item names are predefined for one type of item. The dictionary data DD is created in advance for each of the plurality of types of items. The storage unit 101 stores in advance a plurality of dictionary data DD corresponding to each of the plurality of types of items. The plurality of dictionary data DD are created in advance by the manufacturer of the multifunction device 100 and pre-stored in the storage unit 101. In FIG. 8, items are indicated by A, B, etc. Similar item names corresponding to item A are indicated by a1, a2, a3, etc. Similar item names corresponding to item B are indicated by b1, b2, b3, etc.

[0079] Taking an identification number as an example of an item, one identification card may use the character string "ID" as the item name, another identification card may use the character string "Number" as the item name, and yet another identification card may use the character string "Num" as the item name. Although the character strings listed here are different from one another, they all correspond to the same item. Therefore, in the dictionary data DD corresponding to the identification number as an item, the character string "ID," the character string "Number," the character string "Num," and the like are predefined as similar item names.

[0080] The flow of the area determination process will be described below with reference to the flowchart shown in Fig. 9. The flow shown in Fig. 9 starts when the area linking process by the control unit 10 is completed.

[0081] In step #11, the control unit 10 performs character recognition processing on the image data of the ID card obtained by the image reading unit 2 reading the ID card (i.e., the processing of step #1 shown in FIG. 4). The control unit 10 performs OCR processing on the image data of the ID card. As a result, the control unit 10 extracts character strings from each of the multiple first areas present in the image data of the ID card. The control unit 10 recognizes the character strings in each of the multiple first areas. Note that the first areas contain character strings indicating item names or item values.

[0082] In step #12, the control unit 10 performs preprocessing on the multiple character strings extracted by the character recognition process. The type of preprocessing is not particularly limited. For example, the control unit 10 performs preprocessing by unifying uppercase and lowercase letters in the character string. This process makes the character string indistinguishable from uppercase and lowercase letters. Furthermore, for example, the control unit 10 performs preprocessing by removing spaces and predetermined symbols.

[0083] After the process of step #12, the control unit 10 performs an item name area determination process. When performing the item name area determination process, the control unit 10 sets one of the multiple first areas detected in the first area detection process as a processing target.

[0084] Then, the control unit 10 performs a similarity calculation process (the process of step #13) as one process of the item name region determination process. In step #13, the control unit 10 calculates the similarity between each character string of all similar item names defined in the multiple dictionary data DD and the character string of the first region to be processed. Hereinafter, the similarity between character strings will be referred to as character string similarity, and will be distinguished from image similarity.

[0085] When performing the similarity calculation process, the control unit 10 selects one of the plurality of dictionary data DD (here, the selected dictionary data DD is referred to as target dictionary data DD) and also selects one of the plurality of similar item names defined in the target dictionary data DD (here, the selected similar item name is referred to as target similar item name).

[0086] The control unit 10 then calculates the similarity between the character string in the first region to be processed and the character string of the target similar item name. For example, the control unit 10 can calculate the similarity Sd between the character string in the first region to be processed and the character string of the target similar item name by using the following formula (1):

[0087] In formula (1), S1 is one character string and S2 is the other character string. For example, S1 is the character string of the similar item name defined in the dictionary data DD, and S2 is the character string of the first region extracted from the image data of an identification card. len(S1) is the number of characters in the character string S1, and len(S2) is the number of characters in the character string S2. M is the number of characters that match between the character string S1 and the character string S2.

[0088] For example, suppose that the character string S1 is "Number" and the character string S2 is "Number." Even if "Number" is written on the identification card, it may be recognized as "Number" due to a misrecognition in the character recognition process.

[0089] In this example, len(S1) = 6 and len(S2) = 7. Furthermore, between string S1 and string S2, counting from the first character, the first character "N," the third character "m," the fourth character "b," the fifth character "e," and the sixth character "r" match each other. As a result, M = 5.

[0090] In this example, the similarity value is 0.76923. If the character strings S1 and S2 contain completely different characters (i.e., M = 0), the similarity value will be 0, and if the character strings S1 and S2 contain exactly the same characters (i.e., M = len(S1) = len(S2)), the similarity value will be 1. That is, in calculating the similarity between the character string in the first region to be processed and the character string of the target similar item name, the higher the similarity, the larger the similarity value, and the lower the similarity, the smaller the similarity value.

[0091] After determining the string similarity between the character string in the first region to be processed and the character string of the target similar item name, the control unit 10 newly selects an unselected similar item name from among the multiple similar item names defined in the target dictionary data DD. The control unit 10 switches the target similar item name and determines the string similarity between the character string in the first region to be processed and the character string of the new target similar item name. The control unit 10 determines the string similarity with the character string in the first region to be processed for all of the multiple similar item names defined in the target dictionary data DD. The control unit 10 then stores the similar item name, among the multiple similar item names defined in the target dictionary data DD, that has the highest string similarity with the character string in the first region to be processed, in association with the value of the string similarity.

[0092] The control unit 10 also calculates the string similarity between all similar item names defined in the multiple dictionary data DD and the string in the first region to be processed. That is, the control unit 10 newly selects an unselected dictionary data DD from the multiple dictionary data DD. The control unit 10 switches the target dictionary data DD and calculates the string similarity between all similar item names defined in the new target dictionary data DD and the string in the first region to be processed. The control unit 10 then stores the similar item name, among the multiple similar item names defined in the target dictionary data DD, that has the highest string similarity with the string in the first region to be processed, in association with its string similarity value. As a result, multiple similar item names corresponding to multiple types of items are stored, each associated with a string similarity value. In other words, multiple similarity values ​​(the similarity values ​​are string similarity values) corresponding to multiple types of items are stored.

[0093] The control unit 10 performs a maximum value detection process (step #14) as part of the item name area determination process. In step #14, the control unit 10 compares the multiple similarity values ​​(the similarity values ​​are string similarity values) calculated and stored in the similarity calculation process. In other words, the control unit 10 compares the multiple similarity values ​​corresponding to the multiple types of items.

[0094] The control unit 10 then detects the highest value of the character string similarity (i.e., the highest similarity value). That is, the control unit 10 calculates the character string similarity between the character string in the first region to be processed and each character string of all similar item names, and detects the highest value among the calculated similarity values.

[0095] The control unit 10 performs a threshold comparison process (processing of step #15) as part of the item name region determination process. In step #15, the control unit 10 compares a predetermined threshold for the item name region determination process with the highest value of the character string similarity.

[0096] For example, to set a threshold value for the item name area determination process, an identification card is actually read by the multifunction device 100, and the result of a similarity calculation process for character strings present in the item name area of ​​the image data obtained by the read (here, the resulting value is referred to as the similarity value of the item name area) is obtained. Also, the result of a similarity calculation process for character strings present in areas of the image data other than the item name area (here, the resulting value is referred to as the similarity value of the other area) is obtained. Note that multiple similarity values ​​for the item name area and multiple similarity values ​​for the other areas are obtained.

[0097] Furthermore, a first average value, which is the average value of the similarity values ​​of the plurality of item name regions, is calculated, and a second average value, which is the average value of the plurality of other regions, is calculated. The average value of the first average value and the second average value is then set as the threshold value for the item name region determination process.

[0098] In step #16, the control unit 10 determines whether the highest value of the character string similarity is equal to or greater than the threshold value for the item name area determination process. In Fig. 9, the threshold value for the item name area determination process is represented as Th. If the control unit 10 determines that the highest value of the character string similarity is equal to or greater than the threshold value for the item name area determination process, the process proceeds to step #17.

[0099] In step #17, the control unit 10 determines that the first region to be processed is an item name region. Furthermore, in step #18, the control unit 10 determines an item corresponding to the first region to be processed based on a character string as an item name present in the first region to be processed (i.e., the first region determined to be an item name region). Here, the character string in the first region to be processed is defined as a similar item name in one of the dictionary data DDs. Therefore, the control unit 10 recognizes an item corresponding to the dictionary data DD in which the character string in the first region to be processed is defined, and determines that the recognized item is an item corresponding to the first region to be processed.

[0100] If the control unit 10 determines in step #16 that the highest value of the character string similarity is less than the threshold value for the item name area determination process, the process proceeds to step #19. If the process proceeds to step #19, the control unit 10 determines that the first area to be processed is another area (i.e., an area different from the item name area).

[0101] Each of the processes from step #13 to step #19 is performed as part of the item name area determination process. After completing the item name area determination process for a certain first area to be processed, the control unit 10 determines whether any first areas remain for which the item name area determination process has not been performed (referred to here as unprocessed first areas). If any unprocessed first areas remain, the control unit 10 sets the unprocessed first area as the new processing object. In other words, the control unit 10 switches the processing object. Then, the control unit 10 performs the item name area determination process (each of the processes from step #13 to step #19) on the first area to be newly processed.

[0102] In this embodiment, dictionary data DD, in which multiple similar item names are predefined for one type of item, is stored in advance in the storage unit 101 for each of multiple types of items. Then, the item name area determination process is performed using the dictionary data DD. This prevents an inconvenience that occurs when an item with multiple corresponding item names is to be masked, where the masked item is properly masked on an identification card bearing one of the multiple item names, but the masked item is not masked on an identification card bearing another item name (in other words, the masked item is not detected).

[0103] For example, suppose an identification number is an item to be masked. In this case, in this embodiment, the identification number is properly masked on both an identification card with "ID" written as the item name and an identification card with "Num" written as the item name. On the other hand, in a conventional configuration, the identification number is properly masked on an identification card with "ID" written as the item name, but the identification number is not masked on an identification card with "Num" written as the item name, which can be an inconvenience.

[0104] To prevent these problems, a large dictionary is required. Furthermore, when using machine learning models, the model size becomes large. However, because memory capacity is limited, processing is performed on the cloud.

[0105] In this embodiment, by using dictionary data DD, processing on the cloud is not required. That is, there is no need to transfer image data of the identification card (i.e., personal information) to a processing device on the cloud. This makes it possible to prevent leakage of personal information. In other words, in this embodiment, only the areas corresponding to the user-specified items in the image data obtained by scanning the document D (identification card) can be easily masked without transferring the image data of the identification card to a processing device on the cloud.

[0106] Furthermore, in this embodiment, the string similarity between the string in the first area to be processed and each string in all similar item names is calculated, and if the highest value of the calculated string similarity is equal to or greater than a threshold value, the first area to be processed is determined to be an item name area. Therefore, even if misrecognition occurs in the character recognition process, the item name area can be detected with high accuracy.

[0107] <Abnormality Countermeasure Processing> First, the first abnormality countermeasure processing will be described.

[0108] Typically, the multiple items of personal information listed on an ID card are different from each other. For example, it is highly unlikely that a single ID card will have multiple items such as "Name." However, if the accuracy of the character recognition process for the image data of the ID card is low, multiple item names (character strings thereof) corresponding to the same item may be extracted from the image data of the ID card. If there is dirt or discoloration on parts of the ID card, or if the resolution when reading the ID card is low, the accuracy of the character recognition process will decrease.

[0109] To prevent such problems from occurring, the control unit 10 performs a first abnormality countermeasure process. Specifically, when the character strings in multiple first areas determined to be item name areas correspond to the same first item, the control unit 10 determines the first area containing the character string with the highest maximum value as the area corresponding to the first item, and determines the remaining first areas as not areas corresponding to the first item. In other words, the control unit 10 corrects the results of the item name area determination process.

[0110] The flow of the first abnormality countermeasure processing will be described below with reference to the flowchart shown in Fig. 10. The flow shown in Fig. 10 starts when the item name area determination processing (i.e., processing according to the flow shown in Fig. 9) by the control unit 10 is completed. After completing the item name area determination processing, the control unit 10 subsequently performs the first abnormality countermeasure processing on the first area to be processed.

[0111] In step #21, the control unit 10 determines whether the first area to be processed is an area containing item names. If the control unit 10 determines that the first area to be processed is an area containing item names, the process proceeds to step #22. If the control unit 10 determines that the first area to be processed is not an area containing item names, the process ends.

[0112] In step #22, the control unit 10 determines whether the item corresponding to the first region to be processed is the same as the item corresponding to another first region. In other words, the control unit 10 determines whether the character strings in each of the first regions determined to be region names of items correspond to the same item (first item).

[0113] If the control unit 10 determines in step #22 that the item corresponding to the first region to be processed is the same as the item corresponding to another first region, the process proceeds to step #23. In other words, if the control unit 10 determines that the character strings in each of the multiple first regions determined to be item name regions correspond to the same item (first item), the process proceeds to step #23. If the control unit 10 determines that the item corresponding to the first region to be processed is different from the item corresponding to another first region, the process ends. Hereinafter, another first region that has the same corresponding item as the first region to be processed is referred to as a first region to be compared.

[0114] In step #23, the control unit 10 recognizes the highest value found in the highest value detection process (i.e., the process of step #14 shown in FIG. 9 ) for each of the first region to be processed and the first region to be compared. The control unit 10 then compares the highest values ​​of the first region to be processed and the first region to be compared. As a result, the control unit 10 compares the highest values ​​of the first region to be processed and the first region to be compared. For example, the control unit 10 determines whether the highest value M1 of the first region to be processed is equal to or greater than the highest value M2 of the first region to be compared.

[0115] In step #23, if the control unit 10 determines that the maximum value M1 of the first region to be processed is equal to or greater than the maximum value M2 of the first region to be compared, the process proceeds to step #24. If the control unit 10 determines that the maximum value M1 of the first region to be processed is less than the maximum value M2 of the first region to be compared, the process proceeds to step #25.

[0116] When the process proceeds to step #24, the control unit 10 does not change the determination result that the first region to be processed is the region corresponding to the first item, but changes the determination result for the first region to be compared. Specifically, the control unit 10 recognizes the second item, which is the item corresponding to the similar item name with the second highest string similarity to the string of the first region to be compared, and determines that the first region to be compared is the region corresponding to the second item.

[0117] When the process proceeds to step #25, the control unit 10 does not change the determination result that the first region to be compared is the region corresponding to the first item, but changes the determination result for the first region to be processed. Specifically, the control unit 10 recognizes the second item, which is the item corresponding to the similar item name with the second highest string similarity to the string of the first region to be processed, and determines that the first region to be processed is the region corresponding to the second item.

[0118] In other words, the following processes are carried out in steps #23 to #25.

[0119] When the character strings in each of the multiple first regions determined to be regions of an item name correspond to the same first item, the control unit 10 determines, among the multiple first regions, the first region that contains the character string with the highest maximum value as the region corresponding to the first item. Furthermore, the control unit 10 recognizes, among the multiple first regions, a second item that is an item corresponding to a similar item name that has the second highest character string similarity to the character string in a first region that was not determined to be a region corresponding to the first item, and determines the first region that was not determined to be a region corresponding to the first item as the region corresponding to the second item.

[0120] Next, the second abnormality countermeasure process and the third abnormality countermeasure process will be described.

[0121] An identification card has multiple item columns. Each item column contains one item name, and it is rare for multiple item names to be written in one item column. That is, typically, two or more of the multiple first areas contained in the same second area are not determined to be item name areas. Also, typically, one of the multiple first areas contained in the same second area is determined to be an item name area. However, if the accuracy of the character recognition process for the image data of the identification card is low, it is possible that two or more of the multiple first areas contained in the same second area are determined to be item name areas, or none of the first areas are determined to be item name areas.

[0122] Therefore, the control unit 10 performs the second abnormality countermeasure process and the third abnormality countermeasure process. Here, the first areas included in the same second area are linked to each other. Therefore, the control unit 10 performs the second abnormality countermeasure process or the third abnormality countermeasure process on the linked first areas.

[0123] The flow of the second abnormality countermeasure processing and the third abnormality countermeasure processing will be described below with reference to the flowcharts shown in FIGS.

[0124] First, the control unit 10 performs processing according to the flow shown in Fig. 11. The flow shown in Fig. 11 starts when the area determination processing by the control unit 10 (i.e., processing according to the flow shown in Fig. 9) is completed. After completing the item name area determination processing, the control unit 10 continues processing according to the flow shown in Fig. 11.

[0125] In step #31, the control unit 10 performs a character string acquisition process to acquire a character string in a first region associated with the first region to be processed. Then, in step #32, the control unit 10 determines whether or not a region determination process has been performed on the first region containing the character string acquired in the character string acquisition process (i.e., whether or not the region has been determined). If it is determined that the region has been determined, the control unit 10 proceeds to step #33. If it is determined that the region has not been determined, the control unit 10 ends this flow.

[0126] In step #33, the control unit 10 determines whether the first area to be processed and the first area linked to the first area to be processed are both determined to be item name areas. In other words, the control unit 10 determines whether two or more of the multiple first areas linked to each other are determined to be item name areas.

[0127] In step #33, if the control unit 10 determines that two or more of the first areas linked to each other are determined to be areas of item names, the process proceeds to step #34. In step #34, the control unit 10 performs a second abnormality countermeasure process.

[0128] If the control unit 10 determines in step #33 that two or more of the first areas linked to each other are not determined to be item name areas, the process proceeds to step #35. In step #35, the control unit 10 determines whether the first area to be processed and the first area linked to the first area to be processed are both determined to be areas other than item name areas. In other words, the control unit 10 determines whether all of the first areas linked to each other are determined to be areas other than item name areas.

[0129] In step #35, if the control unit 10 determines that all of the linked first areas are determined to be areas different from the area of ​​the item name, the process proceeds to step #36. In step #36, the control unit 10 performs a third abnormality countermeasure process.

[0130] In step #35, if the control unit 10 has not determined that all of the linked first areas are areas other than the item name areas, the flow ends. In other words, if only one of the linked first areas is determined to be an item name area, neither the second abnormality countermeasure process nor the third abnormality countermeasure process is performed.

[0131] The second abnormality countermeasure process is performed according to the flow shown in Fig. 12. That is, when the control unit 10 determines that two or more of the multiple first areas linked to each other (i.e., multiple first areas included in the same second area) are areas of item names, the process is performed according to the flow shown in Fig. 12.

[0132] In step #41, the control unit 10 recognizes the highest value found in the highest value detection process (i.e., the process of step #14 in FIG. 9) for each of the multiple first regions linked to each other. In step #42, the control unit 10 recognizes the highest of the highest values.

[0133] Then, in step #43, the control unit 10 determines that the first region containing the character string with the highest maximum value (herein referred to as the first region with the highest maximum value) is an item name region among the multiple first regions linked to each other. In step #44, the control unit 10 determines that the remaining first regions other than the first region with the highest maximum value are regions other than the item name region among the multiple first regions linked to each other.

[0134] Here, if a first area in the image data of the ID card is determined to be an area containing an item name, the control unit 10 stores information indicating that the first area corresponds to the item name. In this configuration, when the second abnormality countermeasure process is performed, the information indicating that the first area corresponds to the item name should be deleted for any first area that is re-determined to be an area other than the area containing the item name. Therefore, in step #45, the control unit 10 deletes the information indicating that the remaining first areas, other than the first area with the highest maximum value, correspond to the item name.

[0135] The third abnormality countermeasure process is performed according to the flow shown in Fig. 13. That is, when the control unit 10 determines that all of the multiple first areas linked to each other (i.e., multiple first areas included in the same second area) are areas different from the area of ​​the item name, the process is performed according to the flow shown in Fig. 13.

[0136] In step #51, the control unit 10 recognizes the maximum value found in the maximum value detection process (i.e., the process of step #14 in FIG. 9) for each of the multiple first regions linked to each other. In step #52, the control unit 10 recognizes the highest maximum value.

[0137] In step #53, the control unit 10 determines that the first region containing the character string with the highest maximum value (herein referred to as the first region with the highest maximum value) is the region of the item name among the plurality of first regions linked to each other. In step #54, the control unit 10 determines that the remaining first regions other than the first region with the highest maximum value are regions other than the region of the item name among the plurality of first regions linked to each other.

[0138] Here, if the control unit 10 determines that a first region in the image data of the ID card is an item name region, the control unit 10 stores information indicating that the first region corresponds to the item name. In this configuration, when the third abnormality countermeasure process is performed, information indicating that the first region corresponds to the item name should be added to the first region that is re-determined as an item name region by that process. Therefore, in step #55, the control unit 10 adds information indicating that the first region with the highest maximum value corresponds to the item name.

[0139] In this embodiment, by performing the first to third abnormality countermeasure processes, it is possible to prevent the occurrence of a problem in which multiple areas with an item name corresponding to the same item are detected, and also to prevent the occurrence of a problem in which an area with an item name is not detected.

[0140] <Masking Job Settings> The user can set up a masking job. The masking job settings are performed via the operation and display unit 3. For example, the user can specify an item corresponding to an item value to be masked (i.e., a target item).

[0141] Specifically, the operation display unit 3 accepts an item designation operation to designate one of the items. The control unit 10 recognizes the item designated by the item designation operation as the target item. Then, as a masking process, the control unit 10 performs a process of masking at least the area of ​​the item value among the areas of the target item present in the image data of the identification card to generate masked data.

[0142] The flow of processing performed by the control unit 10 when accepting settings for a masking job will be described below with reference to the flowchart shown in Fig. 14. For example, in a masking job, after the image reading unit 2 reads an identification card, and before the masking process starts, an item specification operation is accepted by the operation display unit 3. That is, the flow shown in Fig. 14 starts after the process of step #1 shown in Fig. 4 and before the process of step #7.

[0143] In step #61, the operation display unit 3 displays a reception screen 31 as shown in FIG. 15. The reception screen 31 has reception buttons RB arranged thereon. The reception buttons RB include a basic settings button B1 and a custom button B2. The operation display unit 3 accepts an operation on any of the reception buttons RB as an item designation operation. The control unit 10 determines the user-designated item (i.e., the target item) based on which reception button RB the item designation operation was performed on.

[0144] Specifically, the storage unit 101 stores a masking list in which at least one item is registered. The reception button RB is associated with the masking list. If there are multiple masking lists, multiple reception buttons RB (specifically, the same number as the number of masking lists) are displayed on the reception screen 31. The multiple reception buttons RB are associated with different masking lists.

[0145] The masking list corresponding to the basic settings button B1 contains items preselected by the manufacturer. The masking list corresponding to the custom button B2 contains items arbitrarily selected by the user. By creating a new masking list, the custom button B2 (i.e., the reception button RB) corresponding to the newly created masking list is displayed on the reception screen 31.

[0146] In order to receive a request to create a new masking list from the user, a new creation button NB is displayed on the reception screen 31. The operation display unit 3 receives an operation on the new creation button NB as a request to create a new masking list.

[0147] When a request to create a new masking list is received, the operation display unit 3 displays a registration screen 32 as shown in Fig. 16 and receives a registration operation from the user to select an item to be registered. A plurality of items (the item names) are displayed as options on the registration screen 32.

[0148] The operation display unit 3 accepts an operation on the display field of any of the displayed items as a registration operation. For example, if you want to register the item "height," you can select the item "height" as the item to be registered by operating the display field of the item "height." A check mark is added to the display field of the item selected in the registration operation. By operating the enter button DB in the registration screen 32, you can complete the selection of the item through the registration operation.

[0149] When the decision button DB is operated, the control unit 10 determines that acceptance of the registration operation has been completed. After accepting the registration operation, the control unit 10 generates a new masking list in which the items selected in the registration operation are registered, and stores the masking list in the storage unit 101.

[0150] When a new masking list is created and the operation display unit 3 subsequently accepts an item designation operation, the control unit 10 causes the operation display unit 3 to display a reception screen 31 on which an acceptance button RB corresponding to the new masking list is arranged in addition to the acceptance button RB corresponding to the existing masking list.

[0151] When any of the reception buttons RB is operated, the control unit 10 determines that the operation display unit 3 has received an item designation operation. At this time, the control unit 10 determines that the masking list corresponding to the reception button RB for which the item designation operation was performed has been designated. In other words, the item designation operation is an operation for designating any of the masking lists.

[0152] 14, in step #62, the control unit 10 recognizes the reception button RB on which the item designation operation has been performed, i.e., the masking list designated by the item designation operation, and then recognizes the item registered in the masking list designated by the item designation operation as the target item.

[0153] In step #63, the control unit 10 performs a masking process. The process of step #63 is the same as the process of step #7 shown in Fig. 4. The control unit 10 performs the masking process after performing each of the processes of steps #2 to #6 shown in Fig. 4.

[0154] In step #64, the control unit 10 causes the operation display unit 3 to display a preview image (not shown) showing the contents of the masked data, and also causes the operation display unit 3 to accept an editing operation for editing the masked data. The editing operation is an operation for changing an item to be masked. In other words, the editing operation includes an operation for adding a new item to be masked, and an operation for removing a masked item from the masking target.

[0155] In step #65, the control unit 10 determines whether an editing operation has been performed. If an editing operation has been performed, the process proceeds to step #61. After that, in step #63, the control unit 10 generates new masked data that reflects the edited content of the editing operation.

[0156] If it is determined in step #65 that no editing operation has been performed, the process proceeds to step #66. In step #66, the control unit 10 causes the output unit to perform output processing of the masked data. The processing in step #66 is the same as the processing in step #8 shown in FIG.

[0157] In this embodiment, after scanning the ID card, an item designation operation is received to designate the items to be masked (i.e., target items). By performing the item designation operation, at least the area of ​​the item value among the areas of the target items present in the image data of the ID card is masked. In other words, it is possible to easily mask only the areas of the image data obtained by scanning the document D (ID card) that correspond to the items designated by the user.

[0158] For example, in a conventional configuration, it is necessary to scan an ID card in advance and register the area to be masked (the position within the image data). In other words, the ID card must be pre-registered. In the conventional configuration, if the ID card is pre-registered, it is possible to mask the desired area within the image data of the ID card.

[0159] However, from the user's perspective, the task of pre-registering ID cards is cumbersome. If there are many types of ID cards that can be used, pre-registration of each type of ID card must be performed, which increases the amount of work. In addition, the masking target cannot be changed flexibly. If a user wants to change the masking target, they must re-register the ID card, which is cumbersome.

[0160] On the other hand, in this embodiment, when a certain identification card is read, by performing an item designation operation, the area of ​​the image data of the item designated by the item designation operation is masked, and when another identification card of a different type from the identification card is read, the area of ​​the image data of the item designated by the item designation operation is also masked. Furthermore, for example, when the masking target is item A, if it is necessary to further add item B as a masking target (i.e., it is necessary to change the masking target), it is sufficient to simply designate items A and B by the item designation operation. This eliminates the need for work equivalent to pre-registering an identification card, improving user convenience.

[0161] In this embodiment, the item designation operation is an operation for designating a masking list. This eliminates the need to designate each item one by one when masking multiple items. This further improves user convenience.

[0162] In addition, in this embodiment, a new masking list can be created, which is convenient for the user as the masking list can be created as desired.

[0163] Furthermore, in this embodiment, the masked data can be edited, improving user convenience. When editing the masked data, a preview image showing the contents of the masked data is displayed. This allows the masked data to be previewed while being edited, thereby enabling efficient editing of the masked data.

[0164] The embodiments disclosed herein are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the claims, not by the description of the above embodiments, and further includes all modifications within the meaning and scope of the claims.

Claims

1. An image processing device comprising: an image reading unit that reads a document on which multiple sets of item names and item values ​​for items related to personal information are written; a control unit that recognizes target items, which are the items to be masked, and performs a masking process on image data obtained by reading the document with the image reading unit, thereby generating masked data in which at least a portion of the area of ​​the target items present in the image data is masked; and an operation display unit that displays information and accepts operations, wherein the operation display unit accepts an item designation operation that designates any of the items, and the control unit recognizes the item designated by the item designation operation as the target item and performs a process of masking at least the area of ​​the item value among the areas of the target items present in the image data, as the masking process, to generate masked data.

2. An image processing device as described in claim 1, further comprising a memory unit for storing a masking list in which at least one of the items is registered, the item designation operation being an operation for designating one of the masking lists, and the control unit recognizing the item registered in the masking list designated by the item designation operation as the target item.

3. The image processing device of claim 2, wherein the operation display unit accepts a request to create a new masking list, and when the request to create a new masking list is accepted, the operation display unit accepts a registration operation to select the item to be registered, and after accepting the registration operation, the control unit generates a new masking list in which the item selected in the registration operation is registered, and stores the new masking list in the memory unit.

4. The image processing device of claim 1, wherein after the masked data is generated, the control unit causes the operation display unit to display a preview image showing the contents of the masked data and causes the operation display unit to accept editing operations to edit the masked data, the editing operations including an operation to newly add any of the items as masking targets and an operation to remove the masked items from the masking targets, and when the editing operation is performed, the control unit newly generates the masked data reflecting the editing contents of the performed editing operation.

5. An image processing device as described in claim 1, further comprising an output unit that performs output processing of the masked data, the output unit being at least one of a printing unit that performs the output processing by printing an image based on the masked data onto a sheet, and a communication unit that performs the output processing by transmitting the masked data to an external device.

6. An image processing device comprising: an image reading unit that reads a document on which multiple sets of item names and item values ​​for items related to personal information are written; and a control unit that recognizes target items that are the items to be masked, and performs a masking process on image data obtained by reading the document with the image reading unit, thereby generating masked data in which at least a portion of the areas of the target items present in the image data is masked, wherein the control unit performs a first area detection process that detects from the image data a first area that is predicted to individually include either an area of ​​the item name or an area of ​​the item value; a second area detection process that detects from the image data a second area that is predicted to include both areas of the item name and the item value corresponding to the same item; an including area detection process that is a process performed for each of the multiple first areas, and detects the second area that includes the first area to be processed by detecting the second area having an image portion whose image similarity with the first area to be processed is equal to or greater than a threshold; and an area linking process that links the multiple first areas included in the same second area to each other.

7. An image processing device as described in claim 6, comprising a memory unit, the memory unit pre-storing a trained first learning model that has been machine-learned to detect the first region from the image data, and a trained second learning model that has been machine-learned to detect the second region from the image data, and the control unit performing a process of detecting the first region from the image data using the first learning model as the first region detection process, and performing a process of detecting the second region from the image data using the second learning model as the second region detection process.

8. An image processing device as described in claim 6, further comprising an output unit which performs output processing of the masked data, the output unit being at least one of a printing unit which performs printing an image based on the masked data onto a sheet as the output processing, and a communication unit which performs transmission of the masked data to an external device as the output processing.

9. An image processing device comprising: an image reading unit that reads a document on which multiple sets of item names and item values ​​for items related to personal information are written; a control unit that recognizes target items that are to be masked, and performs a masking process on image data obtained by reading the document with the image reading unit, thereby generating masked data in which at least a portion of the area of ​​the target items present in the image data is masked; and a memory unit, wherein the memory unit stores dictionary data in which a plurality of different similar item names are predefined for one type of item, for each of the multiple types of items, and the control unit performs a first area detection process that detects, from the image data, a first area that is predicted to individually include either an area for the item name or an area for the item value, a character recognition process that recognizes a character string in the first area, and an item name area determination process that is a process performed for each of the multiple first areas, in which character string similarities between the character string in the first area to be processed and each of the character strings of all the similar item names are obtained, and if the highest value of the obtained character string similarities is equal to or greater than a threshold value, the first area to be processed is determined to be the area for the item name.

10. An image processing device as described in claim 9, wherein, when the character strings in each of the multiple first areas determined to be areas of the item name correspond to the same first item, the control unit determines that, among the multiple first areas determined to be areas of the item name, the first area containing the character string with the highest maximum value is the area corresponding to the first item.

11. The image processing device described in claim 10, wherein the control unit recognizes a second item, which is the item corresponding to the similar item name that has the second highest string similarity to the string of a first area not determined to be an area corresponding to the first item, among the multiple first areas determined to be areas of the item name, and determines the first area not determined to be an area corresponding to the first item to be an area corresponding to the second item.

12. The image processing device of claim 9, wherein the control unit performs a second region detection process to detect from the image data a second region predicted to include regions of both the item name and the item value corresponding to the same item, and a region linking process to link multiple first regions included in the same second region to each other, and when two or more of the multiple first regions linked to each other are determined to be the item name region, the control unit determines the first region among the multiple first regions linked to each other that contains the character string with the highest maximum value to be the item name region, and determines the remaining first regions to be regions other than the item name regions.

13. The image processing device of claim 9, wherein the control unit performs a second region detection process to detect from the image data a second region predicted to include both the region of the item name and the region of the item value corresponding to the same item, and a region linking process to link multiple first regions included in the same second region to each other, and when all of the multiple first regions linked to each other are determined to be regions other than the region of the item name, the control unit determines the first region including the character string with the highest maximum value among the multiple first regions linked to each other to be the region of the item name, and determines the remaining first regions to be regions other than the region of the item name.

14. An image processing device as described in claim 9, further comprising an output unit which performs output processing of the masked data, the output unit being at least one of a printing unit which performs printing an image based on the masked data onto a sheet as the output processing, and a communication unit which performs transmitting the masked data to an external device as the output processing.

Images