Image processing device

Abstract

To generate an image data indicating a document to which an appropriate face picture that is matched to a name written into the document is attached without applying a workload to a user.SOLUTION: An image formation device 1 comprises: an analysis part that analyzes an imaging data of documents D1 and D2, detects picture attachment regions PA1 and PA2 provided to the documents D1 and D2, and extracts a character string of "PATENT TARO" and "PATENT TARO" as a name written into the documents D1 and D2; an acquisition part 103 that requests a transmission of a picture data of a face picture of a person matched to the name indicated by the character string extracted by the analysis part against a LDAP server or the like providing a directory service via a network, and acquires the picture data; and a generation part 104 that processes the size of the face picture indicated by the picture data acquired by the acquisition part and arranges them to picture attachment regions PA1 and PA2 of the documents D1 and D2, and generates a composite image.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing device, and more particularly to an image processing device capable of editing a document. [Background technology]

[0002] Application forms for various licenses and qualifications, as well as resumes, examination applications, and other documents require users to not only fill in necessary information such as their name and address, but also to attach a photograph of themselves (passport photo). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent No. 10110767 Summary of the Invention [Problem to be solved by the invention]

[0004] However, it is time-consuming for users to prepare a facial photograph suitable for attaching to a document and then attach it to the document.

[0005] The above-mentioned Patent Document 1 describes that a user operates a mobile device to select an image (face photo) suitable for attachment from images stored in a gallery application, and that the mobile device attaches the image selected by the user to a scanned document, but when a large number of images are stored in the gallery application, it is troublesome for the user to select a face photo suitable for attachment to a document. In particular, when images of many people, not just the user himself, are stored in the gallery application, it is very difficult to select a face photo suitable for attachment to a document.

[0006] The present invention has been made in consideration of the above circumstances, and has an object to generate image data showing a document to which an appropriate facial photograph that matches the name written on the document is attached, without imposing a heavy workload on the user. [Means for solving the problem]

[0007] An image processing device according to one aspect of the present invention comprises a document reading unit which reads an image of a document; a communication unit which communicates via a network with a management device which manages photo information having photo data of a face photograph and name data of the person in the face photograph; an analysis unit which detects a photo attachment area provided on the document from the image data of the document obtained by reading by the document reading unit and extracts a character string as the name written on the document; an acquisition unit which, via the communication unit, requests the management device to transmit the photo data of the face photograph associated with name data matching the name indicated by the character string extracted by the analysis unit, and acquires the photo data; and a generation unit which generates a composite image in which the photo data acquired by the acquisition unit is placed in the photo attachment area in the image data, with the size adjusted to fit the photo attachment area. Effect of the Invention

[0008] According to the present invention, photo data of a face photo of a person matching the name written on a document is obtained from a management device that manages photo information, and the photo data is used to generate a composite image in which the face photo is placed in the photo attachment area of ​​the document. Therefore, image data showing a document to which a proper face photo matching the name written on the document is attached can be generated without imposing a large workload on the user. [Brief description of the drawings]

[0009] [Figure 1] 1 is a perspective view showing the appearance of an image forming apparatus as an image processing apparatus according to a first embodiment of the present invention. [Diagram 2] 1 is a functional block diagram illustrating an outline of a main internal configuration of an image forming apparatus as an image processing apparatus according to a first embodiment. [Diagram 3] 13A and 13B are diagrams showing examples of documents that require the attachment of a photograph. [Figure 4] 13A and 13B are diagrams showing examples of documents that require the attachment of a photograph. [Diagram 5] 6 is a flowchart showing an example of a first embodiment of a process for combining photographic images in the image forming apparatus 1. [Figure 6] FIG. 11 is a functional block diagram illustrating an outline of a main internal configuration of an image forming apparatus as an image processing apparatus according to a second embodiment. [Figure 7] 10 is a flowchart showing an example of a second embodiment of a process for combining photographic images in the image forming apparatus 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] An image processing device according to an embodiment of the present invention will be described below with reference to the drawings. Fig. 1 is a perspective view showing the appearance of an image forming device as an image processing device according to a first embodiment of the present invention. Fig. 2 is a functional block diagram showing a schematic main internal configuration of the image forming device as an image processing device according to the first embodiment.

[0011] Image forming apparatus 1 is a multifunction device having multiple functions such as a copy function, a printer function, a scanner function, and a facsimile function, and includes a control unit 10, a document feed section 6, a document reading section 5, an image forming section 12, a fixing section 13, a paper feed section 14, an operation section 47, a network interface section 91, and a storage section 8. The document reading section 5 and the network interface section 91 are examples of a document reading section and a communication section, respectively, in the claims.

[0012] The document feeder 6 is configured to be openable and closable by a hinge or the like (not shown) on the upper surface of the document reading unit 5, and functions as a document pressing cover when reading a document placed on a platen glass (not shown). The document feeder 6 is an automatic document feeder called an ADF (Auto Document Feeder), and includes a document placement tray 61 on which documents are placed, and a document discharge tray 62 provided below the document placement tray 61. The document feeder 6 supplies documents placed on the document placement tray 61 to the document reading unit 5 one by one, and discharges the documents to the document discharge tray 62.

[0013] A description will be given of a case where a document reading operation is performed by the image forming apparatus 1. The document reading unit 5 optically reads an image of a document supplied to the document reading unit 5 by the document feeding unit 6 or a document placed on the platen glass, and generates image data. The image data generated by the document reading unit 5 is stored in an image memory or the like (not shown).

[0014] A description will be given of an image forming operation performed by the image forming apparatus 1. Based on image data generated by a document reading operation or image data received from a computer serving as an external device (e.g., a personal computer) connected via a network, the image forming unit 12 forms a toner image on a recording paper serving as a recording medium fed from a paper feed unit 14.

[0015] The fixing unit 13 applies heat and pressure to the recording paper on which the toner image has been formed by the image forming unit 12, thereby fixing the toner image to the recording paper. The recording paper that has been subjected to the fixing process is discharged onto a discharge tray 151. The paper supply unit 14 includes a plurality of paper supply cassettes 141.

[0016] The operation unit 47 receives instructions, such as an instruction to execute an image forming operation, from an operator regarding various operations and processes that can be executed by the image forming apparatus 1. The operation unit 47 includes a display unit 473 that displays operation guides and the like for the operator. The operation unit 47 also receives input of instructions from the user via a touch panel included in the display unit 473, based on the user's operation (touch operation) on an operation screen displayed on the display unit 473 or the user's operation on physical keys.

[0017] The display unit 473 is composed of an LCD (Liquid Crystal Display) or the like. The display unit 473 is equipped with a touch panel. When the operator touches a button or key displayed on the screen, the touch panel receives an instruction associated with the touched position.

[0018] The network interface unit 91 is a communication interface that transmits and receives various data to and from external devices (e.g., personal computers and servers) within a local area or on the Internet. For example, the network interface unit 91 enables communication with a management device that manages photo information having photo data of a face photograph and the name data of the person in the face photograph via a network.

[0019] The management device is, for example, a Lightweight Directory Access Protocol (LDAP) server that provides a directory service. The photo information collected and managed by the management device includes collection date and time information of the photo data. If the photo data collected by the management device includes shooting date and time information, the photo information managed by the management device further includes shooting date and time information of the photo data.

[0020] The storage unit 8 is a large-capacity storage device such as a hard disk drive (HDD) or a solid state drive (SSD), and stores various control programs and the like.

[0021] The control unit 10 includes a processor, a random access memory (RAM), a read only memory (ROM), and a dedicated hardware circuit. The processor is, for example, a central processing unit (CPU), an application specific integrated circuit (ASIC), or a micro processing unit (MPU). The control unit 10 includes a control unit 100, an analysis unit 101, an acquisition unit 103, and a generation unit 104.

[0022] The control unit 10 functions as a control unit 100, an analysis unit 101, an acquisition unit 103, and a generation unit 104 by the operation of the processor in accordance with a control program stored in the storage unit 8. However, the control unit 100 and the like can also be configured by hardware circuits, regardless of the operation in accordance with the control program by the control unit 10. The same applies to each embodiment below unless otherwise stated.

[0023] The control unit 100 is responsible for controlling the overall operation of the image forming apparatus 1. The control unit 100 is connected to the document feeding unit 6, the document reading unit 5, the image forming unit 12, the fixing unit 13, the paper feeding unit 14, the operation unit 47, the network interface unit 91, and the storage unit 8, and performs drive control of these units. For example, the control unit 100 executes various processes necessary for image formation by the image forming apparatus 1.

[0024] The analysis unit 101 analyzes the image data of the document obtained by reading by the document reading unit 5, detects a photo attachment area provided on the document, and extracts a character string as the name written on the document. The analysis unit 101 includes a character recognition unit 102. The character recognition unit 102 uses an existing OCR (Optical Character Recognition) technology to recognize characters included in the image represented by the image data.

[0025] Figures 3(A) and (B) are diagrams showing an example of a document that requires the attachment of a facial photograph. Documents D1 and D2 shown in Figures 3(A) and (B) are provided with name entry areas NA1 and NA2 for entering a name, and entry areas for entering other necessary information (e.g., address), as well as photo attachment areas PA1 and PA2 for attaching a facial photograph. In addition, the word "PHOTO" is written in the photo attachment area PA1, and "PHOTO" is written near the upper outer side of the photo attachment area PA2.

[0026] As described above, the photo attachment area contains an image consisting of a rectangle or a line drawing approximating a rectangle, and a word indicating a photo (for example, "photo" in English or "photo" in Japanese) inside or near the outside (within a predetermined distance from the outer shape). The photo attachment area is generally shown as a rectangle. As shown in FIG. 3(A), some photo attachment areas have rounded corners, but the shape is generally rectangular.

[0027] Since the size of the ID photo varies from document to document, the size of the photo attachment area also varies from document to document. The smallest size of a typical ID photo in Japan is 30 mm long and 24 mm wide, which is used for a driver's license, and the largest is 70 mm long and 50 mm wide, which is used for university entrance exams. In other words, the vertical side of the photo attachment area falls within a range of 30 to 70 mm, and the horizontal side falls within a range of 24 to 50 mm. Therefore, the analysis unit 101 can determine that an image that satisfies the condition that (i) the image is a rectangle or a line drawing that approximates a rectangle, and a phrase indicating a photo is written inside or near the outside of the image that is a rectangle or a line drawing that approximates a rectangle, or (ii) the image is a rectangle or a line drawing that approximates a rectangle, and has the above-mentioned predetermined vertical and horizontal dimensions, indicates a photo attachment area.

[0028] Also, the name entry area NA1 is indicated by a ruled line L1 and a name item name N1 called "NAME" as shown in Fig. 3(A). Also, the name entry area NA2 is indicated by a frame F1 and a name item name N2 called "Applicant's Name" as shown in Fig. 3(B). Therefore, the analysis unit 101 can determine that an image that satisfies a condition that a word indicating a name is placed adjacent to a ruled line or that a word indicating a name is placed above a ruled line indicates a name entry area.

[0029] Next, a method for detecting the photo attachment areas PA1, PA2 provided in the documents D1, D2 by the analysis unit 101 will be described. For example, the analysis unit 101 detects edge pixels from the documents D1, D2 indicated by the image data indicating the documents D1, D2, detects a plurality of straight lines from the detected edge pixels, detects two straight lines extending parallel to each other and two straight lines approximately perpendicular to each of the two straight lines, and detects a rectangle (including an approximately rectangular shape) constituted by these four straight lines.

[0030] Then, based on the recognition result by the character recognition unit 102, the analysis unit 101 determines whether or not a predetermined word indicating a photograph (e.g., photo or photograph) is present inside or near the outside of the detected rectangle, and detects the rectangle within or near the outside of which the word is present as a photo attachment area.

[0031] Alternatively, the analysis unit 101 detects, among the detected rectangles, those with vertical sides of 30 to 70 mm and horizontal sides of 24 to 50 mm as photo attachment areas. As described above, the vertical sides of photo attachment areas fall within the range of 30 to 70 mm and the horizontal sides fall within the range of 24 to 50 mm, so the analysis unit 101 determines that rectangles outside these ranges are not photo attachment areas.

[0032] Next, a method for the analysis unit 101 to extract character strings as the name written in the documents D1 and D2 will be described. The analysis unit 101 analyzes image data showing the documents D1 and D2, detects layout information such as ruled lines L1 and frames F1 included in the documents D1 and D2, and detects writing areas for writing necessary information such as the name and address, and detects predetermined words and phrases indicating the name (e.g., name, given name, first name) included in the documents D1 and D2 as name item names based on the recognition results by the character recognition unit 102.

[0033] Then, the analysis unit 101 detects the writing areas existing near the name item name as the name writing areas NA1, NA2. That is, the analysis unit 101 determines that an image that satisfies a condition that a word indicating a name is placed adjacent to a ruled line or a word indicating a name is placed above a ruled line indicates a name writing area. The analysis unit 101 extracts the character strings that are written in the name writing areas NA1, NA2 recognized by the character recognition unit 102 as the name.

[0034] For example, as shown in FIG. 4(A), if "PATENT TARO" is written in the name entry area NA1 of document D1, and as shown in FIG. 4(B), if "Patent Taro" is written in the name entry area NA2 of document D2, the analysis unit 101 extracts the character string "PATENT TARO" as the name from document D1, or extracts the character string "Patent Taro" as the name from document D2.

[0035] The acquisition unit 103 requests the management device (e.g., an LDAP server) via the network interface unit 91 to transmit photo data of a facial photograph associated with the name data that matches the name indicated by the character string extracted by the analysis unit 101, and acquires the photo data.

[0036] The generating unit 104 generates a composite image by arranging the face photo indicated by the photo data acquired by the acquiring unit 103 in a size that matches the photo attachment area of ​​the document indicated by the image data obtained by reading by the document reading unit 5. For example, the generating unit 104 recognizes the face included in the face photo indicated by the photo data using an existing face recognition program such as OpenCV, processes the size of the face photo so that the face part from the top of the head to the tip of the chin is included in the upper 60% of the photo attachment area, and places the face photo in the photo attachment area to generate a composite image.

[0037] Next, an example (first embodiment) of a process for compositing a photographic image in the image forming apparatus 1 will be described with reference to the flowchart shown in Fig. 5. This process is performed when the operation unit 47 receives a composition instruction for compositing a photographic image with image data read by the document reading unit 5.

[0038] When the operation unit 47 receives the above-mentioned synthesis instruction from the user, the control unit 100 causes the document reading unit 5 to read the document placed on the document placement tray 61 or the platen glass, and acquires image data of the document (S1). After that, the analysis unit 101 executes a process of detecting a photo attachment area included in the document image data from the document image data obtained by reading by the document reading unit 5 (S2).

[0039] The analysis unit 101 judges whether or not a photo attachment area included in the document image data has been detected (S3), and if it judges that a photo attachment area has not been detected (NO in S3), it judges whether or not photo data showing a face photo is included in the document image data (S5). If the analysis unit 101 judges that photo data showing a face photo is not included in the document image data (NO in S5), the control unit 100 causes the display unit 473 to display a pop-up screen having a message saying "There is no need to attach a face photo to this document" (S4). After this, the process ends.

[0040] On the other hand, if the analysis unit 101 determines that the image data of the document contains photographic data showing a face photograph (YES in S5), the control unit 100 causes the display unit 473 to display a pop-up screen having a message saying "A face photograph is attached to this document" (S6). After this, the process ends.

[0041] For example, the analysis unit 101 uses an existing face recognition program such as the above-mentioned OpenCV to determine whether or not a face image exists in the detected photo attachment area, and if a face image exists in the image data showing the document, it determines that a face photo is attached to the photo attachment area. Alternatively, the analysis unit 101 simply determines whether or not an image image exists in the image data showing the document, and if an image image exists, it determines that a face photo is attached to the photo attachment area.

[0042] Furthermore, in S3, when the analysis unit 101 determines that a photo attachment area included in the document image data has been detected (YES in S3), it extracts a character string as the name from the document image data (S7). The detection of a photo attachment area in the document image data indicates that photo data showing a facial photograph is not included in the document image data. If a facial photograph is pasted on top of the photo attachment area in the signature, the image data obtained by reading the type includes photo data showing the facial photograph, but the image showing the photo attachment area is hidden by the photo data showing the facial photograph and cannot be read.

[0043] Then, the analysis unit 101 judges whether or not a character string representing the name written on the document has been extracted (S8), and if it is judged that a character string representing the name has not been extracted (NO in S8), the control unit 100 causes the display unit 473 to display a pop-up screen having a message saying "The name has not been written on the document. Please write the name on the document." After this, the process ends.

[0044] On the other hand, if the analysis unit 101 determines that it has been able to extract a character string as a given name (YES in S8), the acquisition unit 103 sends a request to the management device via the network interface unit 91 to send photo data of a facial photograph associated with the given name data that matches the given name indicated by the character string extracted by the analysis unit 101 (S10).

[0045] Then, the acquiring unit 103 judges whether or not the photo data corresponding to the request has been acquired from the management device (S11). If the acquiring unit 103 judges that the photo data has not been acquired (NO in S11), the control unit 100 causes the display unit 473 to display a pop-up screen having a message saying "There is no photo of the person whose name matches the name entered in the document" (S12). After this, the process ends.

[0046] On the other hand, if the acquisition unit 103 determines that it has acquired the photo data (YES in S11), the generation unit 104 generates a composite image in which the photo data acquired by the acquisition unit 103 is resized to fit the photo attachment area of ​​the document indicated by the image data obtained by reading by the document reading unit 5 and placed in the photo attachment area (S13).

[0047] In addition, when the acquisition unit 103 acquires from the management device multiple photo data of facial photos associated with name data that matches the name indicated by the character string extracted by the analysis unit 101, the generation unit 104 selects, based on the collection date and time information or the shooting date and time information contained in the photo information, the facial photo indicated by the photo data with the latest collection date and time or shooting date and time from the multiple photo data acquired by the acquisition unit 103, and generates the composite image using the selected photo data.

[0048] Next, the control unit 100 (a) causes the image forming unit 12 to form on a recording sheet a composite image indicated by the image data indicating the composite image generated by the generation unit 104, or (b) causes the image forming unit 12 to store the image data in the storage unit 8, according to the output method (printing or saving) indicated by the instruction received from the user by the operation unit 47 (S14). The control unit 100 causes the display unit 473 to display a pop-up screen having a message such as "The document with the facial photo attached has been printed" or "The file of the document with the facial photo attached has been saved" (S15). After this, the process ends.

[0049] According to the above embodiment, photo data of a face photo associated with the name data matching the name written on the document is obtained from a management device that manages photo information, that is, photo data of a face photo of a person matching the name written on the document, and a composite image in which the face photo is placed in the photo attachment area of ​​the document is generated using the photo data. Therefore, image data showing a document to which an appropriate face photo matching the name written on the document is attached can be generated without imposing a large workload on the user.

[0050] 6 is a functional block diagram showing a schematic diagram of a main internal configuration of an image forming apparatus as an image processing apparatus according to the second embodiment. In the second embodiment, the control unit 10 functions as a control unit 100, an analysis unit 101, an acquisition unit 103, a generation unit 104, and further a determination unit 105, by the operation of the processor in accordance with the control program stored in the storage unit 8. In the description of the second embodiment, the description of the same configuration and processing as the first embodiment will be omitted.

[0051] In the first embodiment, only when the analysis unit 101 determines that photo data showing a facial photo is not included in the image data of the document (i.e., when the user has not attached a facial photo to the document), the acquisition unit 103 requests and acquires photo data from the management device, and the generation unit 104 generates the composite image. In the second embodiment, however, the acquisition unit 103 requests and acquires photo data from the management device regardless of whether a facial photo is attached to the document, and in this respect, it differs from the first embodiment.

[0052] 7 shows a flowchart showing an example of a second embodiment of the process of combining photo images in the image forming apparatus 1. In the process of the second embodiment, when the analysis unit 101 determines that photo data showing a face photo is included in the image data of the document (YES in S5), the determination unit 105 determines whether or not the face photo shown by the photo data arranged in the photo attachment area in the image data of the document matches the face photo shown by the photo data acquired by the acquisition unit 103 (S21). For example, the determination unit 105 performs pattern matching between the two and determines whether or not the feature amounts related to the two match, and when it is determined that the feature amounts match a predetermined ratio (for example, 90%), it determines that the two face photos match.

[0053] If the determination unit 105 determines that the facial photo indicated by the photo data placed in the photo attachment area in the image data of the document does not match the facial photo indicated by the photo data acquired by the acquisition unit 103 (NO in S21), the control unit 106 causes the display unit 473 to display a facial photo indicating the facial photo determined by the determination unit 105 to not match the facial photo attached to the photo attachment area (S25).

[0054] When an instruction to accept the facial photograph displayed on the display unit 473 is received from the user via the operation unit 47, the generation unit 104 generates the above-mentioned composite image using the displayed facial photograph (S26). After this, the process ends.

[0055] If the determination unit 105 determines that the face photo indicated by the photo data arranged in the photo attachment area in the document image data matches the face photo indicated by the photo data acquired by the acquisition unit 103 (YES in S21), the control unit 100 causes the display unit 473 to display a pop-up screen having a message saying "A face photo is attached to this document" (S6). After this, the process ends.

[0056] According to the second embodiment, even if the user has attached a face photo to the document, if the face photo indicated by the photo data arranged in the photo attachment area in the image data of the document does not match the face photo indicated by the photo data acquired by the acquisition unit 103, another face photo is presented to the user, and the user is provided with an opportunity to change the face photo to be attached to the document. Therefore, the user can efficiently and accurately generate image data indicating a document to which an appropriate face photo that matches the name and surname entered on the document is attached.

[0057] The present invention is not limited to the configuration of the above embodiment, and various modifications are possible. In addition, in the above embodiment, the configuration and processing shown in the above embodiment using Figures 1 to 7 are merely one embodiment of the present invention, and the present invention is not limited to the configuration and processing. [Explanation of symbols]

[0058] 1. Image forming device 5 Document reading section 47 Operation section 91 Network Interface Section 100 Control section 101 Analysis Department 103 Acquisition Department 104 Generation part 105 Judgment section 106 Control section 473 Display section

Claims

1. a document reading unit that reads an image of a document; A communication unit that communicates with a management device that manages photo information including photo data of a face photo and name data of the person in the face photo via a network; an analysis unit that detects a photo attachment area provided on the document from the image data of the document obtained by reading the document by the document reading unit, and extracts a character string as a name written on the document; an acquisition unit that requests the management device to transmit, by the communication unit, the photograph data of a face photograph associated with name data that matches the name indicated by the character string extracted by the analysis unit, and acquires the photograph data; a generation unit that generates a composite image in which the photo data acquired by the acquisition unit is resized and placed in the photo attachment area in the image data to fit the photo attachment area.

2. The analysis unit further determines whether the image data of the document includes photographic data showing a facial photograph, The image processing device according to claim 1 , wherein the generating unit generates the composite image only when the analyzing unit determines that the image data of the document does not include photographic data showing a face photograph.

3. the photo information collected and managed by the management device includes collection date and time information or photography date and time information of the photo data, When the acquisition unit acquires from the management device a plurality of pieces of face photograph data associated with name data that matches the name indicated by the character string extracted by the analysis unit, The image processing device according to claim 2, wherein the generation unit generates the composite image using a facial photo indicated by the photo data with the latest collection date and time or shooting date and time among the multiple photo data acquired by the acquisition unit, based on the collection date and time information or the shooting date and time information.

4. A display unit; an operation unit for receiving input of instructions; A determination unit that determines whether or not a facial photo indicated by the photo data arranged in the photo attachment area matches a facial photo indicated by the photo data acquired by the acquisition unit; A control unit that causes the display unit to display a face photo list indicating face photos that are determined by the determination unit to not match a face photo indicated by the photo data arranged in the photo attachment area, The image processing device according to claim 1 , wherein the generating unit generates the composite image using photographic data indicating a face photograph indicated by a selection instruction received by the operation unit.

Images