Image processing device, method, and system

Abstract

To acquire an image in which an expression according to an imaging situation is expressed, when modifying a missing area.SOLUTION: A person himself / herself image selection part 16C selects person himself / herself images which include the face of an object person, from a plurality of reference images 30 which are stored in a storage circuit 17 in advance. A specific image selection part 16D selects images which are imaged in a situation similar to or matching to a situation where an object image 20 is imaged, as specific images, from among the selected person himself / herself images. An image extraction part 16B extracts an image of a corresponding area to be used for modifying a missing area from the selected specific images. An image modification part 16E modifies the missing area, using the extracted image of the corresponding area.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing technique for correcting a target area, including a missing area that cannot be seen, in a face image of a person, using a face image of the person stored in advance. [Background technology]

[0002] When taking images of people at events or other such occasions, there are cases where the person's face is not clearly discernible in an image due to the person's wearing a mask, sunglasses, or other obstruction. Conventionally, Patent Document 1 discloses a technology for generating a self-portrait image of a person wearing an HMD (Head Mounted Display) and experiencing VR (Virtual Reality) by combining an image of the VR space with an image of the VR space, in which the person appears as if they are in the VR space. In this case, the HMD part of the face image is modified with a part of the face image of the person that has been prepared in advance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-121074 A Summary of the Invention [Problem to be solved by the invention]

[0004] According to the technology of Patent Document 1 mentioned above, for a target image that includes missing areas that cannot be seen due to the wearing of an obstruction such as a mask or sunglasses, a partial image corresponding to the missing area is extracted from a previously prepared facial image of the person, and this partial image is used to correct the missing area in the target facial image, thereby obtaining an image without the missing area.

[0005] On the other hand, when photographing a person's image at an event or other such event, the person's facial expression changes depending on the shooting conditions. In particular, the facial expressions of the eyes and mouth give the photographed image a certain atmosphere and lingering impression. However, the above technology has a problem in that, even if the shooting conditions are different, the same partial image prepared in advance is used for correction, so that the facial expression according to the shooting conditions is not expressed, and an image without a sense of realism is generated.

[0006] The present invention is made to solve such problems, and aims to provide an image processing technique that makes it possible to obtain an image that expresses an expression appropriate to the shooting conditions when correcting missing areas. [Means for solving the problem]

[0007] In order to achieve this object, the image processing device of the present invention comprises a storage unit configured to store in advance a plurality of reference images including a person's face and taken under various conditions; a memory unit that stores a target image including a target person whose face has a missing area that is not visible; and a control unit configured to select a specific image including the face of the target person from the plurality of reference images stored in the storage unit and correct the missing area of ​​the target image using the specific image, and the control unit includes a personal image selection unit configured to select a personal image including the face of the target person from the plurality of reference images; a specific image selection unit configured to select, from the selected personal images, an image taken under conditions that match or are similar to the conditions under which the target image was taken as the specific image; an image extraction unit configured to extract an image of a corresponding area used to correct the missing area from the selected specific image; and an image correction unit configured to correct the missing area using the extracted image of the corresponding area.

[0008] In addition, in one configuration example of the image processing device of the present invention, the image extraction unit extracts a target face image showing the face of the target person from the target image, and the person image selection unit is configured to select a person image including the target person from among the multiple reference images by comparing the target face image extracted by the image extraction unit with each of the multiple reference images.

[0009] Furthermore, in one configuration example of the image processing device of the present invention, the storage unit further pre-stores, for each of the plurality of reference images, a person keyword indicating the person contained in the reference image, the memory unit stores, together with the target image, a person keyword indicating the target person contained in the target image, and the person image selection unit is configured to select, from among the plurality of reference images, a reference image to which a person keyword matching the person keyword stored in the memory unit has been assigned as the person image.

[0010] In addition, in one example configuration of the above-mentioned image processing device of the present invention, the image extraction unit extracts a target background image showing the background of the target person from the target image, and the specific image selection unit is configured to select, as the specific image, an image from the person images whose background matches or is similar to the target background image by comparing the target background image extracted by the image extraction unit with each of the person images.

[0011] In addition, in one example configuration of the image processing device of the present invention, the storage unit further stores, for each of the multiple reference images, a situation keyword indicating the situation in which the reference image was taken, the memory unit stores, together with the target image, the situation keyword indicating the situation in which the target image was taken, and the specific image selection unit is configured to select, from among the images of the person, an image of the person to which a situation keyword that matches or is similar to the situation keyword stored in the memory unit has been assigned as the specific image.

[0012] In addition, in one example configuration of the above-mentioned image processing device of the present invention, the storage unit further stores in advance a plurality of reference background images captured against various backgrounds, and for each of the plurality of reference background images, a situation keyword corresponding to the background of the reference background image is stored in advance, the image extraction unit extracts a target background image indicating the background of the target person from the target image, and the specific image selection unit is configured to compare the target background image extracted by the image extraction unit with each of the plurality of reference background images, and thereby obtain a situation keyword of a reference background image from among the plurality of reference background images whose background matches or is similar to that of the target background image, as the situation keyword indicating the situation in which the target image was captured.

[0013] Moreover, the image processing method according to the present invention is an image processing method used by a computer having a storage unit configured to store in advance a plurality of reference images including a person's face and taken under various conditions; a memory unit that stores a target image including a target person whose face has a missing area that cannot be seen; and a control unit configured to select a specific image including the face of the target person from the plurality of reference images stored in the storage unit and correct the missing area of ​​the target image using the specific image, and includes a person image selection step in which the control unit is configured to select a person image including the face of the target person from the plurality of reference images; a specific image selection step in which the control unit is configured to select, from the selected person images, an image taken under conditions that match or are similar to the conditions under which the target image was taken, as the specific image; an image extraction step in which the control unit is configured to extract an image of a corresponding area used to correct the missing area from the selected specific image; and an image correction step in which the control unit is configured to correct the missing area using the extracted image of the corresponding area.

[0014] In addition, the image processing system of the present invention comprises an image processing device and a storage device connected to each other via a communication network so as to be capable of data communication, and the storage device is configured to store in advance a plurality of reference images including a person's face and taken under various circumstances, and the image processing device comprises a memory unit that stores a target image including a target person having a missing area on the face that is not visible, and a control unit configured to select a specific image including the face of the target person from the plurality of reference images stored in the storage device and correct the missing area of ​​the target image using the specific image, and the control unit includes a person image selection unit that selects a person image including the face of the target person from the plurality of reference images, a specific image selection unit configured to select an image of the target person including the face of the target person from the selected person images as the specific image, an image taken under circumstances that match or are similar to the circumstances under which the target image was taken, from the selected person images, and an image correction unit configured to correct the correction area including the missing area of ​​the selected target image using an image of an area of ​​the specific image that corresponds to the correction area. Effect of the Invention

[0015] According to the present invention, when a missing area is corrected, an image expressing a facial expression according to the shooting conditions can be obtained. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a block diagram of an image processing apparatus according to a first embodiment. [Diagram 2] FIG. 2 is a flowchart showing the image processing method according to the first embodiment. [Diagram 3] FIG. 3 is an explanatory diagram showing a target image. [Figure 4] FIG. 4 is a flowchart showing the personal image selection process according to the first embodiment. [Diagram 5] FIG. 5 is an explanatory diagram showing a reference image. [Figure 6] FIG. 6 is a flowchart illustrating the specific image selection process according to the first embodiment. [Figure 7] FIG. 7 is an explanatory diagram showing the target image after correction. [Figure 8] FIG. 8 is a flowchart showing the personal image selection process according to the second embodiment. [Figure 9] FIG. 9 is a flowchart illustrating the specific image selection process according to the second embodiment. [Figure 10] FIG. 10 is a flowchart showing another specific image selection process according to the second embodiment. [Figure 11] FIG. 11 is a block diagram of an image processing system according to the third embodiment. As shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Next, an embodiment of the present invention will be described with reference to the drawings. [First embodiment] First, with reference to FIG. 1, an image processing device 10 according to a first embodiment of the present invention will be described. When taking pictures of people at events, etc., there are cases where missing areas of the person's face are not visible due to the wearing of masks, sunglasses, eyepatches, scarves, etc., resulting in an image in which the person's facial expression is unclear. This is because the missing areas occur in the mouth and eyes, which are important for recognizing facial expressions. On the other hand, the facial expressions of people when taking an image are greatly affected by the shooting conditions. For example, images taken at amusement parks tend to show many people with happy expressions, while images taken while studying or working tend to show many people with concentrated expressions. Therefore, if the shooting conditions of images are the same or similar, the facial expressions of people in the images tend to be similar.

[0018] The present invention focuses on the fact that there is a certain relationship between the photographing conditions of such an image and the facial expression of a person, and selects an image whose photographing conditions match or are similar to those of the target image from among images containing the face of the person himself / herself. That is, the image processing device 10 is configured to correct the missing area by using an image of the target person having a missing area that cannot be seen on the face, which is included in the input target image, and which was photographed under conditions that match or are similar to those under which the target image was photographed.

[0019] Therefore, when correcting missing areas using another image of the person himself, if the correction is made uniformly using a pre-prepared image that simply contains the person's face, the facial expression according to the shooting conditions will not be expressed and an image without a sense of realism will be generated. However, according to the present invention, it is possible to express facial expressions according to the shooting conditions and generate a realistic image.

[0020] [Image processing device] Next, a configuration of the image processing device 10 according to the present embodiment will be described. As shown in Fig. 1, the image processing device 10 is generally composed of an information processing device (computer) such as a PC or a server device, and includes a communication I / F 11, an operation input unit 12, a display unit 13, an input / output I / F 14, a storage unit 15, a control unit 16, and an accumulation unit 17 as main components.

[0021] [Communication I / F] The communication I / F 11 is connected to a communication network NW such as the Internet via a communication line L1, and is configured to perform data communication with various devices (not shown) connected to the communication network NW. For example, the communication I / F 11 connects to a smartphone via the communication network NW in response to an instruction from the control unit 16, acquires a target image that is to be subjected to image processing and is taken by the smartphone, and returns the corrected target image obtained by the image processing to the smartphone.

[0022] [Operation input section] The operation input unit 12 is made up of operation input devices such as a keyboard, a mouse, and a touch panel, and is configured to detect operations by a user and output the operations to the control unit 16 . [Display] The display unit 13 consists of a screen display device such as an LCD, and is configured to display operation screens such as operation menus output from the control unit 16, as well as target images to be subjected to image processing and corrected target images obtained by image processing.

[0023] [Input / output interface] The input / output I / F 14 is composed of an input / output I / F such as a USB, and is configured to perform data communication with the peripheral device D via the connection cable L2. For example, when the peripheral device D composed of a USB memory is connected, the input / output I / F 14 acquires a target image to be subjected to image processing stored in the peripheral device D in response to an instruction from the control unit 16, and stores the corrected target image obtained by the image processing in the peripheral device D.

[0024] [Storage] The storage unit 15 is made up of a storage device such as a semiconductor memory, and is configured to store various processing data and programs 15P used in the image processing executed by the control unit 16. The main processing data stored in the storage unit 15 is a target image 20. The target image 20 is an image that is to be subjected to image processing in the image processing device 10, and is acquired from an external device (not shown) via the communication I / F 11 and the input / output I / F 14 and stored in the storage unit 15.

[0025] The program 15P is a program that cooperates with the CPU of the control unit 16 to realize various processing units for executing image processing in the control unit 16. The program 15P is obtained from an external device or a recording medium (neither of which is shown) via the communication I / F 11 or the input / output I / F 14, and is stored in the storage unit 15 in advance.

[0026] [Control Unit] The control unit 16 has a CPU and its peripheral units, and is configured to cooperate with the program 15P in the storage unit 15 to realize various processing units for executing image processing. Main processing units realized by the control unit 16 include a processing control unit 16A, an image extraction unit 16B, a person's image selection unit 16C, a specific image selection unit 16D, and an image correction unit 16E.

[0027] [Processing control section] The processing control unit 16A is configured to control the overall image processing executed by each processing unit. This control includes, for example, a process of acquiring a target image 20 to be subjected to image processing from an externally connected device or equipment via the communication I / F 11 or the input / output I / F 14 and storing the image in the storage unit 15, and a process of returning the corrected target image 20 obtained by the image processing to the device or equipment.

[0028] [Image extraction section] The image extraction unit 16B is configured to extract an image of any area, such as a person's face, a background, and a corresponding area (to be described later), from a specified image. The image extraction process in the image extraction unit 16B may be performed using a general well-known image processing technique.

[0029] [Personal image selection section] The personal image selection unit 16C is configured to select an image of a target person having an unrecognizable missing area in the face as a personal image from among a plurality of reference images 30 stored in the storage unit 17. As a specific example of selecting a personal image using a face image, the image extraction unit 16B may be configured to extract an image of the target person's face from the target image 20, and the personal image selection unit 16C may compare the extracted target face image with each of the reference images 30 and select the reference image 30 including the target person's face as the personal image.

[0030] [Specific image selection section] The specific image selection unit 16D is configured to select, as a specific image, from among the personal images selected by the personal image selection unit 16C, an image that was taken under circumstances that match or are similar to the circumstances under which the target image 20 was taken. As a specific example of the process of selecting a specific image using a background image, the image extraction unit 16B may be configured to extract an image of the background of the target person from the target image 20, and the specific image selection unit 16D may compare the extracted target background image with each of the personal images to select, as a specific image, a personal image that includes a background that matches or is similar to the background of the target person.

[0031] [Image correction section] The image correction unit 16E is configured to correct the correction area including the missing area of ​​the target image 20 by using an image of a corresponding area corresponding to the correction area in the specific image selected by the specific image selection unit 16D. When the same area as the missing area is used as the correction area, the missing area of ​​the target image 20 is corrected by the image of the corresponding area in the specific image. A larger area including the missing area may also be used as the correction area. For example, when the entire face area of ​​the target person is selected as the correction area, the entire face image of the target person including the missing area in the target image 20 is corrected by the face image of the target person in the specific image. As for specific image correction processing, in addition to processing to replace the correction area with the image of the corresponding area, additional processing such as adjusting the brightness and color of the replaced area according to the correction area and its surrounding area may be performed.

[0032] [Storage section] The storage unit 17 is generally composed of a storage device such as a semiconductor memory or a hard disk, and is configured to store in advance a plurality of reference images 30 including a person's face and photographed under various circumstances. These reference images 30 are images of the target person included in the target image 20, and are composed of images photographed under various circumstances such as the location of the photograph, the event at the time of photographing, or the facial expression at the time of photographing.

[0033] [Operation of the first embodiment] Next, the operation of the image processing device 10 according to the first embodiment will be described with reference to Fig. 2. Here, an example will be described in which a face image is used to select a person's image, and a background image is used to select a specific image.

[0034] First, in the control unit 16, the processing control unit 16A acquires a target image 20 to be subjected to image processing from an externally connected device or equipment via the communication I / F 11 or the input / output I / F 14, and stores it in the memory unit 15 (step 100). As shown in Fig. 3, the target image 20 includes a person image 20A of a target person. The person image 20A includes a face image 20B corresponding to a face portion, and a background image 20D is included as the background of the person image 20A. In the example of Fig. 3, the face image 20B includes a missing area 20C that cannot be seen due to an obstruction such as a mask. This example also shows an image in which the shooting situation is an amusement park indicating the shooting location, and the amusement park is shown in the background of the target person.

[0035] Next, the personal image selection unit 16C selects, from among the plurality of reference images 30 stored in the storage unit 17, an image including the target person in the target image 20 as a personal image (step 101).

[0036] In the process of selecting the person's image in step 101, as shown in FIG. 4, first, the image extraction unit 16B extracts an image of the face of the target person from the target image 20 (step 110), and the person's image selection unit 16C compares the extracted target face image 20B with each of the reference images 30 (step 111) to select the reference image 30 containing the face of the target person as the person's image (step 112).

[0037] 5(a), 5(b), and 5(c), when reference images 31, 32, and 33 are stored in storage unit 17 as reference image 30, face image 20B in FIG. 2 is collated with each of reference images 31, 32, and 33. In this example, among reference images 31, 32, and 33, face images 31B and 32B similar to face image 20B of the target person are present in reference images 31 and 32, and face image 33B of reference image 33 is not similar to face image 20B. Therefore, as a result of collation, reference images 31 and 32 are selected as the person's image.

[0038] Next, the specific image selection unit 16D selects, from among the personal images selected by the personal image selection unit 16C, an image taken under circumstances that match or are similar to the circumstances under which the target image 20 was taken, as a specific image (step 102).

[0039] In the specific image selection process in step 102, as shown in FIG. 6, first, the image extraction unit 16B extracts an image of the background of the target person from the target image 20 (step 120), and the specific image selection unit 16D compares the extracted target background image with each of the person's images (step 121) to select the person's image that includes a background that matches or is similar to the background of the target person as the specific image (step 122). When multiple specific images are selected, the image with the highest similarity to the person's image may be selected as the specific image, or the multiple selected specific images may be displayed on the display unit 13, and any one of them selected by the user using the operation input unit 12 may be selected as the specific image.

[0040] When reference images 31 and 32 in Fig. 5(a) and 5(b) are selected as the person's image, background image 20D in Fig. 3 is matched with each of reference images 31 and 32. In this example, of reference images 31 and 32, reference image 31 contains background image 31D of an amusement park that is similar to background image 20D of the target person, and background image 32D of reference image 32 shows a person studying and is not similar to background image 20D. Therefore, as a result of matching, reference image 31 is selected as the specific image from among the person's images.

[0041] Next, the image extraction unit 16B extracts an image of a corresponding area used to correct the missing area 20C from the specific image selected by the specific image selection unit 16D. Thereafter, the image correction unit 16E corrects the correction area including the missing area 20C of the target image 20 using the image of the corresponding area extracted by the image extraction unit 16B (step 103), and ends the series of processes.

[0042] Therefore, as shown in FIG. 7, when the same area as the missing area 20C is used as the correction area, the image of the corresponding area 31C is extracted from the reference image 31, which is a specific image, by the image extraction unit 16B. In addition, the image of the extracted corresponding area 31C is used to correct the missing area 20C in the target image 20 by the image correction unit 16E. In the example of FIG. 7, the image of the corresponding area 31C in the reference image 31, which was previously taken under the circumstances in which the target image 20 was taken, in this case, under circumstances that match or are similar to an amusement park, is replaced with the missing area 20C in the target image 20. As a result, it can be seen that the corrected target image 20 expresses a smiling expression according to the shooting circumstances, and an image with a sense of realism is obtained.

[0043] As for specific image correction processing, for example, in addition to processing to replace or replace the correction area with the image of the corresponding area, additional processing such as adjusting the brightness and color of the replaced area according to the correction area and its surrounding area may be performed. Also, a larger area including the missing area 20C may be used as the correction area. For example, when the entire face image 20B of the target person is selected as the correction area, the entire face image 20B of the target person including the missing area 20C in the target image 20 is corrected with the face image 20B of the target person in the specific image.

[0044] [Advantages of the first embodiment] In this manner, in the control unit 16, the present embodiment is configured such that the personal image selection unit 16C selects a personal image including the face of the target person from among a plurality of reference images 30 pre-stored in the storage unit 17, the specific image selection unit 16D selects, from among the selected personal images, an image taken under circumstances that match or are similar to the circumstances under which the target image 20 was taken as a specific image, the image extraction unit 16B extracts an image of a corresponding area to be used to correct the missing area 20C from the selected specific image, and the image correction unit 16E corrects the missing area 20C using the image of the extracted corresponding area.

[0045] When the missing area is corrected by another image of the person as in the past, if the image is corrected uniformly by a prepared image simply including the person's face, the facial expression according to the shooting situation is not expressed, and an image without a sense of realism is generated. However, according to the present invention, from among the images of the person including the face of the person, an image whose shooting situation is the same as or similar to the target image 20 is selected as a specific image, and the missing area 20C is corrected using this specific image, so that the facial expression according to the shooting situation can be expressed, and an image with a sense of realism can be obtained.

[0046] In addition, in this embodiment, when selecting a personal image, the image extraction unit 16B extracts a target face image 20B showing the face of the target person from the target image 20, and the personal image selection unit 16C selects a personal image including the target person from among the multiple reference images 30 by comparing the target face image 20B extracted by the image extraction unit 16B with each of the multiple reference images 30. This allows the personal image including the target person to be selected only by image processing. Therefore, it is possible to omit preparatory work such as work to identify and classify people included in the target image 20 and the multiple reference images 30 in advance, and it is possible to reduce the workload required to correct the missing area 20C.

[0047] In addition, in this embodiment, when selecting a specific image, the image extraction unit 16B extracts a target background image 20D showing the background of the target person from the target image 20, and the specific image selection unit 16D compares the target background image 20D extracted by the image extraction unit 16B with each of the person's images to select an image whose background matches or is similar to the target background image 20D as a specific image from among the person's images. This makes it possible to select a specific image to be used for correcting the missing area 20C only by image processing. Therefore, it is possible to omit preparatory work such as work to specify and classify the shooting conditions of the target image 20 and the multiple reference images 30 in advance, and it is possible to reduce the workload required for correcting the missing area 20C.

[0048] [Second embodiment] Next, an image processing device 10 according to a second embodiment of the present invention will be described. In the first embodiment, a case where a target face image 20B and a target background image 20D extracted from a target image 20 are used when selecting a person's image or a specific image is described as an example. In the present embodiment, a case where a person keyword and a situation keyword are used instead of the target face image 20B and the target background image 20D is described as an example.

[0049] The person keyword is a keyword indicating a person included in the target image 20 or the multiple reference images 30, and specifically, is composed of a keyword specific to the person, such as the name of the person. The situation keyword is a keyword indicating the situation in which the target image 20 or the multiple reference images 30 was taken, and specifically, is composed of a keyword indicating the place or background in which the image was taken, or a keyword indicating the facial expression or pose of the person included in the image. In this embodiment, these keywords are assigned in advance to the target image 20 or the multiple reference images 30, and the person's image or a specific image is selected by comparing these keywords.

[0050] [Image processing device] The configuration of the image processing device 10 according to this embodiment is different from that of the first embodiment described above in that the configurations of the memory unit 15, the accumulation unit 17, the personal image selection unit 16C, and the specific image selection unit 16D are different. The other configurations are the same as those in FIG. 1 described above, and detailed description thereof will be omitted here.

[0051] [Storage] The storage unit 15 is configured to store, as processing data, person keywords indicating people included in the target image 20 and situation keywords indicating the situation in which the target image 20 was captured, together with the target image 20. These person keywords and situation keywords may be stored together with the target image 20 when the target image 20 is acquired from an external device (not shown) via the communication I / F 11 or the input / output I / F 14 and stored in the storage unit 15. Alternatively, keywords input from the operation input unit 12 by a user operation may be acquired and stored in the storage unit 15 before or after the target image 20 is stored.

[0052] [Storage section] The storage unit 17 is configured to store, together with the plurality of reference images 30, person keywords indicating people included in the reference images 30 and situation keywords indicating the situation in which the reference images 30 were captured for each of the plurality of reference images 30. These person keywords and situation keywords may be stored together with the reference images 30 when the reference images 30 are stored in the storage unit 17. Alternatively, keywords inputted from the operation input unit 12 by a user operation may be acquired and stored in the storage unit 17 before or after the reference images 30 are stored.

[0053] [Personal image selection section] The person image selection unit 16C is configured to search among the multiple reference images 30 for an image to which a person keyword matching the person keyword of the target person obtained from the memory unit 15 has been assigned, and select the image as the person image.

[0054] [Specific image selection section] The specific image selection unit 16D is configured to search for an image to which a situation keyword that matches or is similar to the situation keyword of the target image 20 acquired from the storage unit 15 has been added from among the personal images selected by the personal image selection unit 16C, and select the image as a specific image. Since the situation keyword is a keyword that indicates the situation in which the image was taken, subtle differences in expression tend to occur. For this reason, when searching for a personal image from the reference image 30 using the situation keyword of the target image 20, by searching not only for the matching situation keyword but also for images to which a similar situation keyword has been added, subtle differences in expression between keywords can be absorbed and made equivalent.

[0055] [Operation of the second embodiment] Next, the operation of the image processing device 10 according to this embodiment will be described with reference to the above-mentioned Fig. 2. The operation of the image processing device 10 according to this embodiment differs from that of the first embodiment described in Fig. 2 in the process of selecting a person's image in step 101 and the process of selecting a specific image in step 102, but the processes in the other steps are the same, and detailed description thereof will be omitted here.

[0056] In the process of selecting a personal image in step 101 of FIG. 2, as shown in FIG. 8, the personal image selection unit 16C acquires a person keyword for the target person from the memory unit 15 (step 130), searches multiple reference images 30 for the acquired person keyword (step 131), and selects an image to which a person keyword matching the acquired person keyword has been assigned as a personal image (step 132).

[0057] Furthermore, in the specific image selection process in step 102 of FIG. 2, as shown in FIG. 9, the specific image selection unit 16D acquires a situation keyword for the target image 20 from the memory unit 15 (step 140), searches for the personal image selected by the personal image selection unit 16C using the acquired situation keyword (step 141), and selects an image to which a situation keyword that matches or is similar to the acquired situation keyword has been assigned as a specific image (step 142). When multiple specific images are selected, the image having the highest similarity to the acquired situation keyword may be selected as the specific image, or the multiple selected specific images may be displayed on the display unit 13, and any one of them selected by the user using the operation input unit 12 may be selected as the specific image.

[0058] [Advantages of the second embodiment] In this manner, in this embodiment, when selecting a personal image, the personal image selection unit 16C searches among the multiple reference images 30 for an image to which a person keyword matching the person keyword of the target person acquired from the storage unit 15 has been assigned, and selects the image as the personal image. Also, when selecting a specific image, the specific image selection unit 16D searches among the personal images selected by the personal image selection unit 16C for an image to which a situation keyword matching or similar to the situation keyword of the target image 20 acquired from the storage unit 15 has been assigned, and selects the image as the specific image.

[0059] As a result, the image of the person in question is selected from among multiple reference images 30 using the person keyword, so that the reference image 30 of the person in question included in the target image 20 can be accurately selected, and image processing such as matching of image data can be omitted in the image of the person in question selection process, thereby reducing the processing burden on the control unit 16 and shortening the processing time. In addition, since a specific image is selected from among the images of the person using the situation keyword, it is possible to accurately select an image of the person that matches or is similar to the shooting situation of the target image 20, and it is possible to omit image processing such as matching image data from the specific image selection process. Therefore, it is possible to reduce the processing load on the control unit 16 and shorten the processing time.

[0060] The personal image selection process and the specific image selection process are not limited to the combination of the processes described in the first and second embodiments, and any combination can be adopted. For example, the personal image selection process of Fig. 4 according to the first embodiment may be combined with the specific image selection process of Fig. 9 according to the second embodiment, or the personal image selection process of Fig. 8 according to the second embodiment may be combined with the specific image selection process of Fig. 6 according to the first embodiment.

[0061] In the specific image selection process of FIG. 9, the case where the situation keyword of the input target image 20 is stored in the storage unit 15 has been described as an example, but the present invention is not limited to this. For example, the storage unit 17 stores in advance a plurality of reference background images obtained by photographing various backgrounds, and also stores in advance, for each of these reference background images, a situation keyword corresponding to the background of the reference background image.

[0062] In the specific image selection process, as shown in FIG. 10, the image extraction unit 16B extracts a target background image from the target image 20 (step 150), and the specific image selection unit 16D compares the target background image with each of a plurality of reference background images in the storage unit 17 (step 151), selects a reference background image whose background is similar to that of the target background image (step 152), and obtains a situation keyword of the selected reference background image (step 153). Next, the specific image selection unit 16D searches for the personal image selected by the personal image selection unit 16C using the acquired situation keyword (step 154), and selects an image that has been assigned a situation keyword that matches or is similar to the acquired situation keyword as a specific image (step 155).

[0063] According to this configuration, since the person himself / herself does not need to be included in the reference background image, it is possible to estimate the situation keyword of the target image 20 based on many images, thereby improving the estimation accuracy. Also, since it is not necessary to input the situation keyword of the target image 20, it is possible to reduce the operational burden on the user when correcting the missing area 20C.

[0064] [Third embodiment] Next, an image processing system 5 according to a third embodiment of the present invention will be described with reference to Fig. 11. In the first and second embodiments, the image processing device 10 operates independently. In the present embodiment, a system in which the storage unit 17 of the image processing device 10 is separated into an independent device will be described as an example.

[0065] [Image processing system] As shown in FIG. 11, the image processing system 5 is composed of an image processing device 50 and a storage device 60, and the image processing device 50 is connected to the storage device 60 so as to be able to communicate data with each other via a communication line L5 and a communication network NW.

[0066] [Image processing device] The image processing device 50 is generally composed of an information processing device (computer) such as a PC or a server device, and includes, as its main components, a communication I / F 51, an operation input unit 52, a display unit 53, an input / output I / F 54, a storage unit 55, and a control unit 56. These components correspond to the communication I / F 11, the operation input unit 12, the display unit 13, the input / output I / F 14, the storage unit 15, and the control unit 16 in the above-mentioned FIG.

[0067] [Communication I / F] The communication I / F 51 is connected to a communication network NW such as the Internet via a communication line L5, and is configured to perform data communication with various devices (not shown) connected to the communication network NW. For example, the communication I / F 51 connects to a smartphone via the communication network NW in response to an instruction from the control unit 56, acquires a target image 20 that is to be subjected to image processing and that was taken by the smartphone, and returns the corrected target image 20 obtained by the image processing to the smartphone.

[0068] [Operation input section] The operation input unit 52 is made up of operation input devices such as a keyboard, a mouse, and a touch panel, and is configured to detect operations by the user and output them to the control unit 56 . [Display] The display unit 53 consists of a screen display device such as an LCD, and is configured to display operation screens such as operation menus output from the control unit 56, as well as the target image 20 that is the subject of image processing, and the corrected target image 20 obtained by image processing.

[0069] [Input / output interface] The input / output I / F 54 is composed of an input / output I / F such as a USB, and is configured to perform data communication with the peripheral device D via a connection cable L6. For example, when the peripheral device D composed of a USB memory is connected, the input / output I / F 54 acquires the target image 20 to be subjected to image processing stored in the peripheral device D in response to an instruction from the control unit 56, and stores the corrected target image 20 obtained by the image processing in the peripheral device D.

[0070] [Storage] The storage unit 55 is made up of a storage device such as a semiconductor memory, and is configured to store various processing data and programs 55P used in the image processing executed by the control unit . The main processing data stored in the storage unit 55 is a target image 20. The target image 20 is an image that is to be subjected to image processing in the image processing device 50, and is acquired from an external device (not shown) via the communication I / F 51 and the input / output I / F 54 and stored in the storage unit 55.

[0071] The program 55P is a program that cooperates with the CPU of the control unit 56 to realize various processing units for executing image processing in the control unit 56. The program 55P is obtained from an external device or a recording medium (neither of which is shown) via the communication I / F 51 or the input / output I / F 54, and is stored in the storage unit 55 in advance.

[0072] [Control Unit] The control unit 56 has a CPU and its peripheral units, and is configured to cooperate with the program 55P in the storage unit 55 to realize various processing units for executing image processing. The main processing units realized by the control unit 56 are a processing control unit 56A, an image extraction unit 56B, a person's image selection unit 56C, a specific image selection unit 56D, and an image correction unit 56E. These processing units correspond to the processing control unit 16A, the image extraction unit 16B, the person's image selection unit 16C, the specific image selection unit 16D, and the image correction unit 16E in FIG. 1 described above.

[0073] Of these components, the processing control unit 56A is configured to access the storage device 60 via the communication I / F 51, the communication line L5, and the communication network NW when referring to the reference image 30. The configuration other than the access to the storage device 60 is similar to that of the first and second embodiments described above, and detailed description thereof will be omitted here.

[0074] [Storage device] The storage device 60 is generally composed of a data storage device such as a database device, and includes a storage unit 67 which corresponds to the storage unit 17 in FIG. The storage unit 67 is generally composed of a storage device such as a semiconductor memory or a hard disk, and is configured to store in advance a plurality of reference images 30 including a person's face and photographed under various circumstances. These reference images 30 are images of the target person included in the target image 20, and are composed of images photographed under various circumstances such as the location of the photograph, the event at the time of photographing, or the facial expression at the time of photographing.

[0075] [Advantages of the third embodiment] Thus, the image processing system 5 of this embodiment comprises an image processing device 50 and a storage device 60, and is configured so that the image processing device 50 is connected to the storage device 60 via a communication line L5 and a communication network NW so as to enable data communication therebetween. This allows the reference images 30 to be stored in a large-scale storage device 60, making it possible to store reference images 30 that include more people and backgrounds, thereby improving the quality of correction of missing areas 20C. In addition, the same storage device 60 can be accessed from multiple image processing devices 50, greatly improving the scalability of the hardware and software used to correct missing areas 20C.

[0076] [Extended embodiment] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In addition, the respective embodiments can be implemented in any combination within a range that does not contradict each other. [Explanation of symbols]

[0077] 10,50...image processing device, 11,51...communication I / F, 12,52...operation input unit, 13,53...display unit, 14,54...input / output I / F, 15,55...storage unit, 15P,55P...program, 16,56...control unit, 16A,56A...processing control unit, 16B,56B...image extraction unit, 16C,56C...personal image selection unit, 16D,56D...specific image selection unit, 16E,56E...image correction unit, 17 ,67...storage unit, 20...target image, 20A...person image, 20B...face image, 20C...missing area, 20D...background image, 31, 32, 33...reference image, 31A, 32A, 33A...person image, 31B, 32B, 33B...face image, 31C...corresponding area, 31D, 32D, 33D...background image, 60...storage device, L1, L5...communication line, L2, L6...connection cable, NW...communication network, D...peripheral device.

Claims

1. a storage unit configured to store in advance a plurality of reference images including a human face and taken under various conditions; A storage unit that stores a target image including a target person having a missing area that cannot be visually recognized on the face; a control unit configured to select a specific image including a face of the target person from the plurality of reference images stored in the storage unit, and to correct the missing area of ​​the target image using the specific image; The control unit is a person-image selection unit configured to select a person-image including a face of the target person from among the plurality of reference images; a specific image selection unit configured to select, from among the selected person images, an image taken under a situation that matches or is similar to a situation under which the target image was taken, as the specific image; an image extracting unit configured to extract an image of a corresponding area used for correcting the missing area from the selected specific image; and an image correction unit configured to correct the missing region using the extracted image of the corresponding region.

13. An image processing device comprising:

2. 2. The image processing device according to claim 1, The image extraction unit extracts a target face image indicating a face of the target person from the target image, The personal image selection unit is configured to compare the target face image extracted by the image extraction unit with each of the plurality of reference images, and select a personal image including the target person from among the plurality of reference images.

13. An image processing device comprising:

3. 2. The image processing device according to claim 1, The storage unit further stores in advance, for each of the plurality of reference images, a person keyword indicating the person included in the reference image; The storage unit stores, together with the target image, a person keyword indicating the target person included in the target image; The personal image selection unit selects, from among the plurality of reference images, a reference image to which a personal keyword that matches the personal keyword stored in the storage unit is assigned as the personal image.

13. An image processing device comprising:

4. In the image processing device according to any one of claims 1 to 3, The image extraction unit extracts a target background image indicating a background of the target person from the target image, The specific image selection unit compares the target background image extracted by the image extraction unit with each of the personal images, and selects an image of the personal image whose background matches or is similar to the target background image as the specific image.

13. An image processing device comprising:

5. In the image processing device according to any one of claims 1 to 3, The storage unit further stores, for each of the plurality of reference images, a situation keyword that indicates a situation in which the reference image was captured; The storage unit stores, together with the target image, a situation keyword indicating a situation in which the target image was captured; The specific image selection unit selects, from among the personal images, a personal image to which a situation keyword that matches or is similar to a situation keyword stored in the storage unit is assigned as the specific image.

13. An image processing device comprising:

6. In the image processing device according to any one of claims 1 to 3, The storage unit further stores in advance a plurality of reference background images obtained by photographing various backgrounds, and stores in advance, for each of the plurality of reference background images, a situation keyword corresponding to the background of the reference background image; The image extraction unit extracts a target background image indicating a background of the target person from the target image, The specific image selection unit compares the target background image extracted by the image extraction unit with each of the plurality of reference background images, and acquires, from among the plurality of reference background images, a situation keyword of a reference background image whose background matches or is similar to that of the target background image, as the situation keyword indicating a situation in which the target image was captured.

13. An image processing device comprising:

7. An image processing method for use in a computer, the method comprising: a storage unit configured to store in advance a plurality of reference images including a person's face and captured under various conditions; a memory unit configured to store a target image including a target person having a missing area that cannot be seen in the face; and a control unit configured to select a specific image including the face of the target person from the plurality of reference images stored in the storage unit, and to correct the missing area of ​​the target image using the specific image, a person-image selecting step in which the control unit selects a person-image including a face of the target person from among the plurality of reference images; A specific image selection step in which the control unit selects, from among the selected person images, an image taken under a situation that matches or is similar to a situation in which the target image was taken, as the specific image; an image extraction step in which the control unit is configured to extract an image of a corresponding area used for correcting the missing area from the selected specific image; and an image correction step in which the control unit is configured to correct the missing area using an image of the extracted corresponding area.

13. An image processing method comprising:

8. An image processing device and a storage device connected to each other so as to be capable of data communication via a communication network, the storage device is configured to store in advance a plurality of reference images including a human face and taken under various conditions; The image processing device includes a memory circuit configured to store a target image including a target person having a missing area that cannot be visually recognized on the face, and a control unit configured to select a specific image including the face of the target person from the plurality of reference images stored in the storage device, and to correct the missing area of ​​the target image using the specific image; The control unit is a person-image selection unit that selects a person-image including a face of the target person from among the plurality of reference images; a specific image selection unit configured to select, from among the selected person images, an image taken under a situation that matches or is similar to a situation under which the target image was taken, as the specific image; and an image correction unit configured to correct a correction area including the missing area in the selected target image by using an image of a region in the specific image corresponding to the correction area.

1. An image processing system comprising:

Images