Image processing device, image processing method, and program

Abstract

To prevent a nonexistent person from becoming similar to an existent person in a case of processing an image showing the nonexistent person.SOLUTION: An image processing device comprises: an acquisition unit which acquires a first material image and a second material image being face images showing the face of a person; a MIX image generation unit which generates a MIX image in which a first feature amount being the feature amount of the first material image and a second feature amount being the feature amount of the second material image are mixed by using the first material image and the second material image; a similarity determination unit which determines whether or not the MIX image is similar to the first material image; and a device control unit which causes the MIX image generation unit to generate the MIX image that is determined not to be similar to the first material image by the similarity determination unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] Currently, a technology for generating an image (face image) showing a face of a non-existent person has become widespread. Such a face image of a non-existent person is used by many companies from the viewpoint that it is not necessary to obtain permission to use an image that may occur when using a face image of an existent person. For example, it is conceivable to use a GAN (Generative Adversarial Network) to generate a face image that is not similar to any of a large number of face images used for learning as a non-existent person. For example, Patent Document 1 discloses a technique for improving learning efficiency based on the similarity between an image output during the learning process in GAN learning and an image for which an output is expected.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, during the image processing process, there were cases where the facial images of non-existent people ended up resembling the faces of real people. For example, when processing the gaze and face orientation of a non-existent person's facial image according to a company's needs, the gaze and face orientation of a real person's facial image are used to generate a natural, high-quality image that doesn't look unnatural. In such cases, image processing is performed to move the landmarks (feature points) of the non-existent person's facial image according to the landmarks of the real person's facial image. As a result, even when using a facial image of a non-existent person, the landmarks of the real person remain strongly in the processed facial image, sometimes resulting in the generation of an image with a high degree of similarity to a real person.

[0005] The present invention has been made to solve the above problems, and its objective is to provide an image processing apparatus, an image processing method, and a program that can process images showing non-existent people in a way that prevents them from resembling real people. [Means for solving the problem]

[0006] To solve the above problem, one aspect of the present invention includes: an acquisition unit that acquires a first material image and a second material image which are face images showing a person's face; a MIX image generation unit that uses the first material image and the second material image to generate a MIX image in which a first feature quantity which is a feature quantity of the first material image and a second feature quantity which is a feature quantity of the second material image are mixed; a similarity determination unit that determines whether the MIX image and the first material image are similar; and a device control unit that causes the MIX image generation unit to generate a MIX image which the similarity determination unit determines is not similar. The MIX image generation unit generates a MIX image in which the first feature quantity and the second feature quantity are mixed at a predetermined mixing ratio, and further comprises a threshold determination unit that determines as a threshold the mixing ratio in the image with the largest proportion of the first feature quantity among the MIX images that the similarity determination unit has determined to be dissimilar, and the device control unit causes the MIX image generation unit to generate the MIX image by specifying the mixing ratio in which the proportion of the second feature quantity is greater than the threshold. It is an image processing device.

[0007] Furthermore, one aspect of the present invention is an image processing method performed by a computer image processing device, wherein an acquisition unit acquires a first material image and a second material image, which are face images showing a person's face; a MIX image generation unit generates a MIX image using the first material image and the second material image, in which a first feature quantity, which is a feature quantity of the first material image, and a second feature quantity, which is a feature quantity of the second material image, are mixed; a similarity determination unit determines whether the MIX image and the first material image are similar; and a device control unit causes the MIX image generation unit to generate the MIX image that the similarity determination unit has determined to be insufficient. The MIX image generation unit generates a MIX image in which the first feature quantity and the second feature quantity are mixed at a predetermined mixing ratio, the threshold determination unit determines the mixing ratio in the image with the largest proportion of the first feature quantity among the MIX images that the similarity determination unit has determined to be dissimilar as the threshold, and the device control unit specifies the mixing ratio in which the proportion of the second feature quantity is greater than the threshold and causes the MIX image generation unit to generate the MIX image. This is an image processing method.

[0008] Furthermore, in one aspect of the present invention, an image processing device, which is a computer, is made to acquire a first material image and a second material image, which are face images showing a person's face; a MIX image is generated using the first material image and the second material image, which are a mixture of a first feature quantity, which is a feature quantity of the first material image and a second feature quantity, which is a feature quantity of the second material image; a determination is made as to whether the MIX image and the first material image are similar; and a MIX image is generated that is determined to be dissimilar. The system generates a mixed image in which the first and second features are mixed at a predetermined mixing ratio, determines the mixing ratio in the image with the largest proportion of the first feature among the mixed images that are determined to be dissimilar as a threshold, and generates a mixed image by specifying a mixing ratio in which the proportion of the second feature is greater than the threshold. It is a program. [Effects of the Invention]

[0009] According to this invention, when processing an image of a non-existent person, it is possible to make it so as not to resemble a real person. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram showing the configuration of the image processing apparatus 10 of the embodiment. [Figure 2] This figure shows an example of the MIX image information 120 of the embodiment. [Figure 3] This figure shows an example of the composite image information 121 of the embodiment. [Figure 4] This diagram illustrates the processing performed by the image processing device 10 of this embodiment. [Figure 5]This flowchart shows the processing flow performed by the image processing device 10 of the embodiment. [Figure 6] This flowchart shows the processing flow performed by the image processing device 10 of the embodiment. [Figure 7] This is a block diagram showing the configuration of an image processing system 1 to which the image processing apparatus 10 of this embodiment is applied. [Figure 8] This figure illustrates the processing performed by the image processing device 10 according to a modified embodiment. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described below with reference to the drawings. In the following description, an example will be given of processing an image showing the face of a non-existent person (face image) using a face image of a real person, but the invention is not limited to this. For example, this embodiment can also be applied when processing a face image of a real person using a face image of another real person.

[0012] Figure 1 is a block diagram showing the configuration of an image processing device 10 according to an embodiment. The image processing device 10 is a computer such as a server device, cloud server, or PC (Personal Computer). The image processing device 10 is used for image processing. As shown in Figure 1, the image processing device 10 includes, for example, a communication unit 11, a storage unit 12, a control unit 13, and a display unit 14.

[0013] The communication unit 11 communicates with external devices and external storage media. The storage unit 12 is composed of storage media such as HDD, flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), RAM (Random Access read / write Memory), ROM (Read Only Memory), or a combination thereof. The storage unit 12 stores programs for executing various processes of the image processing device 10, and temporary data used when performing various processes. The storage unit 12 stores MIX image information 120 and composite image information 121. The specific contents of this information stored in the storage unit 12 will be described later. The display unit 14 includes a display device such as a liquid crystal display and displays images according to the control of the control unit 13.

[0014] The control unit 13 is implemented by causing the CPU (Central Processing Unit) provided as hardware in the image processing device 10 to execute a program. The control unit 13 comprehensively controls the image processing device 10. The control unit 13 includes, for example, an acquisition unit 130, a mixed image generation unit 131, a similarity determination unit 132, a threshold determination unit 133, a synthesis unit 134, and a device control unit 135.

[0015] The acquisition unit 130 acquires image data of the first source image SG1 and the second source image SG2. For example, the first source image SG1 is a facial image of a real person. The second image is a facial image of a fictional person. The acquisition unit 130 acquires image data transmitted from an external device, or image data stored on a storage medium such as a USB (Universal Serial Bus) memory, via the communication unit 11, and uses the acquired image data as the image data of the first source image SG1 and the second source image SG2. The acquisition unit 130 outputs the acquired image data to the MIX image generation unit 131.

[0016] The MIX image generation unit 131 generates a MIX image. The MIX image here is an image in which the feature amount (first feature amount) of the first material image SG1 and the feature amount (second feature amount) of the second material image SG2 are mixed.

[0017] First, the MIX image generation unit 131 extracts the landmarks (feature points) of the face part in the first material image SG1. The landmarks of the face part are points indicating the positions of the parts showing the features of the face, for example, the positions of the inner corners of the eyebrows, the peaks of the eyebrows, the outer corners of the eyebrows, the inner corners of the eyes, the outer corners of the eyes, the pupils, the nasal bridge, the lips, the corners of the mouth, the contour, etc. The MIX image generation unit 131 performs the same process for the second material image SG2 and extracts the landmarks.

[0018] Next, the MIX image generation unit 131 determines the positions of the landmarks of the MIX image. For example, the MIX image generation unit 131 converts the magnification in the vertical and horizontal directions of the second material image SG2 so that the height and width of the face part in the second material image SG2 are the same as those of the first material image SG1. In this case, the MIX image generation unit 131 moves the positions of the landmarks to the positions corresponding to the conversion in the vertical and horizontal directions of the second material image SG2. The MIX image generation unit 131 determines the positions corresponding to the middle between the positions of the landmarks in the first material image SG1 and the positions of the converted landmarks in the second material image SG2 as the positions of the landmarks in the MIX image.

[0019] The height of the face here refers to, for example, the length from the arch of the eyebrow to the chin. For example, the MIX image generation unit 131 calculates the height of the face by calculating the distance from the straight line connecting the landmarks corresponding to the left and right arches of the eyebrows to the landmark furthest away (the landmark corresponding to the tip of the chin). The width of the face here refers to, for example, the length from the base of the left ear to the base of the right ear. For example, the MIX image generation unit 131 calculates the width of the face as the maximum value of the distance from the leftmost landmark (the landmark corresponding to the left contour) to the rightmost landmark (the landmark corresponding to the right contour) that is included in the straight line parallel to the straight line connecting the landmarks corresponding to the left and right arches of the eyebrows.

[0020] Next, the MIX image generation unit 131 divides the first source image SG1 into triangles formed by connecting landmarks in the first source image SG1. The MIX image generation unit 131 similarly divides the second source image SG2 and the MIX image into triangles formed by connecting landmarks. As a result, the face portion of the image is divided into multiple polygons.

[0021] Next, the MIX image generation unit 131 moves the position of landmarks in the MIX image by deforming the shape of the triangles that divide the MIX image, based on the shape of the triangles that divide the first material image SG1 and the second material image SG2.

[0022] In this case, the MIX image generation unit 131 deforms the shape of the triangles that divide the MIX image according to a predetermined mixing ratio. The mixing ratio here is the ratio in which the features of the first source image SG1 (first features) and the features of the second source image SG2 (second features) are mixed into the MIX image. For example, if the mixing ratio is 10%, the MIX image will contain 90% of the first features and 10% of the second features. If the mixing ratio is 55%, the MIX image will contain 45% of the first features and 55% of the second features. If the mixing ratio is 70%, the MIX image will contain 30% of the first features and 70% of the second features.

[0023] For example, if the mixing ratio is 10%, the MIX image generation unit 131 moves the position of the vertices of the triangles that divide the MIX image from a position midway between the position of the landmark in the first source image SG1 and the position of the landmark in the second source image SG2 (50%) to a position 90% closer to the position of the landmark in the first source image SG1. As a result, the MIX image generation unit 131 generates a MIX image in which the first feature is included at 90% and the second feature is included at 10%.

[0024] Furthermore, when the MIX image generation unit 131 deforms the shape of a triangle by a mixing ratio, it may adjust the mixing ratio according to the position of the triangle. For example, in some parts of the face, such as the pupils, mixing the first and second feature quantities may degrade the quality of the MIX image. To avoid this situation, the MIX image generation unit 131 may, for example, set the mixing ratio to 0% for the pupil area and apply the same image to the pupil area in the first source image SG1 as the pupil area in the MIX image. This makes it possible to suppress the degradation of the quality of the MIX image.

[0025] The MIX image generation unit 131 deforms the shape of the triangles that divide the MIX image according to a predetermined mixing ratio, and the deformed MIX image is a MIX image generated with a predetermined mixing ratio.

[0026] The similarity determination unit 132 determines the similarity between the MIX image and the first source image SG1. The similarity determination unit 132 may use any prior art method to determine the similarity. For example, the similarity determination unit 132 determines the similarity between the MIX image and the first source image SG1 by comparing the positions of feature points, the color of the eyes, skin, and lips, etc., between the MIX image and the first source image SG1.

[0027] The threshold determination unit 133 determines a threshold. Here, the threshold is the mixing ratio in which the proportion of the first feature contained in the MIX image is largest when the MIX image and the first source image SG1 are not similar. The threshold determination unit 133 determines the threshold based on the mixing ratio of the MIX image generated by the MIX image generation unit 131 and the similarity between the MIX image and the first source image SG1 determined by the similarity determination unit 132.

[0028] The compositing unit 134 generates a composite image. The composite image is an image created by superimposing the face image from the MIX image onto the face image from the first source image SG1. The compositing unit 134 performs blurring on the forehead area of ​​the composite image, for example. Blurring is a process that transforms the color of an area so that it appears slightly blurred, similar to an image blurred by a camera. For example, the compositing unit 134 performs blurring by converting the color of the pixel area near a pixel to the average color of that area. Blurring the forehead area reduces the unnaturalness caused by the superimposition.

[0029] Furthermore, the compositing unit 134 adjusts the color of the areas in the composite image excluding the hair and background. For example, the compositing unit 134 uses a LUT (Lookup Table) to adjust the color. A LUT is a table that associates input values ​​with output values. By using the LUT, the compositing unit 134 converts the pixel values ​​(input values) in the composite image before color adjustment to the pixel values ​​(output values) after color adjustment. The LUT is determined, for example, according to the color of the area to be adjusted in the composite image and the color of the corresponding area in the first source image SG1.

[0030] The device control unit 135 comprehensively controls the image processing device 10. The device control unit 135 causes the MIX image generation unit 131 to generate a MIX image in which the ratio of the second feature quantity is greater than a threshold, and stores information about that MIX image as MIX image information 120 in the storage unit 12. This ensures that only MIX images that are not similar to the first source image SG1 are generated, so that when processing the second source image SG2 (an example of an image showing a non-existent person), the resulting image does not resemble the first source image SG1 (an example of an image showing an actual person).

[0031] Furthermore, the device control unit 135 causes the similarity determination unit 132 to determine the similarity between the composite image generated by the synthesis unit 134 and the first source image SG1, and stores information about the composite image as composite image information 121 in the storage unit 12 according to the determination result. Specifically, the device control unit 135 stores information about composite images that are determined not to be similar to the first source image SG1 as composite image information 121 in the storage unit 12. On the other hand, the device control unit 135 discards composite images that are determined to be similar to the first source image SG1, and prevents information about those images from being stored in the storage unit 12 as composite image information 121. This makes it possible to avoid similarity between a MIX image (an example of an image showing a non-existent person) and the first source image SG1 (an example of an image showing an actual person) when overlaying and compositing (an example of processing).

[0032] Figure 2 shows an example of MIX image information 120 of the embodiment. MIX image information 120 is generated for each combination of the first material image SG1 and the second material image SG2 in the MIX image. As shown in Figure 2, MIX image information 120 stores information corresponding to items such as image ID, MIX image, mixing ratio, threshold, and evidence. The image ID is identification information that uniquely identifies the MIX image. The MIX image is image data of the MIX image identified by the image ID. The mixing ratio is information indicating the mixing ratio of the MIX image identified by the image ID. The threshold is information indicating the threshold of the MIX image identified by the image ID. The evidence is information indicating the evidence (proof) of the MIX image identified by the image ID. The evidence includes, for example, information about the first material image SG1, information about the second material image SG2, information about the mixing ratio, and information about the similarity determination method. Information about the material images includes, for example, image data, source of the image, and information indicating whether the subject is a real person or not. Information regarding the mixing ratio indicates what mixing ratio was used, for example, whether or not a mixing ratio was applied to the pupil area. Information regarding the similarity determination method indicates what similarity determination method was used. By storing this kind of evidence, it is possible to objectively demonstrate that the MIX image is not similar to the first source image SG1.

[0033] Figure 3 shows an example of composite image information 121 in the embodiment. Composite image information 121 is generated for each composite image. As shown in Figure 3, the storage unit 12 stores information corresponding to items such as image ID, composite image, and evidence. The image ID is identification information that uniquely identifies the composite image. The composite image is image data of the composite image identified by the image ID. The evidence is information that shows evidence (proof) for the composite image identified by the image ID. The evidence includes, for example, information about the MIX image, information showing the similarity determination result of the composite image, and information about the similarity determination method. The information about the MIX image corresponds to the evidence in the MIX image information 120. The information showing the similarity determination result of the composite image is information that shows the similarity between the composite image and the first material image SG1. The information about the similarity determination method is information that shows what kind of similarity determination method was used. By storing such evidence, it is possible to objectively show that the composite image is not similar to the first material image SG1.

[0034] Figure 4 is a diagram illustrating the processing performed by the image processing device 10 of the embodiment. Figure 4 schematically shows how the image processing device 10 determines a threshold. The image processing device 10 repeatedly and continuously generates MIX images by, for example, changing the value of the mixing ratio P to increase the mixing ratio of the second feature included in the MIX image. Each time the image processing device 10 generates a MIX image, it determines the similarity between the generated MIX image and the first source image SG1. If the generated MIX image is similar to the first source image SG1, the image processing device 10 discards the MIX image. If the image processing device 10 determines that the MIX image generated by increasing the mixing ratio of the second feature included in the MIX image is not similar to the first source image SG1, it determines the mixing ratio in that MIX image as the threshold.

[0035] This example shows that the mixed image MG1 generated with a mixing ratio P=0.10 was determined to be similar to the first source image SG1. It also shows that the mixed image MG2 generated with a mixing ratio P=0.25 was determined to be similar to the first source image SG1. It also shows that the mixed image MG1 generated with a mixing ratio P=0.55 was determined to be dissimilar to the first source image SG1. It also shows that the mixed image MG1 generated with a mixing ratio P=0.70 was determined to be dissimilar to the first source image SG1. Finally, it shows that the mixed image MG1 generated with a mixing ratio P=0.95 was determined to be dissimilar to the first source image SG1. In this case, the image processing device 10 determines the mixing ratio P=0.55 as the threshold.

[0036] Figure 5 is a flowchart showing the processing flow of the image processing device 10 in the embodiment. Figure 5 shows the processing flow for determining the threshold. First, the image processing device 10 acquires the first source image and the second source image (step S10). Next, the image processing device 10 sets the mixing ratio P to an initial value (for example, 0.10) (step S11). The image processing device 10 generates a MIX image with the set mixing ratio (step S12). The image processing device 10 determines whether the generated MIX image is similar to the first source image SG1 (step S13).

[0037] If the MIX image generated in step S12 is similar to the first source image SG1, the image processing device 10 discards the MIX image generated in step S12 (step S14). The image processing device 10 increases the mixing ratio P (step S15). The image processing device 10 determines whether the value of the mixing ratio P increased in step S14 is greater than or equal to the upper limit (step S16). If the value of the mixing ratio P is less than the upper limit, the image processing device 10 returns to step S12 and generates a MIX image. On the other hand, if the value of the mixing ratio P is greater than or equal to the upper limit, the image processing device 10 terminates the process.

[0038] If the MIX image generated in step S12 is not similar to the first source image SG1, the image processing device 10 stores the MIX image and information associated with the mixing ratio in the storage unit 12 as MIX image information 120 (step S17). The image processing device 10 determines whether or not the threshold is undetermined (step S18). If the threshold is undetermined, the image processing device 10 determines the value of the mixing ratio P in the MIX image generated in step S12 as the threshold (step S19). The image processing device 10 stores the determined threshold as MIX image information 120 in the storage unit 12 (step S20).

[0039] Figure 6 is a flowchart showing the processing flow of the image processing device 10 in the embodiment. Figure 6 shows the processing flow for generating a composite image. First, the image processing device 10 acquires a first source image (step S30). Next, the image processing device 10 sets the mixing ratio P as a threshold (step S31). The image processing device 10 acquires a MIX image generated with the set mixing ratio (step S32). The image processing device 10 overlays the image of the face portion in the MIX image onto the face portion of the first source image SG1 (step S33), and generates a composite image by performing blur processing and color adjustment using a LUT (step S34).

[0040] The image processing device 10 determines whether the generated composite image is similar to the first source image SG1 (step S25). If the composite image is similar to the first source image SG1, the image processing device 10 discards the composite image (step S36). The image processing device 10 increases the mixing ratio P (step S37). The image processing device 10 determines whether the value of the mixing ratio P increased in step S37 is greater than or equal to the upper limit (step S38). If the value of the mixing ratio P is less than the upper limit, the image processing device 10 returns to step S32 and generates a composite image. On the other hand, if the value of the mixing ratio P is greater than or equal to the upper limit, the image processing device 10 terminates the process.

[0041] If the composite image is not similar to the first source image SG1, the image processing device 10 stores the composite image, along with information about the composite image, in the storage unit 12 as composite image information 121 (step S39).

[0042] As described above, the image processing apparatus 10 of this embodiment comprises an acquisition unit 130, a MIX image generation unit 131, a similarity determination unit 132, and a device control unit 135. The acquisition unit 130 acquires a first material image SG1 and a second material image SG2. The MIX image generation unit 131 generates a MIX image. The similarity determination unit 132 determines whether the MIX image and the first material image SG1 are similar or not. The device control unit 135 causes the MIX image generation unit 131 to generate a MIX image that the similarity determination unit 132 has determined to be dissimilar. As a result, the image processing apparatus 10 of this embodiment can generate a MIX image that is not similar to the first material image SG1. Therefore, when processing the second material image SG2 (an example of an image showing a fictitious person), it is possible to make it so that it is not similar to the first material image SG1 (an example of an image showing a fictitious person).

[0043] Furthermore, in the image processing apparatus 10 of the embodiment, the MIX image generation unit 131 generates a MIX image in which the first feature quantity and the second feature quantity are mixed at a predetermined mixing ratio. The image processing apparatus 10 further includes a threshold determination unit 133. The threshold determination unit 133 determines the mixing ratio of the MIX image in which the proportion of the first feature quantity is included is the largest among the MIX images determined to be dissimilar by the similarity determination unit 132 as the threshold. The device control unit 135 specifies a mixing ratio in which the proportion of the second feature quantity is included is greater than the threshold and causes the MIX image generation unit 131 to generate a MIX image. As a result, the image processing apparatus 10 of the embodiment can generate a MIX image according to the mixing ratio, and all MIX images in which the proportion of the second feature quantity is included is greater than the threshold, and can be made to be images that are not similar to the first source image SG1.

[0044] Furthermore, in the image processing apparatus 10 of this embodiment, the apparatus control unit 135 stores MIX image information 120, which associates the mixing ratio and threshold with identification information that identifies the MIX image generated by the MIX image generation unit 131. This allows the image processing apparatus 10 of this embodiment to objectively demonstrate that the MIX image is not similar to the first source image SG1.

[0045] Furthermore, the image processing apparatus 10 of the embodiment further includes a synthesis unit 134. The synthesis unit 134 generates a composite image. The composite image is an image obtained by superimposing the image of the face portion in the MIX image onto the face portion in the first source image SG1. As a result, the image processing apparatus 10 of the embodiment can generate a composite image using a MIX image that is not similar to the first source image SG1.

[0046] Furthermore, in the image processing apparatus 10 of this embodiment, the similarity determination unit 132 determines whether the composite image and the first source image SG1 are similar or not. The device control unit 135 stores the composite image that the similarity determination unit 132 has determined not to be similar to the first source image. As a result, the image processing apparatus 10 of this embodiment can store and retain only composite images that are not similar to the first source image SG1.

[0047] Furthermore, in the image processing apparatus 10 of this embodiment, the device control unit 135 discards composite images that the similarity determination unit 132 has determined to be similar to the first source image SG1, and instructs the synthesis unit 134 to regenerate a composite image using a MIX image in which the proportion of the second feature quantity is larger than that of the discarded composite image. As a result, the image processing apparatus 10 of this embodiment can discard composite images that are similar to the first source image SG1 and regenerate composite images that are not similar to the first source image SG1.

[0048] Furthermore, in the image processing apparatus 10 of this embodiment, the synthesis unit 134 converts the color of the image superimposed on the first material image in the synthesized image according to the color of the first material image SG1. As a result, the image processing apparatus 10 of this embodiment can make the superimposed image blend with the color of the first material image, and can generate a synthesized image with a natural appearance.

[0049] Here, we will describe an example of the application of the embodiment. Figure 7 is a block diagram showing the configuration of an image processing system 1 to which the image processing device 10 of the embodiment is applied. The image processing system 1 comprises an image processing device 10 and a plurality of customer terminals 20 (customer terminals 20-1, 20-2, ..., 20-N). N is any natural number. In this example, for example, the image processing system 1 provides an image processing service that utilizes the processing performed by the image processing device 10.

[0050] The customer terminal 20 is, for example, a computer such as a PC (Personal Computer), a tablet, or a smartphone. The customer terminal 20 is, for example, a terminal device for a customer of an image processing service. The customer terminal 20 sends, for example, a first source image to the image processing device 10 and requests the generation of a composite image. Here, the composite image requested by the customer terminal 20 is an image that reflects the gaze and face angle of the first source image, and in which the face portion is not similar to that of the first source image.

[0051] The image processing device 10 generates a composite image in response to a request from the customer terminal 20. The customer terminal 20 transmits the image data of the generated composite image, along with the evidence in the composite image information 121, back to the customer terminal 20. By transmitting the image data of the composite image along with the evidence to the customer terminal 20, the image processing system 1 can objectively demonstrate that the face portion in the composite image is not similar to the first source image, thereby ensuring the quality of the image processing service.

[0052] Here, a modified version of the embodiment will be described. This modified version differs from the embodiment described above in that the mixing ratio and the synthesis ratio are set individually. The synthesis ratio here is the ratio at which the MIX image is synthesized with the first material image when generating the composite image.

[0053] In the embodiment described above, it was assumed that the mixing ratio and the synthesis ratio were the same value. For example, if the mixing ratio is 90%, a MIX image is generated in which the feature quantities of the first source image are 10% and the feature quantities of the second source image are 90%. In this case, the synthesis process was configured to generate a composite image with a synthesis ratio of 90%, that is, a pixel value of 10% in the first source image and a pixel value of 90% in the MIX image.

[0054] In the embodiment described above, it was assumed that the faces in both the first source image and the mixed image were facing the same direction. For example, if the first source image was a picture of a face facing forward, it was assumed that the mixed image was also a picture of a face facing forward. Under these assumptions, the system was configured to generate a composite image during synthesis in which the ratio of the pixel values ​​of the first source image to the pixel values ​​of the mixed image was set to the same ratio as the blending ratio.

[0055] However, it was found that if the orientation of the face in the first source image and the mixed image are different, setting the mixing ratio and the composite ratio to the same value may result in an unnatural composite image. For example, if the first source image is a face facing slightly to the right, and the mixed image is a face facing forward, even though the outline of the face is facing to the right, a composite image may be generated in which the central parts of the face, such as the eyes, nose, and mouth, are facing forward, resulting in an unnatural composite image (see, for example, composite image GG1# in Figure 8).

[0056] As a countermeasure, in this modified version, the mixing ratio and the composite ratio are set separately. This makes it possible to generate a composite image that looks natural even when the orientation of the faces in the first source image and the mixed image are different. Therefore, for example, even when the mixed image is composited with a variation image of the first source image, such as an image of the same person as in the first source image but in a different pose, it is possible to generate a composite image that looks natural.

[0057] Figure 8 is a diagram illustrating the processing performed by the image processing apparatus 10 according to a modified embodiment. Figure 8 schematically shows the synthesis processing performed by the synthesis unit 134. Figure 8 shows the process of synthesizing the first source image SG1# and the MIX image MG4. The first source image SG1# is a first source image in which a person's face is facing to the right. On the other hand, the MIX image MG4 is an image in which a person's face is facing to the right, and is a MIX image with a mixing ratio of 90%.

[0058] As shown in Figure 8, when the first source image SG1# and the mixed image MG4 are combined at the same blending ratio as the mixing ratio, i.e., a blending ratio of 90%, an unnatural-looking composite image like the composite image GG1# is generated.

[0059] On the other hand, when the first source image SG1# and the mixed image MG4 are combined using a blending ratio set separately from the mixing ratio, for example, a blending ratio of 10%, a seamless blended image like the blended image GG2# can be generated. Similarly, even when combined using a blending ratio of 50%, a seamless blended image like the blended image GG3# can be generated.

[0060] In this way, by setting the synthesis ratio separately from the mixing ratio, it becomes possible to generate a composite image that does not look unnatural even when the orientation of the faces in the first source image and the mixed image are different. For example, the synthesis unit 134 generates multiple composite image groups by gradually increasing the synthesis ratio from 0% to 100%. The synthesis unit 134 then determines whether there is any unnaturalness in each of the generated composite images and whether or not it is similar to the first source image. The synthesis unit 134 selects a composite image from the multiple composite image groups that does not look unnatural and is not similar to the first source image. The synthesis unit 134 then uses the selected composite image as the final composite image.

[0061] As described above, in the image processing apparatus 10 according to a modified embodiment, the synthesis unit 134 sets the synthesis ratio separately from the mixing ratio when the orientation of the face in the first material image SG1# and the orientation of the face in the MIX image MG4 are different. The synthesis ratio is the ratio at which the MIX image is synthesized with the first material image when synthesizing the synthesized image GG. The synthesis unit 134 generates the synthesized image at the set synthesis ratio. As a result, the image processing apparatus 10 according to a modified embodiment can generate a synthesized image that does not look unnatural even when the orientation of the face in the first material image and the orientation of the face in the MIX image are different.

[0062] The image processing system 1 and customer terminal 20 in the above-described embodiment may be implemented using a computer. In that case, the program for implementing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into the computer system and executed. Here, "computer system" includes hardware such as the OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into the computer system. Moreover, "computer-readable recording medium" may also include those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside the computer system that acts as a server or client in that case. Furthermore, the above-mentioned program may be for implementing a part of the above-mentioned function, or it may be a program that can implement the above-mentioned function in combination with a program already recorded in the computer system, or it may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0063] While embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention. [Explanation of symbols]

[0064] 1...Image processing system, 10...Image processing device, 11...Communication unit, 12...Storage unit, 13...Control unit, 14...Display unit, 20...Customer terminal, 120...MIX image information, 121...Composite image information, 130...Acquisition unit, 131...MIX image generation unit, 132...Similarity determination unit, 133...Threshold determination unit, 134...Composite unit, 135...Device control unit

Claims

1. An acquisition unit acquires a first material image and a second material image, which are face images showing a person's face. A MIX image generation unit generates a MIX image in which a first feature quantity, which is a feature quantity of the first material image, and a second feature quantity, which is a feature quantity of the second material image, are mixed using the first material image and the second material image. A similarity determination unit that determines whether the MIX image and the first material image are similar, A device control unit that causes the MIX image generation unit to generate the MIX image which is determined to be dissimilar by the similarity determination unit, Equipped with, The MIX image generation unit generates the MIX image in which the first feature quantity and the second feature quantity are mixed at a predetermined mixing ratio. The system further comprises a threshold determination unit that determines the mixing ratio of the image containing the first feature quantity in the largest proportion among the MIX images determined to be dissimilar by the similarity determination unit as the threshold, The device control unit specifies the mixing ratio in which the proportion of the second feature quantity is greater than the threshold and causes the MIX image generation unit to generate the MIX image. Image processing device.

2. The device control unit stores MIX image information that associates the mixing ratio and the threshold value with identification information that identifies the MIX image generated by the MIX image generation unit. The image processing apparatus according to claim 1.

3. The system further includes a synthesis unit that generates a composite image by superimposing the image of the face portion in the MIX image onto the face portion in the first material image. The image processing apparatus according to claim 1.

4. The similarity determination unit determines whether the composite image and the first material image are similar, The device control unit stores the composite image that the similarity determination unit has determined is not similar to the first material image. The image processing apparatus according to claim 3.

5. The device control unit discards the composite image that the similarity determination unit has determined to be similar to the first material image, and uses the discarded composite image to cause the synthesis unit to regenerate the composite image using the MIX image in which the proportion of the second feature quantity is greater than that of the MIX image. The image processing apparatus according to claim 3 or claim 4.

6. The synthesis unit converts the color of the image superimposed on the first material image in the synthesized image according to the color of the first material image. The image processing apparatus according to claim 3.

7. The synthesis unit, when the orientation of the face in the first material image and the orientation of the face in the MIX image are different, sets a synthesis ratio, which is the ratio at which the MIX image is synthesized with the first material image when synthesizing the synthesized image, separately from the mixing ratio, and generates the synthesized image with the set synthesis ratio. The image processing apparatus according to claim 3.

8. An image processing method performed by an image processing device, which is a computer, The acquisition unit acquires a first material image and a second material image, which are face images showing a person's face. The MIX image generation unit generates a MIX image in which the first feature quantity, which is a feature quantity of the first material image, and the second feature quantity, which is a feature quantity of the second material image, are mixed together, using the first material image and the second material image. The similarity determination unit determines whether the MIX image and the first material image are similar, The device control unit causes the MIX image generation unit to generate the MIX image that the similarity determination unit has determined to be dissimilar. The MIX image generation unit generates the MIX image in which the first feature quantity and the second feature quantity are mixed at a predetermined mixing ratio. The threshold determination unit determines the mixing ratio of the image in which the proportion of the first feature quantity is included is the largest among the MIX images that the similarity determination unit has determined to be dissimilar, as the threshold. The device control unit specifies the mixing ratio in which the proportion of the second feature quantity is greater than the threshold and causes the MIX image generation unit to generate the MIX image. Image processing methods.

9. In an image processing device which is a computer, The system obtains a first and second source image, which are face images showing a person's face. Using the first and second source images, a mixed image is generated in which the first feature, which is a feature of the first source image, and the second feature, which is a feature of the second source image, are mixed. Determine whether the MIX image and the first material image are similar. The aforementioned MIX images that are determined to be dissimilar are generated, The first feature and the second feature are mixed in a predetermined mixing ratio to generate the MIX image. Among the MIX images that were determined to be dissimilar, the mixing ratio in the image with the largest proportion containing the first feature quantity is determined as the threshold. The MIX image is generated by specifying the mixing ratio in which the proportion of the second feature quantity is greater than the threshold. program.

Images