Image processing device and program

Abstract

The present invention provides an image processing device and program capable of automatically removing shadows from facial images. [Solution] The image processing device 1 includes an image preprocessing unit 12 that acquires a target face image by extracting a face from a face image, a shadow removal unit 13 that inputs the target face image acquired by the image preprocessing unit 12 to a shadow removal model 32c that removes shadows from the face region and acquires the processed image output by the shadow removal model 32c, an image synthesis unit 14 that synthesizes the processed image acquired by the shadow removal unit 13 with the face image, and a composite image output unit 15 that outputs the composite image synthesized by the image synthesis unit 14.

Description

[Technical Field]

[0001] This invention relates to an image processing apparatus and a program. [Background technology]

[0002] Conventionally, passport photo machines have been used to take facial photographs for use in employee ID cards, membership cards, My Number cards, etc. Passport photo machines are equipped with lighting devices in front, above, or behind the face of the person being photographed, allowing the face to be photographed with the subject's face properly illuminated. As a result, facial images taken with passport photo machines are accurately lit with no shadows on the face, and therefore meet the desired photographic quality.

[0003] Currently, users use devices such as smartphones to capture images for facial photographs to be used on employee ID cards, My Number cards, etc. However, when using devices such as smartphones, shadows can appear on the subject's face depending on the shooting environment, resulting in facial images with unintended shadows. This requires image processing to remove shadows from the photograph.

[0004] Generally, when removing shadows from a photograph, the image is processed manually using image processing software. The processing involves a combination of steps such as increasing the brightness of dark areas and blurring unnatural areas. This process relies heavily on the experience of the worker, and it often takes several minutes to several tens of minutes to process one image. Furthermore, for example, a technique has been disclosed in which a direction of virtual lighting is designated, and then gamma correction is performed based on illumination characteristics, followed by smoothing processing (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-171391 Summary of the Invention [Problem to be solved by the invention]

[0006] The method described in Patent Document 1 assumes that a virtual light source is irradiated based on a user's instruction to correct the brightness of the shadow area. In addition, the photograph after shadow removal must satisfy the quality of a facial image taken by a passport photo booth.

[0007] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing device and program that can automatically remove shadows that appear in facial images. [Means for solving the problem]

[0008] The present invention solves the above problems by the following means. A first invention is an image processing device comprising a preprocessing means for acquiring a target face image by extracting a face from a target image, and a shadow removal means for inputting the target face image acquired by the preprocessing means into a shadow removal model that removes shadows in the face area, and acquiring a processed image output by the shadow removal model. A second invention is an image processing device according to the first invention, comprising a synthesis means for synthesizing the processed image acquired by the shadow removal means with the target image, and a synthesis image output means for outputting the synthesised image synthesized by the synthesis means. A third invention is an image processing device according to the first or second invention, further comprising a learning means for training the shadow removal model using a combination of a first face image, which is a face image without a shadow in the face area, and a second face image, which is a face image with a shadow in the face area of ​​the first face image, and the shadow removal means acquires the processed image using the shadow removal model trained by the learning means. The fourth invention is an image processing apparatus of the third invention, comprising a shadow application means that inputs the first face image to a shadow application model that applies a shadow to the face region, and acquires the second face image output by the shadow application model, wherein the learning means trains the shadow removal model using a combination of the first face image and the second face image acquired by the shadow application means. The fifth invention is an image processing apparatus of the fourth invention, wherein the shadow application means acquires a plurality of different second face images in which shadows are applied to the face region when light is irradiated from different arbitrary angles using the shadow application model, and the learning means trains the shadow removal model using the first face image and each of the plurality of different second face images acquired by the shadow application means. The sixth invention is an image processing apparatus of the fourth or fifth invention, comprising: an image generation model capable of generating a person's face; an additional learning means for further training the image generation model using an ID photo taken with an ID photo booth; and a face image generation means for acquiring an image output by the image generation model as the first face image by inputting an image generation instruction including attribute items and item values ​​relating to a person to the image generation model further trained by the additional learning means, wherein the shadow application means acquires a second face image using the first face image acquired by the face image generation means. The seventh invention is an image processing device according to the sixth invention, further comprising an instruction generation means for generating the image generation instruction by randomly setting the item values ​​for the attribute items, wherein the attribute items include at least items relating to gender, race, age, body shape, and hairstyle. An eighth aspect of the present invention is a program for causing a computer to function as any one of the image processing devices according to the first to seventh aspects of the present invention. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an image processing device and a program that can automatically remove shadows that appear in face images. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram showing the overall configuration of an image processing system according to an embodiment of the present invention and functional blocks of an image processing apparatus. [Figure 2] 10 is a flowchart showing a shadow removal process of the image processing device according to the present embodiment. [Figure 3] 10A and 10B are diagrams for explaining a shadow removal process of the image processing device according to the embodiment. [Figure 4] 10 is a flowchart showing a shadow removal model learning process of the image processing device according to the present embodiment. [Figure 5] 10A and 10B are diagrams for explaining a shadow removal model learning process of the image processing device according to the present embodiment. [Figure 6] 10A and 10B are diagrams for explaining a shadow removal model learning process of the image processing device according to the present embodiment. [Figure 7] 10A and 10B are diagrams for explaining a shadow removal model learning process of the image processing device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, this is merely an example, and the technical scope of the present invention is not limited to this example. (Present embodiment) <Image processing system 100> FIG. 1 is a diagram showing the overall configuration of an image processing system 100 according to this embodiment and functional blocks of an image processing device 1.

[0012] 1, the image processing system 100 includes an image processing device 1, a user terminal 4, and an image DB (database) server 6. The image processing device 1, the user terminal 4, and the image DB server 6 are communicatively connected via a communication network N. The image processing system 100 is operated by a company that modifies, for example, a facial image (target image) taken by a user with a user terminal 4 such as a smartphone into a standardized format so that the image has the same quality as an image taken with a lighting camera, and provides various services using the modified facial image for various purposes. Examples of the various purposes include issuing ID (identification) cards with facial photographs and using facial images for face authentication when passing through security gates.

[0013] <Image processing device 1> The image processing device 1 is, for example, a device that performs processing to remove shadows in a face area of ​​a face image captured by a user using a user terminal 4. The image processing device 1 is also a device that learns a learning model used for shadow removal. The image processing device 1 can be realized by, for example, a server, or may be a general personal computer (PC) or the like. The image processing device 1 includes a control unit 10, a storage unit 30, and a communication interface unit 39.

[0014] The control unit 10 is a central processing unit (CPU) that controls the entire image processing device 1. The control unit 10 appropriately reads and executes an operating system (OS) and application programs stored in the storage unit 30, thereby cooperating with the above-mentioned hardware and executing various functions. Before describing the control unit 10 in detail, the storage unit 30 will be described. The storage unit 30 is a storage area such as a hard disk, a semiconductor memory element, etc. for storing programs, data, etc. required for the control unit 10 (described later) to execute various processes. The storage unit 30 includes a program storage unit 31 and a learning model storage unit 32.

[0015] The program storage unit 31 is a storage area for storing various programs. The program storage unit 31 stores a shadow removal program 31a and a model learning program 31b. The shadow removal program 31a is a program within the control unit 10 of the image processing device 1 for performing the function of removing shadows from facial images. The model learning program 31b is a program within the control unit 10 of the image processing device 1 for performing the function of learning a learning model for removing shadows. Instead of being divided into the shadow removal program 31a and the model learning program 31b, the program may be configured as a processing program for performing both functions, or the program may be further subdivided.

[0016] The learning model memory unit 32 is a memory area that stores various learning models. The learning model memory unit 32 stores the image generation model 32a, the shadow addition model 32b, and the shadow removal model 32c. The image generation model 32a is a trained model that has undergone additional training using ID photo images taken with an ID photo camera on a model capable of generating a person's face. The image generation model 32a may be, for example, a deep learning text-to-image model represented by Stable Diffusion that has the function of image generation AI (Artificial Intelligence) that generates images.

[0017] The shadow model 32b is a model that adds a shadow to a face region. The shadow model 32b is, for example, a model based on the method described in Andrew Hou, Michel Sarkis, Ning Bi, Yiying Tong, and Xiaoming Liu, “Face Relighting with Geometrically Consistent Shadows,” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 4217-4226, Internet <URL:https: / / openaccess.thecvf.com / content / CVPR2022 / html / Hou_Face_Relighting_With_Geometrically_Consistent_Shadows_CVPR_2022_paper.html> The model may be generated using the techniques described in [reference].

[0018] The shadow removal model 32c is a model that removes shadows from the face region and is trained using a combination of a face image with no shadow in the face region and a face image with a shadow in the face region of the face image. The shadow removal model 32c, when given a face image with shadows as input, outputs a face image without shadows by removing the shadows from the shadowed face image.

[0019] Next, the control unit 10 will be described. The control unit 10 comprises an image receiving unit 11, an image preprocessing unit 12 (preprocessing means), a shadow removal unit 13 (shadow removal means), an image synthesis unit 14 (synthesis means), a synthesized image output unit 15 (synthesis image output means), an image generation additional learning unit 21 (additional learning means, instruction generation means), a face image generation unit 22 (face image generation means), a shadow application unit 23 (shadow application means), and a shadow removal model learning unit 24 (learning means).

[0020] The image receiving unit 11 receives and accepts, for example, a face image from the user terminal 4. The face image to be accepted is an image of the user's face, generally from the shoulders up, and is, for example, an image taken by the user using the user terminal 4. The image preprocessing unit 12 obtains a target face image by extracting the face from the face image received by the image receiving unit 11. Here, "face" refers to the area from the chin to the top of the head, excluding the neck. The image preprocessing unit 12 can extract a target face image from the face image using various known techniques. For example, the image preprocessing unit 12 can recognize a target face image by using a technique that can recognize different parts of the head, such as semantic segmentation. Here, semantic segmentation is an image recognition technique that labels images at the pixel (picture element) level.

[0021] The shadow removal unit 13 inputs the target face image acquired by the image preprocessing unit 12 to the shadow removal model 32c and obtains the processed image output by the shadow removal model 32c. As a result of processing by the shadow removal unit 13, a shadow-free face image can be obtained in which the shadows present in the target face image have been removed. The image synthesis unit 14 synthesizes the image processed by the shadow removal unit 13 with the face image accepted by the image acceptance unit 11 . The composite image output unit 15 outputs the composite image created by the image composition unit 14 to the user terminal 4.

[0022] The image generation additional learning unit 21 additionally learns an image generation model capable of generating a human face, and obtains an image generation model 32a after the additional learning. The image generation additional learning unit 21 uses passport photo images taken at a passport photo booth, stored in the image DB server 6, as training data, so that it can output images with a composition similar to that of a passport photo booth's lighting. The image generation additional learning unit 21 further trains the image generation model using the training data and stores the further trained image generation model 32a in the learning model storage unit 32.

[0023] The face image generation unit 22 generates an image generation instruction by randomly setting item values ​​for attribute items related to a person. Here, the attribute items related to a person include, for example, at least items related to gender, race, age, body shape, and hairstyle. In addition, the attribute items may include various items that characterize the face, such as hair color, eye shape, eye color, glasses, blemishes, moles, freckles, and beard. The image generation instruction is a description of the instructions to the image generation model 32a, called a prompt. Next, the face image generation unit 22 inputs the generated image generation instruction to the image generation model 32a, which has been further trained by the image generation additional learning unit 21. The face image generation unit 22 then obtains the image based on the image generation instruction generated by the image generation model 32a, which is output by the image generation model 32a. The image output by the image generation model 32a is a shadowless face image (first face image) with no shadows in the face region, and is an image in the style of a passport photo.

[0024] The shadow application unit 23 inputs a face image without shadows to the shadow application model 32b and obtains a face image with shadows (second face image) output by the shadow application model 32b. The face image with shadows is an image in which shadows have been applied to the face image without shadows. Using the shadow application model 32b, the shadow application unit 23 can obtain multiple face images with shadows at different positions by applying shadows to the face region that would occur if light were irradiated from different arbitrary angles. Furthermore, using the shadow application model 32b, the shadow application unit 23 can obtain multiple face images with shadows having different shadow densities by applying shadows to the face region that would occur if light of different arbitrary illumination amounts were irradiated.

[0025] The shadow removal model learning unit 24 uses a combination of a face image without a shadow and a face image with a shadow to learn the shadow removal model 32c. The shadow removal model learning unit 24 can train the shadow removal model 32c using a combination of a shadow-free face image generated by the face image generation unit 22 and a shadow-containing face image generated by the shadow adding unit 23 using the shadow-free face image. Furthermore, since the shadow adding unit 23 generates a plurality of different shadow-containing face images for one shadow-free face image, the shadow removal model learning unit 24 can train the shadow removal model 32c using a plurality of combinations of one shadow-free face image and a plurality of shadow-containing face images.

[0026] The communication interface unit 39 is an interface for communication between the user terminal 4 and the image DB server 6 via the communication network N. Here, a computer refers to an information processing device equipped with a control unit, a storage device, etc., and the image processing device 1 is an information processing device equipped with a control unit 10, a storage unit 30, etc., and is included in the concept of a computer. There is no limit to the number of pieces of hardware that constitute the image processing device 1. Furthermore, the image processing device 1 may be, for example, a cloud.

[0027] <User Terminal 4> 1 is a terminal used by a user, and is, for example, a mobile information terminal such as a smartphone. The user terminal 4 may also be a tablet terminal, a personal computer (PC), or the like. For example, the user takes a picture of their own face using a camera and sends it to the image processing device 1 to use the services provided by the image processing system 100. Although not shown, the user terminal 4 includes a control unit, a storage unit, a camera, a touch panel display (input unit and output unit), a communication interface unit, and the like.

[0028] <Image DB Server 6> The image DB server 6 is a server having a DB in which ID photo images taken by the lighting camera are stored. The image DB server 6 includes a control unit, a memory unit, a DB (memory unit), a communication interface unit, etc., all of which are not shown. There is no limit to the number of pieces of hardware that constitute the image DB server 6. The image DB server 6 may be, for example, a cloud.

[0029] The communication network N is a network between the image processing device 1, the user terminal 4, and the image DB server 6, and is, for example, a communication network such as the Internet. The communication network N may include a LAN (Local Area Network) or the like. The communication network N may be wired or wireless.

[0030] <Processing Description> Next, processing using the image processing device 1 will be described. FIG. 2 is a flowchart showing the shadow removal process of the image processing device 1 according to this embodiment. FIG. 3 is a diagram for explaining the shadow removal process of the image processing device 1 according to this embodiment. The shadow removal process is performed, for example, every time a face image is received from the user terminal 4.

[0031] 2 (hereinafter, "step S" will be simply referred to as "S") 11, the control unit 10 (image receiving unit 11) of the image processing device 1 acquires a face image from the user terminal 4. The face image received from the user terminal 4 may have a shadow in the face area. 3 shows an example of a face image received from the user terminal 4. As shown in image 40, the face image is an image of the upper body, generally from the shoulders up. In S12 of FIG. 2, the control unit 10 (image preprocessing unit 12) acquires a target face image obtained by extracting a face from a face image. 3 shows an example of a target face image. As shown in the image 41, the target face image is obtained by extracting a face from the image 40.

[0032] In S13 of FIG. 2, the control unit 10 (shadow removal unit 13) inputs the target face image to the shadow removal model 32c. In S14, the control unit 10 (shadow removal unit 13) acquires the processed image output by the shadow removal model 32c. Image 42 in Figure 3 shows an example of a processed image output by the shadow removal model 32c. As shown in image 42, the processed image is an image in which the shadow has been removed from the target face image. The shadow that appeared on the left side of the face in image 41 has been removed in image 42, and image 42 is an image of a face with no shadow in the facial area.

[0033] In S15 of FIG. 2, the control unit 10 (image synthesis unit 14) synthesizes the processed image acquired in the process of S14 with the face image acquired in the process of S11. In S16, the control unit 10 (composite image output unit 15) outputs the composite image to the user terminal 4. Thereafter, the control unit 10 ends this process. 3 shows an example of a composite image output to the user terminal 4. The image 43 is obtained by combining the face portion of the image 40 received from the user terminal 4 with the image 42.

[0034] In this way, even if a face image received from the user terminal 4 has a shadow in the face area, the shadow in the face area can be automatically removed by using the shadow removal model 32c. Furthermore, by combining the face image from which the shadow has been removed with the face image received from the user terminal 4, the composite image can have the same quality as a face image taken by a passport photo machine.

[0035] Next, a process for learning the shadow removal model 32c used for shadow removal will be described. FIG. 4 is a flowchart showing the shadow removal model learning process of the image processing device 1 according to this embodiment. 5 to 7 are diagrams for explaining the shadow removal model learning process of the image processing device 1 according to this embodiment.

[0036] In S21 of Figure 4, the control unit 10 (image generation additional learning unit 21) performs additional training on an image generation model capable of generating human faces to obtain the image generation model 32a. The control unit 10 (image generation additional learning unit 21) uses passport photo images taken with a photo booth stored in the image DB server 6 as additional training data for the image generation model capable of generating human faces. For additional training, the control unit 10 (image generation additional learning unit 21) uses, for example, LoRa (Low-Rank Adaptation) to obtain additional training data for the image generation AI model. The image generation model 32a is a model to which the additional training data has been applied to the image generation model.

[0037] In S22, the control unit 10 (face image generating unit 22) generates an image generation instruction sentence. The control unit 10 (face image generating unit 22) generates an image generation instruction in which item values ​​for attribute items are set randomly. Figure 5 shows an example of the process from generating image generation instructions to outputting a face image. The attribute information 51 shown in Figure 5 is information that associates attribute items with item values. The attribute information 51 may be stored, for example, in the model learning program 31b or in the storage unit 30. The control unit 10 (face image generation unit 22) generates an image generation instruction statement 52 in which item values ​​are randomly set from the attribute items of the attribute information 51. The image generation instruction statement 52 is an instruction statement for the image generation model 32a. The image generation instruction statement 52 may include instructions such as appropriately setting item values ​​for parts where item values ​​for attribute items have not been set, and then generating the image.

[0038] In S23 of FIG. 4, the control unit 10 (face image generation unit 22) inputs the image generation instruction generated in the processing of S22 to the image generation model 32a that has undergone additional learning in the processing of S21, and obtains the shadow-free face image output by the image generation model 32a. 5, by inputting an image generation instruction statement 52 into the image generation model 32a, the control unit 10 (face image generation unit 22) obtains a face image 53. The face image 53 is an image without a shadow in the face area, for example, an ID photo-like image with 1024 pixels in both height and width. As described above, the image generation model 32a generates human faces that have been further trained using passport photo images taken with a photo booth and stored in the image DB server 6. Therefore, the images output by the image generation model 32a will be passport-style images.

[0039] In S24 of Figure 4, the control unit 10 (shadow application unit 23) inputs the shadowless face image generated in the S23 process to the shadow application model 32b, and obtains multiple face images with different shadow positions from the shadow application model 32b. Figure 6 shows an example of the process of obtaining a face image with a shadow from a face image without a shadow. In this example, a face image 61, which is an image without shadows in the face region, and a face mask image 62 of face image 61 are used. Face image 61 is an image generated by the image generation model 32a in the S23 process. Face mask image 62 is an image generated based on face image 61.

[0040] By inputting the face image 61 and the face mask image 62 into the shadow application model 32b, the control unit 10 (shadow application unit 23) obtains a face image 63. Face image 63 is an image in which a shadow has been applied to the face region of face image 61. The control unit 10 (shadow application unit 23) can, for example, generate a group of images 64 with shadows at various angles. The control unit 10 (shadow application unit 23) can also, for example, generate a group of images 65 in which the intensity of the shadow differs for each face image in the image group 64. In this example, 11 types of images can be generated by changing the shadow angle, and 5 types of images can be generated by changing the shadow intensity, so that 55 different shadowed face images can be generated from a single face image 61.

[0041] In S25 of FIG. 4, the control unit 10 determines whether a predetermined number of facial images have been generated. The predetermined number is the number of images necessary for training the shadow removal model 32c, and in order to accurately remove shadows, training must be performed using a large amount of data. Therefore, for example, a number of images in the thousands or tens of thousands is used. If the predetermined number of facial images have been generated (S25: YES), the control unit 10 proceeds to S26. On the other hand, if the predetermined number of facial images have not been generated (S25: NO), the control unit 10 proceeds to S22 and repeatedly performs processing to generate various types (kinds) of facial images with different item values ​​for the attribute items.

[0042] In S26, the control unit 10 (shadow removal model learning unit 24) uses a combination of the shadow-free face image and the shadow-present face image to learn the shadow removal model 32c, and then the control unit 10 ends the process. FIG. 7 shows an example of input data for training a shadow removal model. In this example, both the shadow-free face image generated in the process of S23 and the shadow-containing face image generated in the process of S24 use extracted images of extracted faces. For example, combination 70a consists of extracted image 71a without a shadow and extracted image 72a with a shadow. Extracted image 71a and extracted image 72a are images of the same person, but differ in the presence or absence of a shadow. Similarly, combination 70b consists of extracted image 71b without a shadow and extracted image 72b with a shadow.

[0043] In the process of S24, a plurality of shadow-containing face images are generated for one shadow-free face image. Therefore, the control unit 10 (shadow removal model learning unit 24) can generate a plurality of combinations by combining each shadow-containing face image with one shadow-free face image. Then, the control unit 10 (shadow removal model learning unit 24) inputs the generated combinations 70a, 70b, ... into the shadow removal model 32c to train the shadow removal model 32c.

[0044] Thus, the image processing apparatus 1 of this embodiment has the following effects. (1) A target face image is obtained by extracting a face from a facial image, and the obtained target face image is input to a shadow removal model 32c that removes shadows in the facial area, and a processed image output by the shadow removal model 32c is obtained. Therefore, even if a face image has a shadow, the shadow can be automatically removed by the shadow removal model 32c simply by inputting the face image, and a face image without a shadow in the face area can be easily obtained.

[0045] (2) The processed image acquired is combined with the face image, and the combined image is output. Therefore, the composite image can be made to meet the quality standards of a face image taken by a photo booth, in terms of shadows in the face image.

[0046] (3) The shadow removal model 32c is trained using a combination of a shadow-free face image with no shadow in the face area and a shadow-containing face image, which is a face image with a shadow in the face area of ​​the shadow-free face image, and the trained shadow removal model 32c is used to obtain a processed image. Therefore, the shadow removal model 32c can be trained using face images of the same subject with and without a shadow, and as a result, the shadow removal model 32c can become a model that accurately removes shadows from face areas.

[0047] (4) The shadow-free face image is input to the shadow-adding model 32b, which adds a shadow to the face area, and the shadow-containing face image output by the shadow-adding model 32b is obtained. The shadow-removal model 32c is trained using a combination of the shadow-free face image and the obtained shadow-containing face image. Therefore, since shadowed face images can be generated from shadowless face images, the data necessary for training the shadow removal model 32c can be easily created and used for training.

[0048] (5) Using the shadow-imparting model 32b, multiple different shadow-included face images are obtained in which shadows that occur when light is irradiated from different arbitrary angles are added to the face area, and the shadow-removal model 32c is trained using a combination of the shadow-free face image and each of the multiple different shadow-included face images obtained. Therefore, the shadow removal model 32c can learn from shadowed face images having shadows at various different positions in the face area, and the shadow removal model 32c can be a learning model with high accuracy for removing shadows.

[0049] (6) By inputting image generation instructions including attribute items and item values ​​related to a person into image generation model 32a, which is an image generation model that has been additionally trained using an image generation model capable of generating a person's face and an ID photo image taken with an ID photo machine, the image output by image generation model 32a is obtained as a shadow-free face image, and the obtained shadow-free face image is used to obtain a shadow-containing face image from shadow-adding model 32b. Furthermore, the image generation instruction is generated with item values ​​for the attribute items set randomly, and the attribute items include at least items relating to sex, race, age, body shape, and hairstyle. Therefore, a large number of combinations of face images with shadows and face images without shadows can be generated as training data for training the shadow removal model 32c. Furthermore, for face images with shadows, images relating to the faces of a wide variety of people can be generated.

[0050] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments. Note that the above-described embodiments and the modified embodiments described below can be used in appropriate combinations, but detailed description thereof will be omitted.

[0051] (Transformed form) (1) In the present embodiment, the learning data for training the shadow removal model 32c is generated using various tools, but the tools used in the present embodiment are merely examples, and other tools may be used for generation. Furthermore, if a combination of a face image with a shadow in the face area and a face image of the same subject without a shadow in the face area can be collected, the generated images do not need to be used.

[0052] (2) In this embodiment, a target face image obtained by extracting a face from a face image is described before shadow removal, but the method is not limited to this. For example, the shadow removal process may be performed using the face image as is. However, using a target face image obtained by extracting a face from a face image results in higher accuracy of shadow removal in the processed image because there is no extraneous image.

[0053] (3) In the present embodiment, shadow removal is performed on a face image captured by the user terminal 4, but the present invention is not limited to this. Shadow removal can be performed on face images for various purposes that require the same quality as a face image captured by a passport photo booth. In addition, in the present embodiment, the composite image after the shadow removal is output to the user terminal 4, but this is not limiting. For example, it may be output to a server or the like of a service provider where the user receives a service. Furthermore, in this embodiment, processing is performed each time a facial image is received from the user terminal 4, but depending on the application, received facial images may be stored and processed by batch processing or the like.

[0054] (4) In the present embodiment, the learning model storage unit 32 is provided in the storage unit 30, but this is not limiting. The learning model storage unit 32 may be provided in an external server or the like that is communicatively connected to the image processing device 1 via the communication network N. Furthermore, the process of learning the shadow removal model 32c (shadow removal model learning process in FIG. 4) and the process of generating an image in which the shadow has been removed from the face image (shadow removal process in FIG. 2) may be performed by different devices. [Explanation of symbols]

[0055] 1. Image processing device 4. User terminal 6 Image DB server 10 Control Unit 11 Image Reception Section 12 Image preprocessing section 13 Shadow removal section 14 Image synthesis unit 15. Composite image output unit 21 Image generation additional learning unit 22 Facial image generation unit 23 Shadow imparting part 24 Shadow removal model learning part 30 Storage section 31a Shadow Removal Program 31b Model Learning Program 32a Image generation model 32b Shadow model 32c Shadow Removal Model 100 Image Processing System N Communication Network

Claims

1. a preprocessing means for extracting a face from a target image and acquiring a target face image; a shadow removal means for inputting the target face image acquired by the preprocessing means into a shadow removal model for removing shadows in the face region, and acquiring a processed image output by the shadow removal model; An image processing device comprising:

2. 2. The image processing device according to claim 1, a combining means for combining the processed image acquired by the shadow removal means with the target image; a composite image output means for outputting a composite image synthesized by the synthesis means; An image processing device comprising:

3. 2. The image processing device according to claim 1, a learning means for learning the shadow removal model using a combination of a first face image that is a face image without a shadow in a face region and a second face image that is a face image having a shadow in a face region of the first face image; The shadow removal means acquires the processed image using the shadow removal model learned by the learning means.

4. 4. The image processing device according to claim 3, a shadow imparting means for inputting the first face image into a shadow imparting model that imparts a shadow to a face region, and acquiring the second face image output by the shadow imparting model; The learning means causes the shadow removal model to learn using a combination of the first face image and the second face image acquired by the shadow adding means.

5. 5. The image processing device according to claim 4, the shadow imparting means acquires a plurality of different second face images in which shadows that occur when light is irradiated from different arbitrary angles are imparted to the face region using the shadow imparting model; The learning means causes the shadow removal model to learn using a combination of the first face image and each of a plurality of different second face images acquired by the shadow adding means.

6. 6. The image processing device according to claim 4, an additional learning means for additionally learning the image generation model using an image generation model capable of generating a person's face and an ID photo image taken with an ID photo machine; a facial image generation means for inputting an image generation instruction including attribute items and item values ​​related to a person to the image generation model that has been additionally trained by the additional learning means, and acquiring an image output by the image generation model as the first facial image; Equipped with The shadow applying means obtains the second face image using the first face image obtained by the face image generating means.

7. 7. The image processing device according to claim 6, an instruction generating means for generating the image generation instruction by randomly setting the item values ​​for the attribute items; The attribute items include at least items related to gender, race, age, body shape, and hairstyle.

8. Computer, a preprocessing means for extracting a face from a target image and acquiring a target face image; a shadow removal means for inputting the target face image acquired by the preprocessing means into a shadow removal model for removing shadows in the face region, and acquiring a processed image output by the shadow removal model; A program to make it function as such.

9. The program according to claim 8, causing the computer to function as a learning means for learning the shadow removal model using a combination of a first face image that is a face image without a shadow in a face region and a second face image that is a face image of the first face image that has a shadow in a face region; a program that causes the shadow removal means to function to acquire the processed image using the shadow removal model learned by the learning means;

Images