Computer programs and image processing devices

Abstract

To change the style of an image in consideration of a face.SOLUTION: Detection processing for detecting a first region representing a face of a living object is executed using an input image. First image processing is executed using a first image of the first region of the input image, to generate a first processed image. The first image processing includes first style change processing. Second image processing is executed using a second image of a second region of the input image, to generate a second processed image. The second region is a region including at least a part of other regions excluding the first region. The second image processing includes second style change processing. First specifying processing, which is one of the first image processing and the second image processing, includes processing not included in second specifying processing, which is different from the first specifying processing being one of the first image processing and the second image processing. An output image is generated by executing synthesis processing of the first processed image with the second processed image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This specification relates to style conversion of images.

Background Art

[0002] Techniques for performing style conversion of images using a machine learning model are known. For example, the following paper discloses a neural network for transferring the style of an image to another image.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Images can represent various subjects. For example, an image can represent various living organisms having faces, such as humans and pets (e.g., dogs, cats). However, in conventional style conversion, faces have not been considered.

[0005] This specification discloses a technique for performing style conversion of images considering faces.

Means for Solving the Problems

[0006] The technique disclosed in this specification can be realized as the following application examples.

[0007] [Application Example 1] A computer program that causes a computer to implement: a detection function that performs a detection process to detect a first region representing the face of a living organism using an input image; a first generation function that generates a first processed image by performing a first image processing using a first image of the first region of the input image, wherein the first image processing includes a first style conversion process; a second generation function that generates a second processed image by performing a second image processing using a second image of a second region of the input image, wherein the second region is a region that includes at least a part of the remaining region excluding the first region, the second image processing includes a second style conversion process, and a first specific processing which is one of the first image processing and the second image processing includes processing that is different from the first specific processing among the first image processing and the second image processing and is not included in the second specific processing; and a third generation function that generates an output image by performing a synthesis process of the first processed image and the second processed image.

[0008] With this configuration, a first image processing step, including a first style transformation step, is performed using a first image of a first region representing the face of an organism, and a second image processing step, including a second style transformation step, is performed using a second image of a second region that includes at least a portion of the remaining region excluding the first region. This allows for image style transformation while taking the face into consideration. [Application Example 2] The computer program described in Application Example 1, The first image processing described above includes a high-resolution processing, The first style conversion process includes a style conversion process for an image with a resolution higher than the resolution of the first image of the first region on the input image. Computer program. [Application Example 3] A computer program as described in Application Example 1 or 2, The second style transfer process includes a style transfer process using a trained style transfer model. The aforementioned trained style transfer model has been trained to minimize losses. The loss includes a first term relating to the difference between the number of faces detected from the image input to the style transfer model and the number of faces detected from the image after the style transfer model has performed the style transfer. Computer program. [Application Example 4] A computer program as described in Application Example 1 or 2, The first style transfer process includes a style transfer process using a trained style transfer model, The aforementioned trained style transfer model has been trained to minimize losses. The loss includes a second term relating to the difference between the position of a first face landmark detected from a first face included in the image input to the style transfer model and the position of a second face landmark detected from a second face included in the image after the style transfer model has performed the style transfer. Computer program. [Application Example 5] The computer program described in Application Example 4, The first face is a face detected by performing the detection process using input training images representing N faces (where N is an integer greater than or equal to 2), The second face is one or more faces detected from the output training image, which is generated by performing a process including the detection process, the first image processing, the second image processing, and the synthesis process using the input training image, and is the face of the same individual as the first face. Computer program. [Application Example 6] The computer program described in Application Example 5, The second face is one of the one or more faces detected from the output training image that satisfies the positional condition that the difference in position between the first rectangle surrounding the first face and the second rectangle surrounding the face detected from the output training image is small. Computer program. [Application Example 7] An image processing device, A detection unit performs a detection process using an input image to detect a first region representing the face of an organism, A first generation unit generates a first processed image by performing first image processing using the first image of the first region of the input image, wherein the first image processing includes a first style conversion process, A second generation unit that generates a second processed image by performing a second image processing using a second image of a second region of the input image, wherein the second region is a region that includes at least a part of the remaining region excluding the first region, the second image processing includes a second style conversion process, and a first specific process which is one of the first image processing and the second image processing includes a process which is different from the first specific process and is not included in the second specific process, the second generation unit A third generation unit generates an output image by performing a synthesis process between the first processed image and the second processed image, An image processing device equipped with the following features.

[0009] Note that the technology disclosed in this specification can be implemented in various forms, for example, in the form of an image processing method and an image processing apparatus, a computer program for realizing the functions of those methods or apparatuses, a recording medium (e.g., a non-transitory recording medium) recording the computer program, and the like.

Brief Description of the Drawings

[0010] [Figure 1] It is a diagram showing an image processing apparatus as an example of an embodiment. [Figure 2] It is a diagram showing an example of a style conversion model M2. [Figure 3] It is a diagram showing an example of an encoder EC. [Figure 4](A) is a diagram showing an example of the formula for calculating the adjusted feature map tz. (B) is a diagram showing the formula for calculating the target feature map t. [Figure 5] This figure shows an example of a decoder DC. [Figure 6] This is a flowchart illustrating an example of the training process for the style transfer model M2. [Figure 7] This is a flowchart illustrating an example of the training process for the style transfer model M2. [Figure 8] Figures (A)-(D) show examples of images processed during the training process. [Figure 9] (A) and (B) are diagrams showing examples of landmark sets. [Figure 10] This figure shows an example of image processing. [Figure 11] (A) is a diagram showing an example of a facial region and a facial landmark. (B) is a diagram showing an example of a correspondence. [Figure 12] This flowchart shows an example of the process for determining correspondence relationships. [Figure 13] (A) is a diagram showing another example of a facial region and a facial landmark. (B) is a diagram showing an example of a correspondence. [Figure 14] Figures (A)-(C) show examples of loss calculation formulas. [Figure 15] This flowchart shows an example of image processing using a pre-trained style transfer model M2. [Figure 16] This flowchart shows an example of image processing using a pre-trained style transfer model M2. [Figure 17] Figures (A)-(D) show examples of images processed by image processing. [Modes for carrying out the invention]

[0011] A. First Example: A1. Equipment configuration: Figure 1 shows an image processing apparatus as one embodiment. In this embodiment, the image processing apparatus 200 is, for example, a personal computer. The image processing apparatus 200 is an example of an image processing apparatus that performs style transformation on images representing the faces of living organisms.

[0012] The image processing device 200 comprises a processor 210, a storage device 215, a display unit 240, an operation unit 250, and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0013] The processor 210 is a device configured to perform data processing, such as a CPU. The volatile memory device 220 is, for example, DRAM, and the non-volatile memory device 230 is, for example, flash memory. The non-volatile memory device 230 stores programs 231 and 232, a face processing model M1, a style transfer model M2, a super-resolution model M3, and a segmentation model M4. The face processing model M1 includes a face detection model M1a and a landmark detection model M1b. In this embodiment, models M1-M4 are each program modules. Models M1-M4 are each so-called machine learning models. Details of programs 231 and 232 and models M1-M4 will be described later.

[0014] The display unit 240 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive user input, such as buttons, levers, or a touch panel superimposed on the display unit 240. The user can input various instructions to the image processing device 200 by operating the operation unit 250. The communication interface 270 is an interface for communicating with other devices. The communication interface 270 includes, for example, one or more of the following: a USB interface, a wired LAN interface, or an IEEE 802.11 wireless interface.

[0015] A2. Style conversion model: Figure 2 shows an example of the style transfer model M2. The style transfer model M2 generates a transformed content image g(tz) represented in the style of style image s by rendering the content image c in the style of style image s. In this way, the style transfer model M2 applies the style of style image s to the content image c. The style transfer model M2 performs style transfer of the content image while suppressing changes in the shape of the subject matter of the content image (e.g., the shape of the edges). In this embodiment, the style transfer model M2 is the style transfer model disclosed in the following paper. The technique in this paper uses a normalization called adaptive instance normalization (AdaIN). Xun Huang and Serge Belongie, "Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization", arXiv:1703.06868, 30 Jul. 2017, http: / / arxiv.org / abs / 1703.06868

[0016] The style transfer model M2 has two input layers, Li1 and Li2, an encoder EC, a normalization layer NL, and a decoder DC. Each processing unit, Li1, Li2, EC, NL, and DC, has an "input" column and an "output" column. The combination of three numbers enclosed in parentheses in the "input" column indicates the size of the input data. The combination of three numbers enclosed in parentheses in the "output" column indicates the size of the output data. The data size is expressed as (width, height, number of channels). For example, the images c and s input to input layers Li1 and Li2 are represented as 256*256*3, respectively. The three channels represent the three color components: red, green, and blue. "None" written to the left of the data size indicates that the batch size for training the style transfer model M2 can be arbitrarily set.

[0017] The encoder EC uses an image represented as 256*256*3 to output a feature map represented as 32*32*512. The feature map represents the features of the image input to the encoder EC. The content feature map f(c) is a feature map output using the content image c, and the style feature map f(s) is a feature map output using the style image s.

[0018] Figure 3 shows an example of an encoder EC. In this embodiment, the encoder EC is the same as the portion of a convolutional neural network called VGG-19 from the beginning up to the relu4_1 layer (the relu4_1 layer represents the activation function (here, ReLU (Rectified Linear Unit)) for outputting data from the first convolutional layer of the fourth block). The computational parameters of VGG-19, trained using image data registered in an image database called ImageNet, are publicly available. In this embodiment, the publicly available pre-trained computational parameters are used as the computational parameters of the encoder EC. Alternatively, the encoder EC may be trained in the training process of the style transfer model M2.

[0019] The encoder EC has, in order from upstream, an input layer EL01, convolutional layers EL02, EL03, pooling layer EL04, convolutional layers EL05, EL06, pooling layer EL07, convolutional layers EL08, EL09, EL10, EL11, pooling layer EL12, and convolutional layer EL13.

[0020] The configurations of the convolutional layers EL02, EL03, EL05, EL06, EL08, EL09, EL10, EL11, and EL13 can vary. In this embodiment, the following configurations are commonly applied. The width * height of the convolution kernel (also called a filter) is 3 * 3. The stride is 1 * 1. The padding method is so-called zero padding. In this embodiment, padding is performed so that the width * height of the output map (image or feature map) from the convolutional layer is the same as the width * height of the input map (image or feature map) input to the convolutional layer. Specifically, a 1-pixel wide area indicating zero is added around the boundary (i.e., contour) of the input map. The activation function is ReLU.

[0021] The configurations of the pooling layers EL04, EL07, and EL12 can be various configurations that reduce either or both the width and height of the feature map. In this embodiment, the following configurations are commonly applied. The pooling method is so-called maximum pooling. The width * height of the pooling kernel (also called a filter) is 2 * 2. The stride is 2 * 2. Such pooling layers EL04, EL07, and EL12 reduce both the width and height by half.

[0022] The data width and height are reduced by the encoder EC. In this embodiment, the width and height are reduced from 256 to 32, respectively. The number of data channels is increased by the encoder EC. In this embodiment, the number of channels is increased from 3 to 512.

[0023] The normalization layer NL (Figure 2) uses the content feature map f(c) and the style feature map f(s) to output the adjusted feature map tz. Figure 4(A) shows an example of the formula for calculating the adjusted feature map tz. The adjusted feature map tz is the weighted sum of the target feature map t (details below) and the content feature map f(c). The weight α is the weight of the target feature map t. The weight of the content feature map f(c) is 1-α. The weight α is set to a value greater than zero and less than or equal to 1.

[0024] Figure 4(B) shows the calculation formula for the target feature map t. The calculation formula for the target feature map t is the same as the calculation formula for the AdaIN layer in the aforementioned AdaIN paper. In the calculation formula, σ represents the standard deviation and μ represents the mean. The standard deviation and mean are calculated over the entire spatial location. The standard deviation and mean are calculated for each channel.

[0025] The target feature map t is generated by matching the standard deviation and mean of the content feature map f(c) with the standard deviation and mean of the style feature map f(s). In this way, AdaIN performs style transformation in the feature space by applying the channel-specific standard deviation and mean of the style feature map f(s) to the content feature map f(c).

[0026] As shown in Figure 4(A), the larger the weight α, the larger the proportion of the target feature map t within the adjusted feature map tz. The larger the weight α, the closer the style of the transformed content image g(tz) (Figure 2) approaches the style of the style image s. Thus, the weight α indicates the degree of style transformation. The weight α is set to 1 during the training of the style transformation model M2. In this case, the adjusted feature map tz = target feature map t. In post-training image processing, the weight α may be set to various values ​​greater than zero and less than or equal to 1.

[0027] The normalization layer NL (Figure 2) calculates the target feature map t using the content feature map f(c) and the style feature map f(s) (Figure 4(B)). Then, the normalization layer NL calculates the adjusted feature map tz using the target feature map t, the content feature map f(c), and the weight α (Figure 4(A)). The size of the target feature map t and the adjusted feature map tz are the same as the size of the feature map f(c) (32*32*512 in this embodiment).

[0028] Decoder DC outputs a converted content image g(tz) represented as 256*256*3, using an adjusted feature map tz represented as 32*32*512. The three channels represent the components of three colors: red, green, and blue.

[0029] Figure 5 shows an example of a decoder DC. The architecture of the decoder DC is a mirror image of the encoder EC architecture, with the following modifications. (1) Zero padding before the convolution is replaced by reflection padding. (2) Pooling is replaced by upsampling.

[0030] Specifically, the decoder DC has the following layers arranged in order from upstream: input layer DL01, padding layer DL02, convolutional layer DL03, upsampling layer DL04, padding layer DL05, convolutional layer DL06, padding layer DL07, convolutional layer DL08, padding layer DL09, convolutional layer DL10, padding layer DL11, convolutional layer DL12, upsampling layer DL13, padding layer DL14, convolutional layer DL15, padding layer DL16, convolutional layer DL17, upsampling layer DL18, padding layer DL19, convolutional layer DL20, padding layer DL21, and convolutional layer DL22.

[0031] The configurations of the padding layers DL02, DL05, DL07, DL09, DL11, DL14, DL16, DL19, and DL21 can vary. In this embodiment, the following configuration is commonly applied. The padding method is so-called reflection padding. Reflection padding determines the value of each pixel in the part outside the boundary by reflecting the inner part outwards around the boundary of the input map (image or feature map) input to the padding layer.

[0032] The configurations of the convolutional layers DL03, DL06, DL08, DL10, DL12, DL15, DL17, DL20, and DL22 can vary. In this embodiment, the following configurations are commonly applied: The width * height of the convolution kernel (also called a filter) is 3 * 3. The stride is 1 * 1. The activation function is ReLU. The output data from the preceding padding layers DL02, DL05, DL07, DL09, DL11, DL14, DL16, DL19, and DL21 are input to the convolutional layers DL03, DL06, DL08, DL10, DL12, DL15, DL17, DL20, and DL22, respectively.

[0033] In this embodiment, padding is performed so that the width * height of the output map (image or feature map) from the convolutional layer is the same as the width * height of the input map (image or feature map) input to the padding layer before the convolutional layer. Specifically, a 1-pixel wide area is added around the boundary of the input map.

[0034] The configurations of the three upsampling layers DL04, DL13, and DL18 may vary, each increasing either the width or height of the feature map or both. In this embodiment, the following configuration is commonly applied. The upsampling method is so-called nearest upsampling. The upsampling layers DL04, DL13, and DL18 increase the width and height by a factor of two, respectively.

[0035] The data width and height are increased by the decoder DC. In this embodiment, the width and height are increased from 32 to 256, respectively. The number of data channels is reduced by the decoder DC. In this embodiment, the number of channels is reduced from 512 to 3.

[0036] A3. Training process for style transfer models: Figures 6 and 7 are flowcharts illustrating an example of the training process for the style transfer model M2. Figure 7 is a continuation of Figure 6. The first program 231 (Figure 1) is a program for training the style transfer model M2. The operator inputs a start command for the training process to the image processing device 200 by operating the operation unit 250. The processor 210 starts the training process for the style transfer model M2 in accordance with the start command.

[0037] In S105, the processor 210 initializes several computational parameters of the style transfer model M2 (Figure 2). In this embodiment, the processor 210 determines each computational parameter using random numbers. The computational parameters of the encoder EC are set to the corresponding computational parameters of the trained VGG-19.

[0038] In S110, processor 210 acquires data for pairs of input images and style images.

[0039] Figures 8(A)-8(D) show examples of images processed in the training process. Figure 8(A) shows an example of an input image. The input image IM1 is a rectangular image having two sides parallel to the first direction Dx (here, the horizontal direction) and two sides parallel to the second direction Dy (here, the vertical direction) which is perpendicular to the first direction Dx. The input image IM1 is represented by the color values ​​of multiple pixels arranged in a matrix along the first direction Dx and the second direction Dy. In this embodiment, the color values ​​are represented by three component values: R (red), G (green), and B (blue). Each component value is represented, for example, in 256 steps from 0 to 255.

[0040] The input image IM1 is a photograph of five people, PR1-PR5. The input image IM1 contains images of the five faces F1-F5 of the five people, PR1-PR5.

[0041] The style images may be various images having a style different from that of the input image IM1. For example, the style image s in Figure 2 and the input image IM1 in Figure 8(A) may form a pair.

[0042] Although not shown in the diagram, in this embodiment, data for multiple input images and data for multiple style images are pre-stored in the non-volatile storage device 230. The multiple input images include various images representing N faces of N people (where N is a non-negative integer). The multiple input images may include multiple captured images. The number of people N may differ among the multiple input images. Note that N may be set to a value of 2 or more. The multiple style images include multiple images having different styles from each other. Styles may be identified from various viewpoints related to image representation, such as handwriting, brightness distribution, and hue distribution.

[0043] In S110 (Figure 6), the processor 210 acquires data for pairs of a predetermined batch size BS from the non-volatile storage device 230. The batch size BS can be any number (for example, 1, 2, 4, 8, or 16). The processing from S120 to S250 (Figure 7), described later, is executed for each of the acquired pairs. The style image may be selected randomly, independently of the input image. Alternatively, the correspondence between the input image and the style image may be predetermined. The same style image may be included in multiple pairs.

[0044] In S120, the processor 210 detects face regions and face landmarks from the input image. There are various methods for detecting face regions. In this embodiment, the processor 210 detects face regions using a trained face detection model M1a (Figure 1). Face detection model M1a may be any model for detecting face regions. In this embodiment, face detection model M1a is an object detection model called "BlazeFace" disclosed in the following paper. Valentin Bazarevsky, Yury Kartynnik, Andrey Vakunov, Karthik Raveendran and Matthias Grundmann, "BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs", arXiv:1907.05047, 14 Jul. 2019, http: / / arxiv.org / abs / 1907.05047

[0045] Figure 8(B) shows an example of a face region detected from the input image IM1. The face detection model M1a detects a rectangular frame (called a bounding box) surrounding a face. The face detection model M1a is pre-trained by the training method described in the BlazeFace paper to detect appropriate bounding boxes representing faces from a face image. The processor 210 can detect five bounding boxes Bp1-Bp5 corresponding to five faces F1-F5 by performing calculations on the face detection model M1a using the input image IM1.

[0046] Hereafter, the processor 210 will assign an ascending identification number starting from 1 to the detected bounding box. The number following "Bp" in the bounding box code will indicate the identification number.

[0047] The processor 210 performs size adjustment to adjust the size of the input image (specifically, width * height) to a size acceptable to the face detection model M1a in order to perform calculations on the face detection model M1a. Size adjustment may include, for example, a resolution conversion process (also called resizing) (e.g., bicubic or bilinear methods). Also, if the aspect ratio of the input image differs from the aspect ratio acceptable to the face detection model M1a, size adjustment may include a process to adjust the aspect ratio. The processor 210 may, for example, perform padding to fill in the area that is lacking for an appropriate aspect ratio with pixels having a predetermined color (e.g., white, black, etc.).

[0048] Facial landmarks are points that indicate parts of the face, such as the chin, nose, eyes, eyebrows, and mouth. There are various methods for detecting facial landmarks. In this embodiment, the processor 210 detects landmarks using the landmark detection model M1b (Figure 1). The landmark detection model M1b may be any of the various models for detecting facial landmarks. In this embodiment, the landmark detection model M1b is a detection model called "Attention Mesh" disclosed in the following paper. Ivan Grishchenko, Artsiom Ablavatski, Yury Kartynnik, Karthik Raveendra and Matthias Grundmann, "Attention Mesh: High-fidelity Face Mesh Prediction in Real-time", arXiv:2006.10962, 19 Jun. 2020, http: / / arxiv.org / abs / 2006.10962

[0049] The Attention Mesh model estimates the three-dimensional coordinates of multiple points representing multiple parts of a face from a two-dimensional image. These points form a face mesh that represents the three-dimensional shape of the face. The landmark detection model M1b is pre-trained using the training method described in the Attention Mesh paper to estimate an appropriate face mesh from a face image. In this embodiment, the processor 210 adopts a set of points representing predetermined parts of the face from among the multiple points that form the face mesh as a landmark set.

[0050] Figures 9(A) and 9(B) show examples of landmark sets. Figure 9(A) shows an image IMh of a person's face Fh. In this example, 68 points P1-P68 are used as landmark set LM. The first point set PS1, points P1-P17, represents the chin. The second point set PS2, points P18-P22, represents the right eyebrow. The third point set PS3, points P23-P27, represents the left eyebrow. The fourth point set PS4, points P28-P36, represents the nose. The fifth point set PS5, points P37-P42, represents the right eye. The sixth point set PS6, points P43-P48, represents the left eye. The seventh point set PS7, points P49-P68, represents the mouth.

[0051] Figure 9(B) shows an example of data representing the landmark set LM. The number nP is the total number of points included in the landmark set LM (in this embodiment, nP = 68). The variables Pux and Puy (where the number u is an integer between 1 and nP) represent the coordinates of the first direction Dx and the second direction Dy of the u-th point Pu. For example, P3x and P3y represent the coordinates of the first direction Dx and the second direction Dy of the third point P3, respectively. The landmark set LM is represented by a vector that sequentially shows the coordinates of the first direction Dx and the second direction Dy of each of the nP points.

[0052] Figure 8(B) shows an example of facial landmarks detected from the input image IM1. Landmark detection is performed for each bounding box. The processor 210 detects the landmark set LMp1 of the first face F1 by performing calculations of the landmark detection model M1b using the image of the first bounding box Bp1. Landmark set LMp1 consists of 68 points P1-P68 (Figure 9). Similarly, landmark sets LMp2-LMp5 are detected from the other bounding boxes Bp2-Bp5. The landmark sets are assigned the same identification numbers as those of the corresponding bounding boxes. The number following "LMp" in the landmark set code indicates the identification number. Hereafter, the landmark sets detected from the input image will also be referred to as the input landmark sets.

[0053] For detecting facial regions and facial landmarks using "BlazeFace" and "Attention Mesh," a library called "MediaPipe" from Google may be used. The face processing model M1, which includes the face detection model M1a and the landmark detection model M1b, may be constructed using "MediaPipe."

[0054] In S130 (Figure 6), the processor 210 generates a processed image, which is a style-transformed image, by performing calculations on the style transfer model M2 using the input image and style image pair. During the training process of the style transfer model M2, the weight α (Figure 4(A)) is set to 1. The processor 210 also adjusts the size of the image input to the style transfer model M2. The size adjustment may include a resolution conversion process and a padding process to adjust the aspect ratio, similar to the size adjustment described in S120 (Figure 6).

[0055] Figure 8(C) shows an example of a processed image. Processed image IM2 shows the image generated from input image IM1 (Figure 8(A)). As shown in the figure, processed image IM2 represents people PR1-PR5. The faces Fs1-Fs5 of people PR1-PR5 are represented in a different style from the style of input image IM1 (in this case, the style of the style image). Hereafter, the processed image generated in S130 (Figure 6) will also be referred to as the overall processed image.

[0056] In the processed image IM2 in Figure 8(C), the shape of the parts of the face Fs2 of the second person PR2 (e.g., the eyes) has changed significantly from the shape of the corresponding parts of face F2 in the input image IM1 (Figure 8(A)). Such changes in shape can be caused by various factors. For example, if the size of face F2 in the input image IM1 is small, the shape of the parts included in face F2 may change significantly due to style transformation.

[0057] In S140 (Figure 6), the processor 210 determines whether the total number of bounding boxes Nfp detected in S120 is greater than zero. If the total number Nfp is greater than zero (S140: Yes), in S150, the processor 210 selects an unprocessed box from the Nfp bounding boxes as a box of interest.

[0058] In S160, processor 210 extracts the focus image, which is the image of the focus box, from the input image.

[0059] Figure 10 shows an example of processing a focus image. The figure shows a focus image FIk, which is the image of the focus box Bpk. This focus image FIk represents the face Fk of person PRk.

[0060] In S170 (Figure 6), the processor 210 performs a resolution enhancement process on the image of interest. The resolution enhancement process may be various processes that increase the resolution (i.e., pixel density) of the image. In this embodiment, the processor 210 generates a high-resolution image using a trained super-resolution model M3 (Figure 1). The super-resolution model M3 may be various models that generate high-resolution images. In this embodiment, the super-resolution model M3 is the image generation model disclosed in the following paper. This paper discloses a technique called "PULSE". Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi and Cynthia Rudin, "PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models", arXiv:2003.03808, 20 Jul. 2020, http: / / arxiv.org / abs / 2003.03808

[0061] The technique described in this paper generates high-resolution images using an image generation model (StyleGAN in the paper) trained to produce natural-looking high-resolution images of faces. Latent variables to be input to the image generation model are searched so that the difference between the low-resolution image obtained by downscaling (bicubic in the paper) the generated high-resolution image and the original low-resolution image is minimized. The high-resolution image generated by the image generation model using the searched latent variables is adopted as the high-resolution image corresponding to the low-resolution image. The search for latent variables is performed in a way that minimizes the loss. Here, a downscaling loss, which shows the difference between the low-resolution image obtained by downscaling and the original low-resolution image, and a cross loss (also called geodesic cross loss) are used to search for latent variables that produce natural-looking images.

[0062] In this embodiment, the super-resolution model M3 is a StyleGAN trained to generate natural, high-resolution images of faces. For the StyleGAN configuration, for example, the configuration disclosed in the following paper is adopted. Tero Karras, Samuli Laine and Timo Aila, "A Style-Based Generator Architecture for Generative Adversarial Networks", arXiv:1812.04948, 29 Mar. 2019, http: / / arxiv.org / abs / 1812.04948 The super-resolution model M3 is pre-trained using the training method described in this paper.

[0063] The processor 210 uses the trained super-resolution model M3 to generate a high-resolution image FIka corresponding to the image of interest FIk (Figure 10). The method for generating the high-resolution image FIka is the same as the method described in the aforementioned PULSE paper. The magnification factor for the resolution enhancement process can be any value greater than 1. For example, the width and height may each be increased by a factor of 4.

[0064] In S180 (Figure 6), the processor 210 uses the high-resolution image FIka and the style image to generate a transformed image, which is a style-transformed image. Image FIkb in Figure 10 shows an example of a transformed image generated from the high-resolution image FIka. The transformed image FIkb represents person PRk. The face Fzk of person PRk is represented in a different style from the style of the input image IM1 (in this case, the style of the style image).

[0065] The style conversion method in S180 (Figure 6) is the same as the style conversion method in S130. The processor 210 generates a converted image FIkb by performing calculations of the style conversion model M2 using the high-resolution image FIka and the style image. As the style image, a style image corresponding to the input image containing the image of interest FIk is used. The processor 210 adjusts the size of the image input to the style conversion model M2. The size adjustment may include a resolution conversion process and a padding process to adjust the aspect ratio, similar to the size adjustment described in S120 (Figure 6). In this embodiment, the width and height of the image input to the style conversion model M2 are larger than the width and height of the image of interest FIk (Figure 10).

[0066] In S180 (Figure 6), the size of the face in the image input to the style transfer model M2 (e.g., width and height) is larger than the size of the face in the image input to the style transfer model M2 in S130. Therefore, in S180, the possibility of deformation of parts of the face (e.g., eyes) is smaller compared to S130.

[0067] In S190 (Figure 6), the processor 210 performs segmentation processing on the converted image FIkb (Figure 10). In this embodiment, the segmentation processing in S190 divides the region representing the face from the other regions. There may be various methods for dividing the regions. In this embodiment, the processor 210 uses a trained segmentation model M4 (Figure 1) to divide the region representing the face from the other regions. The segmentation model M4 may be various models for dividing the regions. In this embodiment, the segmentation model M4 is a model called "FCN (fully convolutional network)" disclosed in the following paper. Yuval Nirkin, Iacopo Masi, Anh Tuan Tran, Tal Hassner and Gerard Medioni, "On Face Segmentation, Face Swapping, and Face Perception", arXiv:1704.06729, 22 Apr. 2017, http: / / arxiv.org / abs / 1704.06729

[0068] This paper proposes an architecture called FCN-8s-VGG. In this embodiment, the partitioned model M4 has this architecture. This model partitions the region that represents the visible part of the face. The partitioned model M4 is pre-trained using the training method described in this paper.

[0069] The processor 210 uses the converted image FIkb (Figure 10) to perform operations on the division model M4, thereby dividing the region representing the face from the other regions. The image SGI in Figure 10 shows the two regions SG1 and SG2 that are divided from the converted image FIkb. The first region SG1 is the region representing the face, and the second region SG2 is the other region. In this embodiment, the size of the image that can be accepted by the division model M4 is the same as the size of the image generated by the style conversion model M2 (e.g., the converted image FIkb).

[0070] In S200 (Figure 6), processor 210 performs masking on the converted image FIkb. Processor 210 masks the second region SG2, which was divided in S190. Image FIkc in Figure 10 shows an example of a masked image. Masked image FIkc represents the image of the first region SG1 (i.e., the face image). The second region SG2 is masked.

[0071] In S210 (Figure 6), processor 210 performs a downscaling process on the masked image, i.e., the style-transformed face image. Image FIkd in Figure 10 shows an example of a processed image, which is the image generated by the downscaling process (hereinafter referred to as the face-processed image). This face-processed image FIkd is the image generated from the masked image FIkc. The size (i.e., width and height) of the face-processed image FIkd is the same as the size of the image of interest FIk. The downscaling process (also called downscaling) can be any of the various processes that reduce the resolution (e.g., nearest neighbor, bilinear, bicubic, etc.).

[0072] The aspect ratio of the masked image FIkc may differ from the aspect ratio of the image in question FIk. For example, the masked image FIkc may include portions that have been padded during the size adjustment process in S180. The processor 210 generates the face-processed image FIkd by removing the padded portions. The position and size of the faces in the face-processed image FIkd are substantially the same as the position and size of the faces in the image in question FIk, respectively.

[0073] In S220 (Figure 6), the processor 210 determines whether processing of all bounding boxes has been completed. If there are any unprocessed bounding boxes remaining (S220: No), the processor 210 proceeds to S150 to process the unprocessed bounding boxes.

[0074] If all bounding box processing is complete (S220: Yes), in S230 (Figure 7), the processor 210 generates an output image by superimposing the style-transformed face image onto the corresponding positions of the style-transformed input image (for example, the overall processed image IM2 (Figure 8(C))). In other words, the processor 210 generates an output image by combining the style-transformed face image and the style-transformed input image.

[0075] Figure 8(D) shows an example of an output image. Output image IM3 is generated by superimposing the images of faces Fz1-Fz5, generated in S210, onto the face Fs1-Fs5 region of the overall processed image IM2 (Figure 8(C)) generated in S130 (Figure 6). For example, assume that the processed image FIkd in Figure 10 corresponds to the second bounding box Bp2 of the input image IM1 (Figure 8(B)). In this case, processor 210 superimposes the image of the first region SG1 of the processed image FIkd (i.e., the image of face Fzk) onto the region corresponding to the first region SG1 within the bounding box Bp2 of the processed image IM2 (Figure 8(C)). As explained in S180 (Figure 6), in S180, the possibility of deformation of parts of the face (such as the eyes) is smaller compared to S130. Therefore, even if a portion of the face in the overall processed image IM2 is significantly deformed, as in face Fs2 in Figure 8(C), the processor 210 can generate an output image IM3 with minimal facial deformation.

[0076] In S240 (Figure 7), the processor 210 detects the face region and facial landmarks from the output image. The processing in S240 is the same as the processing in S120, except that the output image is used instead of the input image. Hereafter, the set of landmarks detected from the output image will also be called the output landmark set.

[0077] Figure 11(A) shows an example of facial regions and facial landmarks detected from the output image IM3. In the example in Figure 11(A), five bounding boxes Bq1-Bq5 corresponding to five faces Fz1-Fz5 are detected from the output image IM3. Additionally, five landmark sets LMq1-LMq5 corresponding to the five bounding boxes Bq1-Bq5 are detected. The number following "Bq" in the bounding box code indicates the identification number. The number following "LMq" in the landmark set code indicates the identification number.

[0078] In S250 (Figure 7), the processor 210 determines the correspondence between the input image IM1 and the output image IM3 for faces of the same person. As will be described later, the processor 210 trains the style transfer model M2 to minimize the loss. In this embodiment, the loss includes a landmark loss that represents the difference between the input landmark set and the output landmark set for the same person. To calculate the landmark loss, the processor 210 determines the correspondence between the landmark sets of the same person, i.e., the correspondence between the bounding boxes of the same person, between the input image IM1 and the output image IM3.

[0079] Figure 12 is a flowchart illustrating an example of the correspondence determination process. In this embodiment, the processor 210 determines the correspondence using the Intersection over Union (IoU) of a pair of bounding boxes (referred to as input boxes) detected from the input image IM1 (Figure 8(B)) and bounding boxes (referred to as output boxes) detected from the output image IM3 (Figure 11(A)). IoU is the ratio obtained by dividing the area of ​​the intersection of the two regions by the area of ​​the union of the two regions. IoU indicates the degree of agreement between the two boxes. The area may be expressed, for example, in terms of the number of pixels. The processor 210 determines the correspondence by assuming that a pair of input and output boxes with an IoU greater than a threshold represents the face of the same person.

[0080] In S310, the processor 210 initializes the number i of the input box Bpi of interest among the input boxes of the input image IM1 (Figure 8(B)) to 1. In S320, the list of IoUs Liou is initialized. The list Liou is a list of IoUs between the input box Bqi of interest and each output box. For example, the IoU of each output box is initialized to zero.

[0081] In S330, the processor 210 initializes the number j of the output box Bqj of interest among the output boxes of the output image IM3 (Figure 11(A)) to 1. In S340, the processor 210 calculates the IoU(i,j) between the input box Bpi of interest and the output box Bqj of interest. Then, the processor 210 sets the j-th data Liou(j) from the list Liou to the calculated IoU(i,j).

[0082] In S350, the processor 210 determines whether all output boxes have been processed. If there are any unprocessed output boxes remaining (S350: No), the processor 210 adds 1 to the number j of the output box Bqj in question in S360 and proceeds to S340.

[0083] If all output boxes have been processed (S350: Yes), in S370, processor 210 obtains the maximum IoU by referring to list Liou. In S380, processor 210 obtains the number v of the output box of interest with the maximum IoU. In S390, processor 210 determines whether the maximum IoU is greater than threshold TH1. Threshold TH1 is determined experimentally in advance such that the IoU of boxes Bpi and Bqj representing the same person is greater than threshold TH1, and the IoU of boxes Bpi and Bqj representing different people is less than threshold TH1 (e.g., TH1=0.4).

[0084] If the maximum IoU is greater than threshold TH1 (S390: Yes), in S400, processor 210 sets the corresponding landmark set LMri, which corresponds to the input landmark set LMpi of input box i, Bpi, to the output landmark set LMqv of the output box of interest with the maximum IoU. Then, processor 210 proceeds to S420.

[0085] In S420, the processor 210 determines whether all input boxes have been processed. If all input boxes have been processed (S420: Yes), the processor 210 terminates the process shown in Figure 12, i.e., the process shown in S250 of Figure 7. If there are any unprocessed input boxes remaining (S420: No), the processor 210 adds 1 to the number i of the input box Bpi in question in S430 and proceeds to S320.

[0086] Figure 11(B) shows an example of a correspondence. This correspondence is an example of a correspondence derived from the detection results in Figure 8(B) and Figure 11(A). The figure shows an example of the positional relationship between pairs of input boxes Bp1-Bp5 and output boxes Bq1-Bq5 that form the maximum IoU. Here, input boxes and output boxes with the same number are assumed to represent the same person. In this case, input boxes and output boxes with the same number are positioned in substantially the same location and have substantially the same shape on images IM1 and IM3. The top of Figure 11(B) shows the positional relationship between the first input box Bp1 and the first output box Bq1. The first input box Bp1 and the first output box Bq1 are positioned in substantially the same location and have substantially the same shape. The IoU (1,1) of the pair of the first input box Bp1 and the first output box Bq1 is greater than the threshold TH1. Therefore, the first corresponding landmark set LMr1, which corresponds to the first input landmark set LMp1, is set to the output landmark set LMq1 of the first output box Bq1. Similarly, corresponding landmark sets are set for the other input boxes. The i-th corresponding landmark set LMri is set to the output landmark set LMqi of the same number i (i.e., the same person).

[0087] Figure 13(A) shows another example of the facial region and facial landmarks detected from the output image IM3. The difference from the detection result in Figure 11(A) is that the bounding box corresponding to face Fz2 of the second person PR2 is not detected. Output boxes Bq2-Bq4 correspond to people PR3-PR5, respectively (the code numbers of the output boxes are shifted by 1 from the code numbers of the people). Thus, a face detected in S120 (Figure 6) may not be detected in S240 (Figure 7).

[0088] Figure 13(B) shows an example of a correspondence determined by the process in Figure 12. This correspondence is an example of a correspondence derived from the detection results in Figure 8(B) and Figure 13(A). The top of Figure 13(B) shows the correspondence for the second input box Bp2. As shown in Figure 13(A), the output box corresponding to the second input box Bp2 has not been detected. Therefore, the maximum IoU(2,v) of the pair of the second input box Bp2 and the output box Bqv is smaller than the threshold TH1.

[0089] If the maximum IoU is less than or equal to the threshold TH1 (Figure 12: S390: No), in S410, processor 210 sets the i-th corresponding landmark set LMri to the i-th input landmark set LMpi, rather than the output landmark set. Then, processor 210 proceeds to S420.

[0090] As shown in Figure 13(B), the maximum IoU(2,v) of the second input box Bp2 is less than the threshold TH1. Therefore, the corresponding landmark set LMr2, which corresponds to the second input landmark set LMp2, is set to the original second input landmark set LMp2.

[0091] For other individuals PR1, PR3-PR5, the corresponding landmark sets for the input landmark sets are set to the output landmark sets of the same individual. For example, the corresponding landmark set LMr3 for the third input box Bp3's third input landmark set LMp3 is set to the output landmark set LMq2 for the second output box Bq2 of the same individual.

[0092] Thus, in the process shown in Figure 12 (i.e., the process at S250 in Figure 7), the processor 210 associates the output landmark sets of the same person (Figures 11(A) and 13(A)) with the input landmark set (Figure 8(B)) (Figure 12: S400). If no landmark set of the same person is detected from the output image IM3, the processor 210 sets the corresponding landmark set to the same input landmark set (Figure 12: S410). The reason for this will be explained later.

[0093] After the process shown in Figure 12, i.e., the process in S250 in Figure 7, the processor 210 proceeds to S260. Also, if the total number of bounding boxes Nfp detected in S120 (Figure 6) is zero (S140: No), the processor 210 proceeds to S260. In S260, the processor 210 calculates the loss and adjusts the computational parameters of the style transfer model M2 (Figure 2) to minimize the loss. In this embodiment, the processor 210 uses the trained encoder EC to adjust the computational parameters of the decoder DC.

[0094] Figures 14(A) and 14(C) show examples of loss calculation formulas. As shown in Figure 14(A), the loss L (also called the total loss L) is the weighted sum of the content loss Lc, style loss Ls, face count loss Ln, and landmark loss Ll. The losses Lc, Ls, Ln, and Ll are assigned weights of 1, λs, λn, and λl, respectively. The variable weights λs, λn, and λl are determined experimentally in advance so that the trained style transfer model M2 performs style transfer appropriately.

[0095] Content loss Lc and style loss Ls are the same as those described in the aforementioned AdaIN paper.

[0096] Content loss Lc represents the difference between the feature map f(g(t)) output from the encoder EC when the converted content image g(t) (Figure 2) is input to the encoder EC, and the target feature map t.

[0097] The style loss Ls represents the difference between the data output from multiple layers of the encoder EC when a converted content image g(t) is input to the encoder EC, and the data output from multiple layers of the encoder EC when a style image s is input to the encoder EC. The difference used is the difference in means and the difference in standard deviations. For example, four layers are used as the multiple layers of the encoder EC: relu1_1, relu2_1, relu3_1, and relu4_1. relu1_1, relu2_1, relu3_1, and relu4_1 represent the activation functions of the first convolutional layers EL02, EL05, EL08, and EL13 (Figure 3) of the first to fourth blocks, respectively.

[0098] Figure 14(B) shows an example of the formula for calculating the face count loss Ln. The input face count Nfp is the total number of bounding boxes detected from the input image (for example, input image IM1 (Figure 8(B))) in S120 of Figure 6. The output face count Nfq is the total number of bounding boxes detected from the output image (for example, output image IM3 (Figure 11(A), Figure 13(A))) in S240 of Figure 7. The face count loss Ln may be various values ​​representing the difference between the input face count Nfp and the output face count Nfq. In this embodiment, the face count loss Ln is calculated by multiplying the square of the difference in the total number of bounding boxes by the input face count N It is calculated by dividing by fp. When the style transfer model M2 is trained to minimize the overall loss L, which includes the face count loss Ln, the likelihood that the total number of bounding boxes will change due to style transfer by the style transfer model M2 is reduced. For example, the likelihood that the bounding box of the second person PR2 will not be detected, as shown in the detection result in Figure 13(A), is reduced. Note that if the input number of faces Nfp is zero (Figure 6: S140: No), the face count loss Ln is set to zero.

[0099] Figure 14(C) shows an example of the formula for calculating the landmark loss Ll. d(LMpi, LMri) represents the distance between the input landmark set LMpi of the face image before style transformation and the corresponding landmark set LMri of the face image after style transformation. As shown in Figure 9(B), in this embodiment, the landmark sets LMpi and LMri are each represented as vectors. The distance d may be various values ​​representing the difference between the elements of the vectors. In this embodiment, the distance d is the Euclidean distance. A large distance d(LMpi, LMri) indicates a large degree of deformation due to style transformation in parts of the face (e.g., eyes). A small distance d(LMpi, LMri) indicates a small degree of deformation due to style transformation in parts of the face.

[0100] R(LMpi) represents the size of the face indicated by the input landmark set LMpi. The size R can be any value representing the size of the face. In this example, the size R(LMpi) is the length of the diagonal of the input box Bpi corresponding to the input landmark set LMpi.

[0101] The landmark loss Ll is the average of the distances d normalized by the face size R. Here, the average is the average of the Nfp faces detected from the input image (e.g., input image IM1 (Figure 8(B))). The reason for normalizing by the face size R is to mitigate the effects of size differences between multiple faces. If the style transfer model M2 is trained to minimize the overall loss L, which includes the landmark loss Ll, ​​the likelihood of parts of a face being deformed by the style transfer performed by the style transfer model M2 is reduced. Note that if the number of input faces Nfp is zero (Figure 6: S140: No), the landmark loss Ll is set to zero.

[0102] As explained in S410 (Figure 12) and Figure 13(A), the output box (and therefore the output landmark set) of the same person as the person in the input landmark set LMpi may not be detected in the output image. In this case, the corresponding landmark set LMri is the same as the input landmark set LMpi. Therefore, the distance d is zero. Let's assume that the distance d increases when the output box of the same person is not detected in the output image. In this case, the landmark loss Ll increases for reasons other than deformation of the parts included in the face (i.e., the output box of the same person is not detected). Such a landmark loss Ll can hinder the proper training of the style transfer model M2. In this embodiment, the possibility of such a problem is reduced.

[0103] The processor 210 calculates the total loss L for each pair of batch size BS (Figure 6: S110). Then, the processor 210 uses the total losses L of BS to calculate an adjustment loss. The adjustment loss is, for example, the average value of the total losses L of BS. The processor 210 adjusts several computational parameters of the style transformation model M2 according to a predetermined algorithm so that the adjustment loss is small. As an algorithm, for example, an algorithm using backpropagation and gradient descent may be employed. The processor 210 may also perform so-called Adam optimization.

[0104] In S270 (Figure 7), the processor 210 determines whether the training termination condition is met. The termination condition may be any condition indicating that the style transfer model M2 has been properly trained. In this embodiment, the termination condition is that training for a given number of epochs is completed. Although not shown in the figure, the processor 210 may change the set of input image and style image pairs for each batch at each epoch. The number of epochs is determined experimentally to allow the style transfer model M2 to be properly trained.

[0105] The training termination condition may be anything else. For example, the termination condition may be that each of the total losses L calculated using a predetermined number of input image-style image pairs not used in training is below a predetermined threshold.

[0106] If it is determined that training is not yet complete (S270: No), the processor 210 proceeds to S110 (Figure 6) and trains the style transfer model M2 using a new pair of input images and style images. If it is determined that training is complete (S270: Yes), in S280, the processor 210 stores data representing the trained style transfer model M2 in the storage device 215 (in this case, the non-volatile storage device 230). Then, the processor 210 terminates the training process (Figures 6 and 7).

[0107] A4. Image processing: Figures 15 and 16 are flowcharts illustrating an example of image processing using a trained style transfer model M2. Figure 16 is a continuation of Figure 15. The second program 232 (Figure 1) is a program for image processing. The user inputs an instruction to start image processing to the image processing device 200 by operating the operation unit 250. The processor 210 starts image processing according to the instruction. This image processing uses the input image and the style image to generate a transformed image that is represented in the style of the style image.

[0108] The image processing in this embodiment is composed of some of the steps S105-S280 of the training process shown in Figures 6 and 7. In Figures 15 and 16, the code for the step corresponding to the training process step is the same as the code for the corresponding step in Figures 6 and 7, but with the letter "a" added to the end. For example, S120a corresponds to S120 in Figure 6.

[0109] In S108, the processor 210 obtains data for the pair of input image and style image to be processed. For example, the processor 210 obtains data specified by the user from a storage device (e.g., a non-volatile storage device 230, a storage device (not shown) connected to the communication interface 270, etc.).

[0110] Figures 17(A)-17(D) show examples of images processed by image processing. Figure 17(A) shows an example of an input image. Input image IMa1 is a photograph of five people PRa1-PRa5. Input image IMa1 contains images of the five faces Fa1-Fa5 of the five people PRa1-PRa5.

[0111] The style images may be various images having a style different from that of the input image IMa1. For example, the style image s in Figure 2 and the input image IMa1 in Figure 17(A) may form a pair.

[0112] In S120a (Figure 15), the processor 210 detects the face region from the input image. The method for detecting the face region is the same as the detection method in S120 shown in Figure 6. Note that in S120a, the detection of face landmarks is omitted.

[0113] Figure 17(B) shows an example of a face region detected from the input image IMa1. The processor 210 uses the input image IMa1 to perform calculations on the face detection model M1a, thereby detecting five bounding boxes Bap1-Bap5 corresponding to five faces Fa1-Fa5.

[0114] In S130a (Figure 15), the processor 210 generates a processed image, which is a style-transformed image, by performing calculations on the trained style transfer model M2 using the input image and style image pair. The transformed content image g(tz) output from the decoder DC (Figure 2) is the processed image.

[0115] The weight α (Figure 4(A)) is predetermined to be greater than zero and less than or equal to 1. Alternatively, the processor 210 may use a weight α specified by the user. The larger the weight α, the larger the ratio of the target feature map t to the adjusted feature map tz (Figure 2). Therefore, the larger the weight α, the closer the style of the processed image (here, the converted content image g(tz) output from the decoder DC) will be to the style of the style image. In this way, the user can adjust the degree of style conversion by adjusting the weight α.

[0116] Figure 17(C) shows an example of a processed image. Processed image IMa2 shows the image generated from input image IMa1 (Figure 17(A)). As shown, processed image IMa2 represents people PRa1-PRa5. The faces Fas1-Fas5 of people PRa1-PRa5 are represented in a different style (in this case, the style of the style image) than the style of input image IMa1.

[0117] In S140a (Figure 15), the processor 210 determines whether the total number of bounding boxes Nfp detected in S120a is greater than zero.

[0118] If the total number Nfp is greater than zero (S140a: Yes), processor 210 proceeds to S150a. The processing in S150a-S220a is the same as the processing in S150-S220 in Figure 6. For each of the bounding boxes Bap1-Bap5 (Figure 17(B)), processor 210 performs high-resolution processing (S170a), style conversion processing (S180a), segmentation processing (S190a), masking processing (S200a), and low-resolution processing (S210a). As a result, a face image with the style appropriately converted is generated, similar to the processed face image FIkd in Figure 10.

[0119] If all bounding box processing is complete (S220a: Yes), in S230a (Figure 16), the processor 210 generates an output image by superimposing the style-transformed face image onto the corresponding positions of the style-transformed input image (for example, the overall processed image IMa2 (Figure 17(C))).

[0120] Figure 17(D) shows an example of an output image. Output image IMa3 is generated by superimposing the images of faces Faz1-Faz5, generated by the processing in S170a-S210a (Figure 15), onto the face Fas1-Fas5 region of the overall processed image IMa2 (Figure 17(C)) generated in S130a (Figure 15). In S170a-S210a, as explained in Figure 10, a style transformation of the face image is performed using the high-resolution face image. Therefore, the possibility of deformation like that of face Fs2 in Figure 8(C) is reduced. In this way, the processor 210 can generate a style-transformed output image IMa3 while suppressing face deformation, even when the face in the input image IMa1 is small.

[0121] In S290 (Figure 16), the processor 210 stores the output image data in the storage device 215 (for example, the non-volatile storage device 230). Then, the processor 210 terminates the image processing shown in Figures 15 and 16.

[0122] If the number of input faces Nfp is zero (Figure 15: S140a: No), the processor 210 skips S150a-S230a (Figures 15 and 16) and proceeds to S290. In this case, in S290, the processor 210 adopts the processed image generated in S130a (Figure 15) as the output image. The processor 210 stores the output image data in a storage device and terminates image processing.

[0123] As described above, in this embodiment, the processor 210 of the image processing device 200 performs the following processing. In S120a (Figure 15), the processor 210 performs a detection process using the input image to detect a region representing a person's face (referred to as the first region). Each region of the bounding boxes Bap1-Bap5 detected from the input image IMa1 in Figure 17(B) is an example of the first region. In S150a-S210a (Figure 15), the processor 210 generates a processed image by performing image processing using the first image of the first region of the input image. The processed image is an image of a face with a modified style (for example, the images of faces Faz1-Faz5 in Figure 17(D)). Hereinafter, the processed image generated in S210a will be referred to as the first processed image. The entire process from S150a to S210a will be referred to as the first image processing IP1a. The first image processing IP1a includes a style conversion process (S180a) (referred to as the first style conversion process).

[0124] In S130a, the processor 210 generates a processed image IMa2 (Figure 17(C)) by performing image processing using the second image of the second region of the input image IMa1 (Figure 17(A)). In this embodiment, the second region is the entire region of the input image IMa1. The second image is the same as the input image. This second region includes at least a part of the remaining region excluding the first region. Hereinafter, the processed image generated in S130a will be referred to as the second processed image. The image processing in S130a will be referred to as the second image processing IP2a. The second image processing IP2a includes style conversion processing (referred to as the second style conversion processing). The first image processing IP1a, which is one of the image processing IP1a and IP2a, includes processing that is not included in the second image processing IP2a (for example, high-resolution processing (S170a)).

[0125] In S230a (Figure 16), the processor 210 generates an output image (for example, the output image IMa3 in Figure 17(D)) by performing a synthesis process between the first processed image (for example, the images of faces Faz1-Faz5 in Figure 17(D)) and the second processed image (for example, the processed image IMa2 in Figure 17(C)).

[0126] Thus, the processor 210 executes the first image processing IP1a, which uses the face image, separately from the second image processing IP2a, which uses the image of other regions. Therefore, the processor 210 can perform image style transformation while taking the face into consideration.

[0127] Furthermore, the first image processing step IP1a (Figure 15) includes a high-resolution processing step (S170a). The processing steps S170a-S180a are the same as the processing steps S170-S180 in Figure 6 (however, the input image and style image to be processed may differ from those processed in Figure 6). The width and height of the image input to the style conversion model M2 in S180 (Figure 6) are greater than the width and height of the image enclosed by the bounding box (for example, the image of interest FIk in Figure 10). The same applies to S180a (Figure 15). Thus, the first style conversion processing step (S180a) includes a style conversion process for an image with a resolution higher than the resolution of the first image in the first region of the input image. Therefore, the possibility of deformation of parts of the face due to the first style conversion processing step (S180a) is reduced.

[0128] Furthermore, the second style transfer process (S130a) includes style transfer processing by the trained style transfer model M2. As explained in Figure 2, the style transfer model M2 performs style transfer on the image. The style transfer model M2 is an example of a style transfer model. As explained in Figures 6, 7, and 14(A), the trained style transfer model M2 is trained to minimize the overall loss L. The overall loss L includes the face count loss Ln. The face count loss Ln (Figure 14(B)) is an example of the first term related to the difference between the total number of faces Nfp detected from the input image input to the style transfer model M2 and the total number of faces Nfq detected from the image after style transfer by the style transfer model M2. With this configuration, the possibility of inappropriate style transfer that changes the total number of detected faces is reduced. For example, the possibility that the background image in the input image is transformed into an image resembling a face by style transfer is reduced.

[0129] Furthermore, the first style transfer process (180a) includes style transfer processing by the trained style transfer model M2. Style transfer model M2 is an example of a style transfer model. As explained in Figures 6, 7, and 14(A), the trained style transfer model M2 is trained to minimize the overall loss L. The overall loss L includes the landmark loss Ll. The landmark loss Ll (Figure 14(C)) is related to the difference between the input landmark set LMpi and the corresponding landmark set LMri. The input landmark set LMpi indicates the location of the first face landmark detected from the face (referred to as the first face) contained in the image input to the style transfer model M2 (e.g., input image IM1 in Figure 8(B)). The corresponding landmark set LMri is set to the output landmark set LMqv in S400 (Figure 12). The corresponding landmark set LMri (here, the output landmark set LMqv) indicates the position of the second face landmark detected from the face (referred to as the second face) contained in the style-transformed image (e.g., output image IM3 (Figure 11(A), Figure 13(A))) by the style-transformed model M2. This configuration reduces the possibility of changes in landmark positions due to the style-transformed process. In other words, it reduces the possibility of deformation of parts of the face due to the style-transformed process.

[0130] Here, the first face is the face described below. In S120 (Figure 6), the processor 210 detects bounding boxes (e.g., bounding boxes Bp1-Bp5) by performing a detection process using an input image (e.g., input image IM1 (Figure 8(B))) that represents N faces (where N is an integer greater than or equal to 2). The first face is the face represented by the region enclosed by the bounding boxes. Input image IM1 is an example of an input training image, which is an image input to the style transfer model M2 for training. The bounding box detection process in S120 is the same as the process in S120a (Figure 15) (however, the input image IM1 for training may differ from the input image IMa1 for image processing).

[0131] Furthermore, the second face is the face described below. In S240 of Figure 7, the processor 210 detects bounding boxes (e.g., bounding boxes Bq1-Bq5) representing faces from the output image (e.g., output image IM3 (Figure 11)). The second face is the face represented by the region enclosed by the bounding boxes. The output image (e.g., output image IM3 (Figure 11)) is an image generated by performing the processing in S120, S130, S150-S210, and S230 of Figures 6 and 7 using the input training image (e.g., input image IM1 (Figure 8(A))). Output image IM3 is an example of an output training image obtained using the input training image. The second face is the face of the same person as the first face, among the one or more faces detected from the output training image.

[0132] Thus, the landmark loss Ll is related to the difference between the position of the first face landmark detected from the first face and the position of the second face landmark detected from the second face of the same person as the first face. When the style transfer model M2 is trained to reduce the overall loss L, which includes the landmark loss Ll, ​​the likelihood of changes in the position of landmarks of the same person due to the style transfer process is reduced. In other words, the likelihood of deformation of parts of the face of the same person due to the style transfer process is reduced. As a result, the image processing in this embodiment (Figures 15 and 16) can maintain the visual identity of the face while changing the style of the face.

[0133] Note that the processes for generating the output training images (S120, S130, S150-S210, S230 in Figures 6 and 7) include the processes of S120a, S130a, S150a-S210a, and S230a in Figures 15 and 16 (wherein the input image and style image to be processed may differ from those in Figures 15 and 16). Specifically, the process of S120 includes the process of S120a (bounding box detection process). The process of S130 includes the second image processing IP2a (here, S130a). The processes of S150-S210 include the first image processing IP1a (here, S150a-S210a). The process of S230 includes the process of S230a (here, synthesis process). Thus, the output training image is generated by performing a process including detection processing (S120a), first image processing IP1a, second image processing IP2a, and synthesis processing (S230a) using the input training image (for example, input image IM1 (Figure 8(A))).

[0134] Thus, the output training image used for detecting the second face in training the style transfer model M2 is generated by image processing (Figures 15 and 16), which includes processing steps (S120a, S130a, S150a-S210a, S230a) to generate the output image. In other words, the training of the style transfer model M2 is performed taking into account the generation of the output image by image processing (Figures 15 and 16). Therefore, when the trained style transfer model M2 is used in image processing, the possibility of deformation of parts of the face due to the style transfer process is appropriately reduced.

[0135] Furthermore, the processor 210 determines the correspondence between the first face and the second face of the same person by executing the process shown in Figure 12. In this embodiment, as explained in S340, S370, S380, and S390, the second face associated with the first face is one or more faces detected from the output training image (e.g., output image IM3 (Figure 11)) that satisfy the condition that IoU is greater than the threshold TH1 (S390: Yes). The IoU is calculated using the first rectangle surrounding the first face (here, the input box Bpi) and the second rectangle surrounding the face detected from the output training image (here, the output box Bqj). When the IoU is large, the positional difference between the first rectangle (input box Bpi) and the second rectangle (output box Bqj) is small. "IoU>TH1" is an example of a positional condition that indicates that the positional difference between the first rectangle (input box Bpi) and the second rectangle (output box Bqj) is small. If the positional condition is met (S390: Yes), in S400, the processor 210 associates the second rectangle (output box Bqj) that satisfies the positional condition with the first rectangle (input box Bpi). This allows the processor 210 to appropriately determine the correspondence between the first and second faces of the same person.

[0136] Furthermore, if multiple faces are positioned close together in the image, multiple second rectangles (output box Bqj) may satisfy the positional condition for a single first rectangle (input box Bpi). In this case, the processor 210 associates the second rectangle (output box Bqj) with the largest IoU with the first rectangle (input box Bpi). Therefore, the processor 210 can appropriately determine the correspondence between the first and second faces of the same person.

[0137] B. Variations: (1) The super-resolution model M3 (Figure 1) is not limited to StyleGAN, but may be any generative model that generates high-resolution natural images, such as variational autoencoders (VAEs) or generative adversarial networks (GANs). Also, the high-resolution processing in S170 (Figure 6) and S170a (Figure 15) may be any other processing instead of the super-resolution processing called "PULSE". For example, the processor 210 may perform resolution conversion processing (nearest neighbor, bilinear, bicubic, etc.) without using a machine learning model such as the super-resolution model M3.

[0138] (2) In S120 (Figure 6), S240 (Figure 7), and S120a (Figure 15), the face region detection process may be replaced with various other processes instead of using an object detection model called "BlazeFace". For example, the processor 210 may use an object detection model called YOLO (You Only Look Once) to detect the face region. Alternatively, the processor 210 may detect the face region by pattern matching using a reference image of the face, without using a machine learning model.

[0139] (3) The process of detecting facial landmarks in S120 (Figure 6) and S240 (Figure 7) may be replaced with various other processes instead of using a detection model called "Attention Mesh". For example, the processor 210 may detect facial landmarks by using the Facemark API of OpenCV (Open Source Computer Vision). Alternatively, the processor 210 may detect landmarks by pattern matching using reference images of parts of the face (mouth, eyes, etc.).

[0140] The number of points nP in the landmark (Figure 9) is not limited to 68, but can be any number. In order to suppress deformation of parts included in the face using the landmark loss Ll (Figure 14(C)), it is preferable to have a large number nP. For example, it is preferable that the number nP is 20 or more. Note that the landmark may represent any part of the face. The landmark may include one or more parts selected from the group consisting of eyebrows, eyes, mouth, nose, and chin.

[0141] (4) The segmentation processes in S190 (Figure 6) and S190a (Figure 15) may be replaced with various other processes instead of the process using FCN. For example, an object detection model called "Mask R-CNN" may be used. The processor 210 may also divide the region representing the face from the other region by pattern matching using a reference image of the face.

[0142] (5) In steps S340, S370, S380, and S390 of Figure 12, one or more faces detected from the output training image (e.g., output image IM3 (Figure 11)) that satisfy the condition "IoU>TH1 (S390:Yes)" are selected as the second face of the same person as the first face detected from the input training image (e.g., input image IM1 (Figure 8(B))). Various positional conditions can be used as correspondence conditions, which are the conditions for selecting the face detected from the output training image as the second face of the same person. Here, the positional conditions are the first rectangle surrounding the first face (e.g., input box Bpi) and the face detected from the output training image. This is a condition indicating that the positional difference between the first rectangle (for example, output box Bqj) and the second rectangle is small. The positional condition may include, for example, that the distance between the centroid of the first rectangle and the centroid of the second rectangle is less than a threshold. The positional condition may also include that the positional difference between a vertex of the first rectangle and the corresponding vertex of the second rectangle is less than a threshold. The positional condition may also include that each of the four positional differences in the four combinations of vertices of the first rectangle and corresponding vertices of the second rectangle is less than a threshold. Furthermore, the correspondence condition is not limited to the positional condition, but may be various conditions indicating that the first face and the second face belong to the same person.

[0143] (6) The style transfer model is not limited to the style transfer model M2 in Figure 2, but may be any model. For example, the architecture of a technology called "Fast Patch-based Style Transfer of Arbitrary Style" or the architecture of a technology called "Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration" may be adopted.

[0144] In any case, the width, height, and number of channels of an image that can be input to the style transfer model may be various values. The style transfer model may be configured to accept, for example, an image represented as 512*512*3.

[0145] (7) The first image processing that generates a first processed image using a first image of a first region representing a person's face is not limited to the first image processing IP1a in Figure 15 (specifically, S150a-S210a), but may include various processes including a first style conversion process. For example, the masking process (S200a) may be performed after the low-resolution processing (S210a). The masking process (S200a) may be omitted. In this case, in S230a (Figure 16), the entire image within the bounding box containing the style-converted face may be superimposed on the overall processed image IMa2 (Figure 17(C)). The high-resolution processing (S170a) may also be omitted.

[0146] (8) The second image processing, which generates a second processed image using the second image of the second region of the input image, is not limited to the second image processing IP2a (specifically S130a) in Figure 15, but may include various processes including a second style conversion process. For example, the second image processing may include a high-resolution processing process. The style-converted high-resolution face image may then be superimposed on the style-converted high-resolution overall processed image.

[0147] In either case, the second region is not limited to the entire input image, but may include various regions that include at least a part of the remaining region of the input image excluding the first region representing the face. For example, the second region may be the remaining region of the input image excluding the first region. That is, the second image of the second region may be an image obtained by masking the first region.

[0148] (9) The first image processing and the second image processing may each be various types of processing. For example, the first image processing may include processing not included in the second image processing (e.g., high-resolution processing). The second image processing may include processing not included in the first image processing (e.g., sharpness enhancement processing).

[0149] The first style conversion process included in the first image processing may be different from the second style conversion process included in the second image processing. For example, the style conversion model used in the second style conversion process may be different from the style conversion model used in the first style conversion process. Also, the style image used in the second style conversion process may be different from the style image used in the first style conversion process. The user may specify the style image for the first style conversion process and the style image for the second style conversion process.

[0150] (10) The input training images that are input to the style transfer model during the training process of the style transfer model (e.g., Figures 6 and 7) preferably include a variety of images of the same type as the images that can be input to the style transfer model in image processing using the trained style transfer model (e.g., Figures 15 and 16). For example, the input training images may include a variety of photographic images such as landscape images without people, portraits of one person, and group photographs of multiple people. The input training images may also include images of a different type from photographic images, such as illustrations.

[0151] (11) The loss used to train the style transfer model is not limited to the losses described in Figures 14(A)-14(C), but may be various other losses. For example, the face count loss Ln may be various values ​​that represent the difference between the input number of faces Nfp and the output number of faces Nfq. For example, the face count loss Ln may be the absolute value of the difference between the input number of faces Nfp and the output number of faces Nfq.

[0152] Furthermore, the landmark loss Ll (Figure 14(C)) may be various values ​​representing the difference between the input landmark set LMpi of the face image before style transformation and the corresponding landmark set LMri of the face image after style transformation. For example, R(LMpi) may be various values ​​representing the size of the face indicated by the input landmark set LMpi. R(LMpi) may be the area of ​​the input box Bpi corresponding to the input landmark set LMpi. Also, the distance d may be various values ​​representing the difference between elements of a vector, such as Manhattan distance, Chebyshev distance, or Mahalanobis distance, instead of Euclidean distance.

[0153] Note that either or both of the face count loss Ln and the landmark loss Ll may be omitted. To reduce the possibility of face deformation due to style transformation, it is preferable to use the landmark loss Ll in training the style transformation model used in the first style transformation process (e.g., S180a in Figure 15) that processes the extracted face images. This maintains the visual identity of faces between the input image and the output image. Furthermore, to reduce the possibility of changes in the number of faces detected due to style transformation, it is preferable to use the face count loss Ln in training the style transformation model used in the second style transformation process (e.g., S130a in Figure 15) that processes images that include parts of the input image other than faces. This reduces the possibility that parts of the input image that do not show faces (e.g., background parts) will be transformed into face-like images by style transformation.

[0154] (12) The training process for the style transfer model is not limited to the processes shown in Figures 6 and 7, but may be any other process suitable for the style transfer model. In any case, the training process may include the following processes. 1) Output training images are generated by performing image processing corresponding to image processing using a pre-trained style transfer model, using the input training images. 2) Calculate the loss using the output training images. 3) Adjust multiple computational parameters of the style transfer model to minimize losses. Such training processes can train a style transfer model to be suitable for image processing. When the image processing using the trained style transfer model includes a first image processing IP1a and a second image processing IP2a, as shown in Figure 15, it is preferable that the training process includes processing corresponding to the first image processing and processing corresponding to the second image processing.

[0155] (13) The faces detected from the input image are not limited to human faces, but may be faces of various living creatures. For example, faces of pets such as dogs and cats may be detected. Faces of various mammals may be detected, not limited to dogs and cats.

[0156] (14) The image processing device using the trained style transfer model may be a different type of device from a personal computer (e.g., a digital camera, scanner, or smartphone). Alternatively, multiple devices (e.g., computers) that can communicate with each other via a network may share some of the image processing functions performed by the image processing device, and together they may provide the image processing functions (a system comprising these devices corresponds to the image processing device).

[0157] The training process for the style transfer model may be performed by a device other than the image processing device that performs the image processing.

[0158] In each of the above embodiments, some of the configurations implemented by hardware may be replaced with software, and conversely, some or all of the configurations implemented by software may be replaced with hardware. For example, the functionality of a style conversion model (e.g., style conversion model M2) may be implemented by a dedicated hardware circuit.

[0159] Furthermore, if some or all of the functions of this disclosure are implemented by a computer program, that program may be provided in the form of a computer-readable recording medium (e.g., a non-temporary recording medium). The program may be used while stored on the same or a different recording medium (computer-readable recording medium) as it was provided. "Computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but may also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as hard disk drives.

[0160] The embodiments described above are for the purpose of facilitating understanding of this disclosure and do not limit the present invention. The present invention can be modified and improved without departing from its spirit, and equivalents thereof are included. [Explanation of Symbols]

[0161] 200…Image processing device, 210…Processor, 215…Storage device, 220…Volatile storage device, 230…Non-volatile storage device, 231…First program, 232…Second program, 240…Display unit, 250…Operation unit, 270…Communication interface

Claims

1. It is a computer program, A detection function that performs a detection process to detect the first region representing the face of an organism using the input image, A first generation function that generates a first processed image by performing first image processing using the first image of the first region of the input image, wherein the first image processing includes a first style conversion process, and the first generation function A second generation function that generates a second processed image by performing a second image processing using a second image of a second region of the input image, wherein the second region is a region that includes at least a part of the remaining region excluding the first region, the second image processing includes a second style conversion process, and the first specific processing, which is one of the first image processing and the second image processing, includes processing that is different from the first specific processing and is not included in the second specific processing, the second generation function, A third generation function generates an output image by performing a synthesis process between the first processed image and the second processed image, To make this a reality on a computer, The second style transfer process includes a style transfer process using a trained style transfer model. The aforementioned trained style transfer model has been trained to minimize losses. The loss includes a first term relating to the difference between the number of faces detected from the image input to the style transfer model and the number of faces detected from the image after the style transfer model has performed the style transfer. Computer program.

2. A computer program according to claim 1, The first image processing described above includes a high-resolution processing, The first style conversion process includes a style conversion process for an image with a resolution higher than the resolution of the first image in the first region of the input image. Computer program.

3. A computer program according to claim 1 or 2, The first style transfer process includes a style transfer process using a trained style transfer model, The aforementioned trained style transfer model has been trained to minimize losses. The loss includes a second term relating to the difference between the position of a first face landmark detected from a first face included in the image input to the style transfer model and the position of a second face landmark detected from a second face included in the image after the style transfer model has performed the style transfer. Computer program.

4. A computer program according to claim 3, The first face is a face detected by performing the detection process using input training images representing N faces (where N is an integer of 2 or more). The second face is one or more faces detected from the output training image, which is generated by performing a process including the detection process, the first image processing, the second image processing, and the synthesis process using the input training image, and is the face of the same individual as the first face. Computer program.

5. A computer program according to claim 4, The second face is one of the one or more faces detected from the output training image that satisfies the positional condition that the difference in position between the first rectangle surrounding the first face and the second rectangle surrounding the face detected from the output training image is small. Computer program.

6. An image processing device, A detection unit performs a detection process using an input image to detect a first region representing the face of a living organism, A first generation unit generates a first processed image by performing first image processing using the first image of the first region of the input image, wherein the first image processing includes a first style conversion process, A second generation unit generates a second processed image by performing a second image processing using a second image of a second region of the input image, wherein the second region is a region that includes at least a part of the remaining region excluding the first region, the second image processing includes a second style conversion process, and the first specific processing, which is one of the first image processing and the second image processing, includes processing that is different from the first specific processing and is not included in the second specific processing, the second generation unit A third generation unit generates an output image by performing a synthesis process between the first processed image and the second processed image, Equipped with, The second style transfer process includes a style transfer process using a trained style transfer model. The aforementioned trained style transfer model has been trained to minimize losses. The loss includes a first term relating to the difference between the number of faces detected from the image input to the style transfer model and the number of faces detected from the image after the style transfer model has performed the style transfer. Image processing device.

Images