Multi-view portrait skin color migration method and device, medium and equipment

Abstract

The embodiment of the invention discloses a multi-view portrait skin color migration method, a multi-view portrait skin color migration device, a medium and equipment, relates to the technical field of three-dimensional portrait models, and aims to solve the problem of non-uniform texture color of a three-dimensional portrait model caused by skin color difference of multi-view portrait images. The method comprises the following steps: preprocessing a reference RGB portrait image, and calculating histogram features and color moment vectors of the reference RGB portrait image; the method comprises the following steps: processing a source RGB portrait image through an encoder-decoder network to obtain a submerged space feature of the source image; mapping the histogram feature, the color moment vector and the submerged space feature of the source image to a submerged space of an output feature of a generation module through a convolutional layer, and outputting a target image similar to the skin color of a reference RGB portrait image; wherein the target image, except for the portrait skin color, is the same as the source RGB portrait image.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional portrait models, and in particular to a multi-view portrait skin color migration method, device, medium and equipment. Background Art

[0002] The construction of a 3D portrait model requires realistic textures, typically acquired from multi-view RGB cameras. However, lighting conditions can vary significantly from viewpoint to viewpoint, leading to noticeable color differences in the facial region between texture images from different viewpoints. Consequently, the texture color of the portrait model after texture fusion also varies significantly across different regions, with distinct boundaries between different color regions. While some image processing software, such as Photoshop, can manually match the facial region colors of facial images from different viewpoints to those from a specific viewpoint, this increases the workload and learning cost. This color matching of portraits from different viewpoints can also be considered a style transfer task. Currently, some work based on neural networks such as U-Net and StyleGAN can transfer color styles, but these efforts focus on the overall color style of the portrait image, leaving significant differences in the local color style of facial skin tone.

[0003] That is, the existing technology cannot effectively solve the problem of uneven texture and color of three-dimensional portrait models caused by skin color differences in multi-view portrait images. Summary of the Invention

[0004] The main purpose of this application is to provide a multi-view portrait skin color migration method, device, medium and equipment, aiming to solve the problem of uneven texture color of three-dimensional portrait models caused by skin color differences in multi-view portrait images.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows: In a first aspect, an embodiment of the present application provides a multi-view portrait skin color migration method, comprising the following steps: Preprocess the reference RGB portrait image and calculate its histogram features and color moment vector; Processing the source RGB portrait image through an encoder-decoder network to obtain the latent space features of the source image; The histogram features, the color moment vectors, and the latent space features of the source image are mapped to the latent space of the output features of the generation module through a convolutional layer, and a target image with a skin color similar to that of the reference RGB portrait image is output; wherein, except for the skin color of the portrait, all other colors of the target image are similar to those of the reference RGB portrait image.

[0006] As some optional implementations of the present application, the reference RGB portrait image is preprocessed to calculate its histogram features and color moment vector, including: The reference RGB portrait image is used for data preprocessing; Calculate and obtain the first feature of the histogram and the second feature of the histogram; the first feature of the histogram , Represents histogram features The dimension of the second feature of the histogram , Represents histogram features Dimension

[0007] Based on the facial skin area of ​​the reference RGB portrait image, a first color moment vector and a second color moment vector are calculated; the first color moment vector , Represents the first vector of color moment The dimension of the second color moment vector , Represents the second vector of color moment Dimension

[0008] As some optional implementations of the present application, the calculation to obtain the first feature of the histogram and the second feature of the histogram includes: Converting the color value of each channel of the RGB channel of the reference RGB portrait image into a logarithmic color space to obtain a color value in the logarithmic color space; Then, the color values ​​in the logarithmic chromaticity space are divided into histogram feature buckets to obtain a first histogram feature and a second histogram feature.

[0009] As some optional implementations of the present application, the calculation of the first color moment vector and the second color moment vector based on the facial skin area of ​​the reference RGB portrait image includes: Based on the reference RGB portrait image, a mask image containing only the facial skin color area is obtained through analysis; After masking the reference RGB portrait image with the mask image, the first-order color moment and the second-order color moment of the skin color area in the image are calculated; the first-order color moment refers to the color mean of the three channels in the image, and the second-order color moment refers to the color difference of the three channels in the image; Reparameterization processing is adopted to apply the first-order color moment and the second-order color moment to a randomly sampled Gaussian distribution noise vector to obtain a first color moment vector and a second color moment vector.

[0010] As some optional embodiments of the present application, mapping the histogram features, the color moment vector, and the latent space features of the source image to the latent space of the output features of the generation module through a convolution layer, and outputting a target image with a skin color similar to that of the reference RGB portrait image, includes: Passing the first feature of the histogram and the second feature of the histogram through a histogram projection network, a latent code mapping network, and a modulation-demodulation module respectively, to obtain a weight value of the first feature of the histogram and a weight value of the second feature of the histogram; Modulating the three channels corresponding to the first color moment vector based on the weight value of the first feature of the histogram to obtain a first weight value for the portrait skin color modulation; modulating the three channels corresponding to the second color moment vector based on the weight value of the second feature of the histogram to obtain a second weight value for the portrait skin color modulation; The first weight value of the portrait skin color modulation and the second weight value of the portrait skin color modulation are convolved with the latent space features of the source image respectively, and then input into the last two generation blocks of HistoGAN with the noise vector to output a target image with a skin color similar to the reference RGB portrait image.

[0011] As some optional implementations of the present application, the first feature of the histogram and the second feature of the histogram are respectively passed through a histogram projection network, a latent code mapping network, and a modulation-demodulation module in sequence to obtain a weight value of the first feature of the histogram and a weight value of the second feature of the histogram, including: Passing the first histogram feature and the second histogram feature through a histogram projection network and a latent code mapping network, respectively, to obtain a first histogram style latent code vector with a length of 32 and a second histogram style latent code vector with a length of 64; The first histogram style latent code vector and the second histogram style latent code vector are respectively input into two networks mod-demod with the same structure as the modulation-demodulation module of StyleGAN2 In the process, the weights in the two networks are modulated to obtain the weight value of the first feature of the histogram and the weight value of the second feature of the histogram.

[0012] As some optional implementations of the present application, the pre-trained ReHistoGAN model weights are used as the initial weights of the ReMomentGAN network to reuse the prior knowledge in its model. The pre-trained ReHistoGAN model weights are publicly available through open source projects, and the ReMomentGAN network is unsupervised fine-tuned. The unsupervised fine-tuning satisfies the following relationship:

[0013]

[0014] in, represents the loss function used to fine-tune the HistoGAN discriminator; Represents the discriminator loss term in HistoGAN; Represents the gradient penalty term in the WGAN-GP model; represents the loss function used to fine-tune all modules of the ReMomentGAN network; express The weight of the loss term, ; Expressed as color moment loss term; Expressed as The weight of the loss term, ; ( ) represents the histogram loss term proposed in ReHistoGAN; Represents the histogram features of the target image output by ReMomentGAN ; Represented as the histogram feature of the reference RGB portrait image ; Expressed as The weight of the loss term, ; ( ) represents the image reconstruction loss term proposed by ReHistoGAN; Represented as source RGB portrait image; represents the target image; Expressed as The weight of the loss term, ; Represents the discriminant output value of the source RGB portrait image; Expressed as The weight of the loss term, ; ( ) represents a loss term proposed by ReHistoGAN to prevent the model from only applying global color bias.

[0015] In a second aspect, the embodiments of the present application further provide a multi-view portrait skin color migration device, comprising: The extraction module is used to preprocess the reference RGB portrait image and calculate its histogram features and color moment vector; The encoding and decoding module is used to process the source RGB portrait image through an encoder-decoder network to obtain the latent space features of the source image; The output module is used to map the histogram features, the color moment vector and the latent space features of the source image to the latent space of the output features of the generation module through a convolution layer, and output a target image with a skin color similar to that of the reference RGB portrait image; wherein, except for the skin color of the portrait, the other colors of the target image are also similar to those of the reference RGB portrait image.

[0016] In a third aspect, an embodiment of the present application further provides a device, including: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, the one or more programs being configured to execute the method as described above.

[0017] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which program code is stored, and the program code can be called by a processor to execute the method described above.

[0018] Compared with the existing technology, the present application proposes a multi-view portrait skin color migration method, that is, by pre-processing the reference RGB portrait image, calculating and obtaining the histogram features and color moment vectors; then processing the source RGB portrait image through the encoder-decoder network to obtain the latent space features of the source image; finally, mapping the histogram features, the color moment vectors and the latent space features of the source image to the latent space of the output features of the generation module through the convolution layer, and outputting a target image with a skin color similar to that of the reference RGB portrait image. It can be seen that the method described in the present application is used to solve the problem of uneven texture color of the three-dimensional portrait model caused by skin color differences of multi-view portrait images in digital dental treatment or other special application scenarios, that is, by accurately migrating the skin color features of the reference image to the input image to eliminate the multi-view skin color differences, avoid artifacts and color distortion, and thus improve the color uniformity and realism of the three-dimensional model after texture fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present application; Figure 2 This is a structural diagram of the RGB portrait skin color matching network ReMomentGAN involved in an embodiment of the present application; Figure 3 A schematic diagram of the process of a multi-view portrait skin color migration method provided in an embodiment of the present application; Figure 4 (a) RGB portrait provided in the embodiment of the present application; (b) corresponding mask image containing only the facial skin color area; Figure 5Comparison of skin color migration results of four methods on the test set; (a) reference image (b) input image (c) result of the proposed method (d) result of the ReHistoGAN method (e) result of the Deep preset method (f) result of the Palette-based method; Figure 6 A schematic diagram of a module of a multi-view portrait skin color migration device provided in an embodiment of the present application; Markings in the figure: 101 - processor, 102 - communication bus, 103 - network interface, 104 - user interface, 105 - memory. DETAILED DESCRIPTION

[0020] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0021] Refer to the attached Figure 1 , attached Figure 1 This is a schematic diagram of the structure of an electronic device of the hardware operating environment involved in the embodiment of the present application. The electronic device may include: a processor 101, such as a central processing unit (CPU), a communication bus 102, a user interface 104, a network interface 103, and a memory 105. Among them, the communication bus 102 is used to realize the connection and communication between these components. The user interface 104 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and optionally the user interface 104 may also include a standard wired interface and a wireless interface. The network interface 103 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 105 may optionally be a storage device independent of the aforementioned processor 101. The memory 105 may be a high-speed random access memory (RAM) memory, or a stable non-volatile memory (NVM), such as at least one disk storage. The processor 101 may be a general-purpose processor, including a central processing unit, a network processor, etc., or may be a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component.

[0022] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation to the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0023] As attached Figure 1As shown, the memory 105 as a storage medium may include an operating system, a network communication module, a user interface module, and a multi-view portrait skin color migration device.

[0024] In the attached Figure 1 In the electronic device shown, the network interface 103 is mainly used for data communication with the network server; the user interface 104 is mainly used for data interaction with the user; the processor 101 and the memory 105 in this application can be set in the electronic device, and the electronic device calls the multi-view portrait skin color migration device stored in the memory 105 through the processor 101, and executes the multi-view portrait skin color migration method provided in the embodiment of this application.

[0025] Based on the hardware devices of the aforementioned embodiments, this application proposes a multi-view portrait texture image skin color migration method ReMomentGAN based on ReHistoGAN, which is used to solve the problem of uneven texture color of three-dimensional portrait models caused by skin color differences in multi-view portrait images in digital dental diagnosis and treatment. It aims to achieve the following goals: accurately transfer the skin color features of the reference image to the input image, eliminate multi-view skin color differences; keep the portrait identity information unchanged, avoid artifacts and color distortion; improve the color uniformity and realism of the three-dimensional model after texture fusion; that is, this application proposes a method based on ReHistoGAN. Figure 2 The ReMomentGAN network shown, the network input is a three-channel RGB portrait , histogram features calculated based on the reference image , , and the color moment vector calculated based on the facial skin area of ​​the reference image , the output is an image with skin color similar to the reference RGB portrait .

[0026] Refer to the attached Figure 3 , an embodiment of the present application provides a multi-view portrait skin color migration method, including steps S10 to S30: Step S10: pre-process the reference RGB portrait image to calculate and obtain histogram features and color moment vectors.

[0027] Step S20: Process the source RGB portrait image through an encoder-decoder network to obtain latent space features of the source image.

[0028] Step S30: Map the histogram features, the color moment vector, and the latent space features of the source image to the latent space of the output features of the generation module through a convolutional layer, and output a target image with a skin color similar to that of the reference RGB portrait image; wherein, except for the skin color of the portrait, all other colors of the target image are similar to those of the reference RGB portrait image.

[0029] In a specific embodiment, step S10 includes: Step S11: Preprocess the reference RGB portrait image to calculate histogram features and color moments. The ReMomentGAN network, based on the ReHistoGAN network, is a network focused on skin color transfer. The ReHistoGAN network is a jointly trained encoder network for unsupervised image recoloring. It further expands upon HistoGAN, an innovative deep learning method that uses color histograms to control the color of GAN-generated images and can even perform color transformations on real images. The core of ReHistoGAN lies in its modification of the StyleGAN architecture, enabling it to maintain the original image content while only altering the color according to a given target histogram. Specifically, the ReMomentGAN network, building on the ReHistoGAN network, introduces color moments into the modulation-demodulation module and proposes a new loss function—the color moment loss function. This transforms the overall color transfer capability of the ReHistoGAN network into precise skin color transfer capability, eliminating skin color differences between multiple viewpoints while ensuring that identity details are not lost. This improvement has significant application advantages in scenarios such as digital dental treatment that require local color correction. For example, consider images of a user captured from multiple perspectives, but with uneven facial skin tone due to varying lighting, which affects texture fusion in the 3D model. To address this, select an image with uniform lighting as a reference image, calculate its histogram features and color moments, and then combine it with images from other perspectives as source images before feeding them into the ReMomentGAN network. The output is a target image with a similar skin tone to the reference image, but unchanged in content. This eliminates the problem of uneven facial skin tone in multi-perspective images.

[0030] The ReMomentGAN network is trained using the following strategy: This application uses the pre-trained ReHistoGAN model weights as the initial weights of the ReMomentGAN network and the pre-trained HistoGAN discriminator weights as the initialization of the discriminator. And the image after skin color migration output by ReMomentGAN The following loss functions are used to perform unsupervised fine-tuning on the HistoGAN discriminator and all modules of the ReMomentGAN network: (2-9) (2-10) in, Represents the weight of each loss term in the loss function (2-10). In formula (2-9), is the discriminator loss term in HistoGAN: (2-11) in, represents the output of the discriminator, represents the source image, Represents the target image after skin color migration output by ReMomentGAN. In formula (2-9) is the gradient penalty term in the WGAN-GP model, used to improve training stability: (2-12) in, is the gradient penalty coefficient, is the Hamiltonian differential operator, Represents the discriminator's response to the source image The discriminant value of yes Norm. In formula (2-10), is the color moment loss term shown in formula (2-11), is the histogram loss term proposed in ReHistoGAN, which represents the histogram of the generated target image and the histogram features of the reference image between Norm: (2-13) in, for The histogram features of Represents the target image after skin color migration output by ReMomentGAN, for The histogram features of As a reference image for portrait skin color migration, is the square root of the histogram feature of the target image, is the square root of the histogram feature of the reference portrait image. This is the image reconstruction loss proposed in ReHistoGAN: (2-14) in, Indicates that the Laplacian operator is used on the image, so that the loss term focuses more on the perceptual details of the generated image rather than maintaining the source image The color value of Represents the target image after skin color migration output by ReMomentGAN, yes norm.

[0031] In formula (2-10), A loss term proposed for ReHistoGAN to prevent the model from only applying global color cast express Three channels: (2-15) in, is the Gaussian blur kernel function, To calculate the standard deviation operation, , for The histogram features of For reference image The histogram features of is the source image aisle, , is the target image after skin color migration of aisle, .

[0032] Formula (2-10) is the discriminant value of the image after skin color matching, and the weights of each loss term in the loss function (2-10) are .

[0033] Step S12: Calculate and obtain the first feature of the histogram and the second feature of the histogram; the first feature of the histogram , Represents histogram features The dimension of the second feature of the histogram , Represents histogram features dimension.

[0034] Specifically, this step includes: converting the color values ​​of each channel of the RGB channel of the reference RGB portrait image into a logarithmic chromaticity space to obtain the color values ​​of the logarithmic chromaticity space; then dividing the color values ​​of the logarithmic chromaticity space into histogram feature buckets to obtain the first histogram feature and the second histogram feature.

[0035] More specifically, this step includes: For the input reference RGB portrait image , first convert each channel color value of its RGB channels to the logarithmic color space: (2-1) in, For the input image of For the main channel coordinate, is the index of the current pixel, subscript for The three channels, is a small constant (usually set to ) is used to enhance numerical stability.

[0036] Logarithmic color space calculation values ​​of the remaining two channels The calculation method is similar to formula (2-1), that is: (2-2) (2-3) in, For the input image of For the main channel coordinate, For the input image of For the main channel coordinate, is the index of the current pixel, subscript for The three channels, is a small constant (usually set to ) is used to enhance numerical stability.

[0037] After calculating the logarithmic color space values ​​of all channel colors, the contribution value of each pixel is calculated based on the pixel intensity value using formula (2-4): (2-4) in is the index of the current pixel, Represents images respectively of The index under the three channels is , and then divide the color values ​​in the logarithmic space into histogram feature buckets, as shown in formula (2-5): (2-5) in is the normalized histogram feature, is the pixel index, Indicates the total number of pixels, To express contribution, Represents the contribution value of the pixel, Indicates channel The logarithmic space calculation value of , is an image The pixel coordinate value of is an inverse-quadratic kernel: (2-6) in, Is the attenuation parameter used to control the smoothness of the histogram feature bucketing.

[0038] Calculate the histogram features using formula (2-5) The process is differentiable and the kernel function of formula (2-6) makes the training process stable.

[0039] Finally Perform normalization: Following this process, we finally get two histogram features ,in Histogram features dimension.

[0040] Step S13: Based on the facial skin area of ​​the reference RGB portrait image, calculate and obtain a first color moment vector and a second color moment vector; the first color moment vector , is the first color moment vector The dimension of the second color moment vector , is the first color moment vector dimension.

[0041] Specifically, this step includes: based on the reference RGB portrait image, analyzing and obtaining a mask image that only contains the facial skin color area; after using the mask image to mask the reference RGB portrait image, calculating the first-order color moment and second-order color moment of the skin color area in the image; the first-order color moment refers to the color mean of the three channels in the image, and the second-order color moment refers to the color difference of the three channels in the image; using reparameterization processing, the first-order color moment and the second-order color moment are applied to a randomly sampled Gaussian distribution noise vector to obtain a first color moment vector and a second color moment vector.

[0042] More specifically, the following steps are included: The color moment calculation of this application refers to the second-order color moment of the facial skin area pixels of the RGB portrait. The calculation process is as follows: Use the portrait analysis method to obtain the mask image of the RGB portrait that only contains the facial skin color area, such as Figure 4As shown. After using this mask to mask the RGB portrait image, use formula (2-7) to calculate the first-order color moment of the skin color area of ​​the image, that is, the color mean of the three channels of the image: (2-7) in, The pixel value in the image is not The number of pixels, For the image The first pixel The color value of each channel, .

[0043] The second-order moment is the color standard deviation of the three channels: (2-8) Then, using reparameterization, the first-order color moment and the second-order color moment are applied to two randomly sampled Gaussian distribution noise vectors of different dimensions. , that is, the re-parameterized portrait skin color moment vector is obtained , the reparameterization process is shown in formulas (2-9) and (2-10): (2-9) (2-10) This color moment vector represents the skin color information of the reference image portrait. The color moment of the reference image portrait skin color will be used together with the histogram features as the constraint information to control the skin color transfer process. In order to enable the model to learn the skin color information of the reference image portrait, this application proposes a color moment loss based on formula (2-9): (2-11) in is the portrait skin color moment vector of the reference image, The image color moment vector after the portrait skin color migration output by ReMomentGAN, yes Norm, ensuring that the color moment characteristics output by the model are consistent with the skin color of the reference image portrait.

[0044] In a specific embodiment, step S20 includes: Step S20: Process the source RGB portrait image through an encoder-decoder network to obtain latent space features of the source image.

[0045] It should be noted that the encoder-decoder network is a U-Net network with dense skip connections. The input image will be mapped to the latent space of the output features of the StyleGAN generation block through this encoder-decoder network and a convolutional layer, and further input into the last two generation blocks of HistoGAN to control the semantic content of the generated image, such as the identity information of the portrait. Combined with the content input to these two generation blocks in the previous text, the migration of the input RGB portrait skin color to the reference portrait RGB skin color is completed.

[0046] Step S30: Map the histogram features, the color moment vector, and the latent space features of the source image to the latent space of the output features of the generation module through a convolutional layer, and output a target image with a skin color similar to that of the reference RGB portrait image; wherein, except for the skin color of the portrait, the other colors of the target image are also similar to those of the reference RGB portrait image.

[0047] In a specific embodiment, step S30 includes: Step S31: Pass the first feature of the histogram and the second feature of the histogram through a histogram projection network, a latent code mapping network, and a modulation-demodulation module respectively to obtain a weight value of the first feature of the histogram and a weight value of the second feature of the histogram.

[0048] Specifically, this step includes: The first histogram feature and the second histogram feature are respectively passed through the histogram projection network and the latent code mapping network in sequence to obtain a histogram style latent code first vector with a length of 32 and a histogram style latent code second vector with a length of 64; the first histogram style latent code vector and the second histogram style latent code vector are respectively input into two networks mod-demod with the same structure as the modulation-demodulation module of StyleGAN2 In the process, the weights in the two networks are modulated to obtain the weight value of the first feature of the histogram and the weight value of the second feature of the histogram.

[0049] More specifically, the above steps are to map the obtained histogram features to the style latent space in StyleGAN2 by passing them through a histogram projection network (eight fully connected layers plus a Leaky ReLU activation function group) and a latent code mapping network (two single linear layers with output dimensions of 32 and 64, respectively), obtaining histogram style latent code vectors of length 32 and 64, respectively. These are then input into two networks (mod-demod) with the same structure as the modulation-demodulation module of StyleGAN2, and the weights in these two networks are modulated to obtain the modulated weights of the two histogram features, namely the weight value of the first feature of the histogram and the weight value of the second feature of the histogram.

[0050] Step S32: modulate the three channels corresponding to the first vector of the color moment based on the weight value of the first feature of the histogram to obtain a first weight value of the portrait skin color modulation; modulate the three channels corresponding to the second vector of the color moment based on the weight value of the second feature of the histogram to obtain a second weight value of the portrait skin color modulation.

[0051] Specifically, the above steps include: using two three-channel color moment vectors calculated based on the portrait skin color area of ​​the reference image , respectively and Modulation on three channels (corresponding to Figure 2 ⊗ operator in the reference image), and obtain the weight modulated according to the skin color of the portrait in the reference image .

[0052] Step S33: After convolving the first weight value of the portrait skin color modulation and the second weight value of the portrait skin color modulation with the latent space features of the source image, they are input into the last two generation blocks of HistoGAN together with the noise vector to output a target image with a skin color similar to the reference RGB portrait image.

[0053] Specifically, the above steps include: the first weight value of the portrait skin color modulation and the second weight value of the portrait skin color modulation Convolving the latent space features of the first two convolutional layers of the encoder with the reference image histogram features And a noise vector is input into the last two generation blocks of HistoGAN. This process uses the histogram features of the reference image and the color style of the facial skin color as the control of the generated image style. The HistoGAN generation block is actually similar to the generation block of StyleGAN2. The HistoGAN structure is based on StyleGAN2, which is a pre-trained network in the work of ReHistoGAN to generate fake portrait images based on the color style of the reference image.

[0054] The skin color migration is compared using the RemomentGAN network described in this application, the ReHistoGAN network, the Deep preset network, and the Palette-based network. The comparison results are shown in the figure below. Figure 5 As shown, the effect will be more obvious on RGB color images, but it can be seen from the picture that the method described in this application has more advantages than other methods in the brightness channel. Figure 3 (c) and (a) are closer in brightness, but the other methods are darker.

[0055] Refer to the attached Figure 6Based on the same inventive concept as in the aforementioned embodiment, the present embodiment further provides a multi-view portrait skin color migration device, comprising: The extraction module is used to preprocess the reference RGB portrait image and calculate its histogram features and color moment vector; The encoding and decoding module is used to process the source RGB portrait image through an encoder-decoder network to obtain the latent space features of the source image; The output module is used to map the histogram features, the color moment vector and the latent space features of the source image to the latent space of the output features of the generation module through a convolution layer, and output a target image with a skin color similar to that of the reference RGB portrait image; wherein, except for the skin color of the portrait, the other colors of the target image are also similar to those of the reference RGB portrait image.

[0056] Those skilled in the art should understand that the division of the various modules in the embodiment is merely a division of logical functions, and in actual application, they can be fully or partially integrated into one or more actual carriers, and these modules can all be implemented in the form of software called through a processing unit, or all be implemented in the form of hardware, or be implemented in the form of a combination of software and hardware. It should be noted that the modules in the multi-view portrait skin color migration device in this embodiment correspond one-to-one to the steps in the multi-view portrait skin color migration method in the aforementioned embodiment. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned multi-view portrait skin color migration method, and will not be repeated here.

[0057] Based on the same inventive concept as in the aforementioned embodiment, an embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, a multi-perspective portrait skin color migration method as provided in the embodiment of the present application is implemented.

[0058] Based on the same inventive concept as in the above embodiment, an embodiment of the present application further provides an electronic device, including a processor and a memory, wherein: Memory is used to store computer programs; The processor is used to load and execute a computer program so that the electronic device can execute the multi-view portrait skin color migration method provided in the embodiment of the present application.

[0059] In some embodiments, the computer-readable storage medium may be a memory device such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface mount memory, optical disk, or CD-ROM; or various devices including any one or any combination of the above memories. The computer may be various computing devices including smart terminals and servers.

[0060] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0061] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0062] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0063] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0064] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0065] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a multimedia terminal device (which can be a mobile phone, a computer, a television receiver, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0066] In summary, the embodiment of the present application provides a multi-view portrait skin color migration method, which is to perform data preprocessing on the reference RGB portrait image to calculate the histogram features and color moment vectors; then process the source RGB portrait image through the encoder-decoder network to obtain the latent space features of the source image; finally, the histogram features, the color moment vectors and the latent space features of the source image are mapped to the latent space of the output features of the generation module through the convolution layer, and the target image with a skin color similar to that of the reference RGB portrait image is output. It can be seen that the method described in the present application is used to solve the problem of uneven texture color of the three-dimensional portrait model caused by skin color differences of multi-view portrait images in digital dental treatment or other special application scenarios, that is, by accurately migrating the skin color features of the reference image to the input image to eliminate the multi-view skin color differences, avoid artifacts and color distortion, and thus improve the color uniformity and realism of the three-dimensional model after texture fusion.

[0067] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

[0068] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A multi-view portrait skin color migration method, characterized in that: The following steps are involved: Preprocess the reference RGB portrait image and calculate its histogram features and color moment vector; Processing the source RGB portrait image through an encoder-decoder network to obtain the latent space features of the source image; The histogram features, the color moment vectors, and the latent space features of the source image are mapped to the latent space of the output features of the generation module through a convolutional layer, and a target image with a skin color similar to that of the reference RGB portrait image is output; wherein, except for the skin color of the portrait, all other colors of the target image are similar to those of the reference RGB portrait image.

2. The multi-view portrait skin color migration method according to claim 1, characterized in that: The reference RGB portrait image is preprocessed to calculate its histogram features and Color moment vector, including: The reference RGB portrait image is used for data preprocessing; Calculate and obtain the first feature of the histogram and the second feature of the histogram; the first feature of the histogram , Represents histogram features The dimension of the second feature of the histogram , Represents histogram features Dimensions; Based on the facial skin area of ​​the reference RGB portrait image, a first color moment vector and a second color moment vector are calculated; the first color moment vector , Represents the first vector of color moment The dimension of the second color moment vector , Represents the second vector of color moment dimension.

3. The multi-view portrait skin color migration method according to claim 2, characterized in that: The calculation to obtain the first feature of the histogram and the second feature of the histogram includes: Converting the color value of each channel of the RGB channel of the reference RGB portrait image into a logarithmic color space to obtain a color value in the logarithmic color space; Then, the color values ​​in the logarithmic chromaticity space are divided into histogram feature buckets to obtain a first histogram feature and a second histogram feature.

4. The multi-view portrait skin color migration method according to claim 2, characterized in that: The step of calculating the first color moment vector and the second color moment vector based on the facial skin area of ​​the reference RGB portrait image includes: Based on the reference RGB portrait image, a mask image containing only the facial skin color area is obtained through analysis; After masking the reference RGB portrait image with the mask image, the first-order color moment and the second-order color moment of the skin color area in the image are calculated; the first-order color moment refers to the color mean of the three channels in the image, and the second-order color moment refers to the color difference of the three channels in the image; Reparameterization processing is adopted to apply the first-order color moment and the second-order color moment to a randomly sampled Gaussian distribution noise vector to obtain a first color moment vector and a second color moment vector.

5. The multi-view portrait skin color migration method according to claim 2, characterized in that: The step of mapping the histogram features, the color moment vector, and the latent space features of the source image to the latent space of the output features of the generation module through a convolutional layer, and outputting a target image with a skin color similar to that of the reference RGB portrait image, includes: Passing the first feature of the histogram and the second feature of the histogram through a histogram projection network, a latent code mapping network, and a modulation-demodulation module respectively, to obtain a weight value of the first feature of the histogram and a weight value of the second feature of the histogram; Modulating the three channels corresponding to the first color moment vector based on the weight value of the first feature of the histogram to obtain a first weight value for the portrait skin color modulation; modulating the three channels corresponding to the second color moment vector based on the weight value of the second feature of the histogram to obtain a second weight value for the portrait skin color modulation; The first weight value of the portrait skin color modulation and the second weight value of the portrait skin color modulation are convolved with the latent space features of the source image respectively, and then input into the last two generation blocks of HistoGAN with the noise vector to output a target image with a skin color similar to the reference RGB portrait image.

6. The multi-view portrait skin color migration method according to claim 5, characterized in that: The step of sequentially passing the first feature of the histogram and the second feature of the histogram through a histogram projection network, a latent code mapping network, and a modulation-demodulation module to obtain a weight value of the first feature of the histogram and a weight value of the second feature of the histogram includes: Passing the first histogram feature and the second histogram feature through a histogram projection network and a latent code mapping network, respectively, to obtain a first histogram style latent code vector with a length of 32 and a second histogram style latent code vector with a length of 64; The first histogram style latent code vector and the second histogram style latent code vector are respectively input into two networks mod-demod with the same structure as the modulation-demodulation module of StyleGAN2 In the process, the weights in the two networks are modulated to obtain the weight value of the first feature of the histogram and the weight value of the second feature of the histogram.

7. The multi-view portrait skin color migration method according to claim 1, characterized in that: The pre-trained ReHistoGAN model weights are used as the initial weights of the ReMomentGAN network to reuse the prior knowledge in its model. The pre-trained ReHistoGAN model weights are publicly available through open source projects. The ReMomentGAN network is fine-tuned unsupervisedly, and the unsupervised fine-tuning satisfies the following relationship: in, represents the loss function used to fine-tune the HistoGAN discriminator; Represents the discriminator loss term in HistoGAN; Represents the gradient penalty term in the WGAN-GP model; represents the loss function used to fine-tune all modules of the ReMomentGAN network; express The weight of the loss term, ; Expressed as color moment loss term; Expressed as The weight of the loss term, ; ( ) represents the histogram loss term proposed in ReHistoGAN; Represents the histogram features of the target image output by ReMomentGAN ; Represented as the histogram feature of the reference RGB portrait image ; Expressed as The weight of the loss term, ; ( ) represents the image reconstruction loss term proposed by ReHistoGAN; Represented as source RGB portrait image; represents the target image; Expressed as The weight of the loss term, ; Represents the discriminant output value of the source RGB portrait image; Expressed as The weight of the loss term, ; ( ) represents a loss term proposed by ReHistoGAN to prevent the model from only applying global color bias.

8. A multi-view portrait skin color migration device, characterized in that: include: The extraction module is used to preprocess the reference RGB portrait image and calculate its histogram features and color moment vector; The encoding and decoding module is used to process the source RGB portrait image through an encoder-decoder network to obtain the latent space features of the source image; The output module is used to map the histogram features, the color moment vector and the latent space features of the source image to the latent space of the output features of the generation module through a convolution layer, and output a target image with a skin color similar to that of the reference RGB portrait image; wherein, except for the skin color of the portrait, the other colors of the target image are also similar to those of the reference RGB portrait image.

9. A device, characterized in that include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in a memory and configured to be executed by one or more processors, the one or more programs being configured to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images