A method and system for color grading of portrait photographs based on deep neural networks

Abstract

This application relates to a method and system for color stylizing portrait photographs based on deep neural networks. It utilizes a feature cross-attention network to perform feature cross-attention encoding on the original and reference images, enabling efficient feature encoding for high-resolution portrait photographs. Simultaneously, it leverages semantic segmentation maps to address fine-grained style mapping issues, allowing the model to better handle fine-grained semantic elements such as skin tone and clothing. After aligning the semantic features of the original and reference images using the cross-attention network, it fuses the global information features output from the global information feature extraction network and the semantic cross-information features output from the style feature cross-attention network to output a stylized image. This ensures consistency with the reference image across overall image factors such as color temperature, hue, exposure, and contrast, as well as at the level of individual fine-grained semantic elements, thereby achieving high-efficiency and high-quality portrait stylization.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a method, system, computer device, and computer-readable storage medium for color grading of portrait photographs based on deep neural networks. Background Technology

[0002] Portrait color grading is a post-processing technique that mimics the color style of other works and applies their color characteristics to the portrait photograph being processed. It can help photographers quickly achieve a specific, popular color atmosphere or maintain a harmonious and unified color style within a group of photos.

[0003] In related technologies, with the rise of generative AI technologies represented by Stable Diffusion, stylized color processing of portrait photographs can be performed using techniques such as Lora (Low-Rank Adaptation) and ControlNet. However, these methods have the following drawbacks: 1) high computational requirements and low inference efficiency; 2) limited output size, requiring the use of super-resolution and other technologies to obtain higher resolution images; 3) difficulty in guaranteeing the original image texture and other issues. Summary of the Invention

[0004] This application provides a method, system, computer device, and computer-readable storage medium for color grading of portrait photographs based on deep neural networks, in order to at least solve the problem in the related art of difficulty in guaranteeing image texture when stylizing portrait photographs.

[0005] In a first aspect, embodiments of this application provide a method for training a color model for portrait photography images based on a deep neural network, wherein any training step in the method includes:

[0006] Receive a stylized training set constructed based on portrait photographic images, wherein the stylized training set includes: an original image, a reference image, and an artificially stylized image corresponding to the original image, as well as semantic segmentation maps of the original image and the reference image;

[0007] Extract image features and global features from the stylized training set;

[0008] By constructing a feature cross-attention network based on the Transformer network, self-attention encoding and cross-attention encoding are performed based on the image features of the original image and the image features of the reference image to obtain style cross features;

[0009] The global features and the style cross features are fused through a style mapping network to output a stylized image.

[0010] In some embodiments, the method further includes:

[0011] Based on the stylized training set, the Adam optimizer is used to iteratively run the training steps and optimize the model parameters. When the model converges, the trained image color model is obtained.

[0012] In some embodiments, extracting image features and global features from the stylized training set includes:

[0013] The image features are obtained by encoding each original image and each reference image using an image feature encoding network with the semantic segmentation map as prior information.

[0014] The global features are obtained by extracting global information from each original image and each reference image through a global feature extraction network.

[0015] In some embodiments, a feature cross-attention network built on a Transformer network is used to perform self-attention encoding and cross-attention encoding based on the image features of the original image and the image features of the reference image, to obtain style cross features including:

[0016] Self-attention encoding is performed on the image features of the original image to obtain the self-attention encoding result;

[0017] The self-attention encoding result and the image features of the reference image are input into the cross-attention module for cross-attention encoding, and the style cross features are output.

[0018] In some embodiments, the style mapping network is built on a Unet architecture. The style mapping network fuses the global features and the style cross-features to output the stylized image, including:

[0019] The original image is encoded and decoded at different levels to obtain the feature representation of the original image;

[0020] The global features and the style cross features are fused, and the fusion result is integrated with the original image feature representation to obtain the stylized image.

[0021] In some embodiments, the following loss function is used during the training of the image color model:

[0022]

[0023] in, The model input is the original image. The output stylized image, A stylized image representing a realistic, manually processed image. Style reference image These represent the weights of each loss, specifically set as follows: , Indicates batch size. Describing the 2-norm, Represents the 1-norm, in the perceptual loss of a human face-weighted image. , This represents the output of the j-th layer of a Vgg16 network pre-trained on ImageNet. Indicates weight, specifically set as follows: , , , It is the original image. The corresponding single-channel dimension is In the image weighting map, regions identified as human figures in the semantic segmentation map are assigned a value of 10, while other regions are assigned a value of 1. This weighting plays a role in the perceptual loss calculation of human figure regions. In style loss... This represents the average pixel value of the image. This represents the standard deviation of the image pixels.

[0024] Secondly, embodiments of this application provide a method for color grading of portrait photographs based on deep neural networks, the method comprising:

[0025] Receive the image to be processed and the reference image;

[0026] Based on the image to be processed and the reference image, the color model trained by the method provided in the first aspect is used to stylize the image to be processed, thereby obtaining a stylized image corresponding to the image to be processed.

[0027] Thirdly, embodiments of this application provide a portrait photography image color grading system based on a deep neural network. The system includes an acquisition module and a stylization processing module, wherein:

[0028] The acquisition module is used to receive the image to be processed and the reference image;

[0029] The stylization processing module is used to perform stylization processing on the image to be processed based on the image to be processed and the reference image, using a color model trained in the first aspect, to obtain a stylized image corresponding to the image to be processed.

[0030] Fourthly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the methods described in the first and second aspects above.

[0031] Fifthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in the first and second aspects above.

[0032] Compared to related technologies, style transfer for portrait photography suffers from poor inference efficiency and difficulty in guaranteeing image texture. This application's solution utilizes a feature cross-attention network built on a Transformer network to perform feature cross-attention encoding on the original and reference images. For high-resolution portrait images, this achieves efficient feature encoding. Simultaneously, semantic segmentation maps are used to assist in solving the fine-grained element style mapping problem, enabling the model to better handle fine-grained semantic elements such as skin tone and clothing. Furthermore, by combining the cross-attention network to align the semantic features of the original and reference images, the original image, the global information features output from the global information feature extraction network, and the semantic cross-information features output from the style feature cross-attention network are fused to output a stylized image. This ensures consistency with the reference image in overall image factors such as color temperature, hue, exposure, and contrast, as well as at the level of each fine-grained semantic element, thereby achieving high-efficiency and high-quality portrait stylization processing. Attached Figure Description

[0033] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0034] Figure 1 This is a flowchart of a method for training a portrait photography image color model based on a deep learning network, according to an embodiment of this application.

[0035] Figure 2 This is a schematic diagram of an image feature coding network according to an embodiment of this application;

[0036] Figure 3 This is a schematic diagram of a global feature extraction network according to an embodiment of this application;

[0037] Figure 4 This is a schematic diagram of a feature cross-attention network according to an embodiment of this application;

[0038] Figure 5This is a network structure diagram of a portrait photography digital image stylization method according to an embodiment of this application;

[0039] Figure 6 This is a schematic diagram illustrating the processing effect of a solid-color indoor scene image in a wedding photography setting according to the solution of this application.

[0040] Figure 7 This is a schematic diagram illustrating the image processing effect of outdoor greenery in a wedding photography scene according to the solution in this application.

[0041] Figure 8 This is a schematic diagram illustrating the effectiveness of the style feature cross-module constructed based on the Transformer network in this application.

[0042] Figure 9 This is a structural block diagram of a portrait photography image color matching system based on a deep neural network, according to an embodiment of this application;

[0043] Figure 10 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0045] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0046] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0047] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0048] In the portrait photography industry, such as wedding photography, children's photography, and portrait photography, the original photos are manually retouched by professional retouchers before delivery. One crucial step in this process is color correction. Depending on the quality of the original photos, this process can take tens of minutes or even several hours to complete. Experienced retouchers can make informed judgments about different original photos and effectively adjust the colors and style. However, less experienced retouchers need to find similar sample photos as references and imitate them to achieve the desired color correction. This process is known in the field as "color grading."

[0049] In recent years, with the rise of generative AI technologies represented by Stable Diffusion, diffusion techniques can be used to stylize images, such as Lora and ControlNet. However, these methods have problems such as: 1) high computational requirements and low inference efficiency; 2) limited output size, requiring super-resolution and other technologies to obtain higher resolution images; and 3) difficulty in guaranteeing the original image texture.

[0050] However, portrait photography is a special real-world photographic scenario. 1) In excellent portrait photographs, the visual effect of the portrait is often prominent enough, so the colors of the foreground portrait, such as skin tone and clothing, are more important than the colors of the background. 2) Compared to general image stylization scenarios, the stylization of portrait images has a more concrete semantic correspondence. For example, for a set of seascape portrait photographs, the ideal result is to achieve consistency between the final image and the reference image, from coarse-grained factors such as image tone, brightness, and contrast, to fine-grained key elements such as skin tone, seawater, sky, and greenery. 3) Image resolution often reaches 3K or higher, requiring the model to have high inference efficiency.

[0051] Existing general color matching methods, because they do not take into account the above characteristics, generally suffer from low efficiency and poor texture processing effects when stylizing portrait photographs.

[0052] In view of this, this application designs a deep neural network-based color matching method for image stylization scenarios in portrait photography. Given a reference image, it can efficiently project the style and color of the reference image onto the original portrait image, and output a stylized result that highlights the key features of the subject and maintains consistency from coarse to fine granular features. This helps retouchers improve efficiency or reduce labor costs. This "one-click" intelligent color matching method can be directly used by users, meeting the daily shooting, retouching, and color grading needs of the general public. Specifically, Figure 1 This is a flowchart illustrating a training method for a portrait photography image color model based on a deep learning network, according to an embodiment of this application. Figure 1 As shown, the process includes the following steps:

[0053] S101, receive a stylized training set constructed based on portrait photographic images, including: original image, reference image and stylized image corresponding to the original image, and semantic segmentation map of original image and reference image;

[0054] The stylized training set is constructed through the following steps:

[0055] S1 collects multiple original images, reference images, and stylized images from different portrait photography scenarios such as wedding photography, personal portraits, and family portraits;

[0056] Specifically, these images should cover as many portrait photography scenarios as possible, including but not limited to: seascapes, greenery, street photography, indoor life, studio photography, Chinese style, and other different shooting scenarios and styles of portrait photography.

[0057] Additionally, it should be noted that the original image is the image to be processed, and the reference image is used to provide reference information for stylizing the original image. Correspondingly, the stylized image is the result of stylizing the original image based on the reference image. The stylized image can be obtained manually by a professional retoucher using Photoshop tools through color dithering.

[0058] S2, using a semantic segmentation model, performs semantic segmentation on the original image and the reference image respectively, to obtain the corresponding semantic segmentation map.

[0059] The allegorical style model used in this embodiment can be obtained by custom training based on the HRNet network structure. Optionally, the custom semantic segmentation and recognition targets include, but are not limited to, elements commonly found in various portrait photography scenes such as people, animals, trees, seawater, sky, and ground.

[0060] It is understandable that since the stylization of portrait photography involves fine-grained element mapping, extracting semantic segmentation maps and integrating them into the stylization training set to participate in model training can provide semantic-level prior support for the model's feature extraction process, thereby helping to extract specific features and assisting in solving the problem of style mapping corresponding to fine-grained elements in the salient features of portrait photography.

[0061] S3, constructing a stylized training set;

[0062] The stylization training set should include at least 50,000 sample pairs of different scenes and styles, with each sample pair containing the original image. Style reference images Stylized images processed manually and the semantic segmentation map corresponding to the original image. Semantic segmentation map corresponding to style reference image .

[0063] In one embodiment, the portrait photography image stylization training dataset can be represented as:

[0064]

[0065] in These are the original image set, the style reference image set, and the stylized result image set, respectively. The size of the original image is defined as the number of elements in the set. The style reference image is [size missing]. The stylized result image has the same size as the original image. In this embodiment, , .

[0066] S102, extract image features and global features from the stylized training set;

[0067] This step is implemented through an image feature encoding network and a global information feature extraction network. Specifically, since style transfer in portrait photography requires consideration of both coarse-grained and fine-grained feature transfer, this embodiment provides the above two feature extraction networks, which are used to effectively extract features related to local details (such as skin color and clothing texture) and global attributes (such as color temperature and hue), respectively, in order to better achieve style transfer.

[0068] Specifically, this step includes the following sub-steps:

[0069] S1, using an image feature encoding network with semantic segmentation maps as prior information, encodes each image in the original image set and the reference image set to obtain image features;

[0070] in, Figure 2 This is a schematic diagram of an image feature coding network according to an embodiment of this application, such as... Figure 2 As shown, in this embodiment, the image feature encoding network is a lightweight convolutional neural network, consisting of seven convolutional layers and a ReLU activation function;

[0071] In one embodiment, the image feature encoding network receives data in dimension 1. The input, which is provided by RGB images and The semantically segmented images are stitched together, and the network output is... The feature map is given by B, where B represents the batch size for model training.

[0072] It's understandable that both the original image and the style reference image are first encoded by this network to obtain image features. This network is mainly used for preliminary encoding of the image, extracting comprehensive features at both the local and semantic levels. These feature maps are then used for more refined local style transfer, such as style changes related to semantic segmentation, like facial details and clothing details.

[0073] S2 uses a global feature extraction network to extract global information from each image in the original image set and the reference image set to obtain global features.

[0074] Specifically, Figure 3 This is a schematic diagram of a global feature extraction network according to an embodiment of this application, such as... Figure 3As shown, the global information feature extraction network in this embodiment is built based on AlexNet, a pioneering convolutional neural network in the field of deep learning that performs exceptionally well in tasks such as image classification. Its network structure includes multiple convolutional layers, pooling layers, and fully connected layers. Through a series of convolutional operations, pooling downsampling, and feature integration by fully connected layers, it can effectively extract image features.

[0075] The global feature extraction network in this embodiment is built on AlexNet, drawing on its mature feature extraction architecture and training methods. Leveraging its advantages in processing image data, it can efficiently and accurately extract global features from portrait photographs.

[0076] Specifically, the input to this network is The RGB image, the network output is The network extracts global feature information from the image. Similarly, both the original image and the style reference image are processed by this network to extract global feature information. The purpose of this network is to extract global aspects of the image, such as color temperature, hue, exposure, and contrast.

[0077] It is understandable that the core function of this network is to extract global information such as color temperature, hue, exposure, and contrast from an image. By extracting these global features, it can capture the overall style and visual characteristics of the image. For a reference image, its global feature information can be used as part of the description of the target style. For the original image, the extracted global features help subsequent network modules to better integrate the style of the style reference image into the original image during style transfer, so that the generated stylized image is consistent with the style reference image in overall visual effect, while retaining some content information of the original image.

[0078] S103 uses a feature cross-attention network built on the Transformer network to perform self-attention encoding and cross-attention encoding based on the image features of the original image and the image features of the reference image to obtain style cross features;

[0079] Figure 4 This is a schematic diagram of a feature cross-attention network according to an embodiment of this application, such as... Figure 4 As shown, the input of the Transformer network is the image features output by the image feature network after encoding the original image and the style reference image. The original image features are first encoded by self-attention, and then cross-attention is performed with the style reference image features to output the style cross features. The query vector Q in the cross-attention encoding process is the original image feature, and the key vector K and value vector V are the style reference image features.

[0080] Specifically, the self-attention encoding process includes the following steps:

[0081] S1, Feature Input and Linear Transformation: First, the original image features output from the image feature encoding network are received. Then, the original image features are linearly transformed using different matrices, mapping them to different spaces and helping the model capture the relationships between features from different perspectives. After the transformation, query vector, key vector, and value vector are obtained. These three vectors are like a re-expression of the original features from different viewpoints.

[0082] S2, Calculate the attention score: Calculate the similarity between the query vector and the key vector to obtain the attention score. That is, calculate the similarity between the query vector at each location and the key vectors at all other locations. The results of this similarity calculation form a matrix. This matrix represents the degree of association between each location in the image and all other locations, like a "relationship map" showing the close connection between different parts of the image.

[0083] S3, Normalized Attention Score: The attention scores obtained in S2 are processed using the Softmax function, so that the sum of these scores equals 1, resulting in the normalized attention weights. These weights reflect the importance of each location relative to others. For example, in a portrait photograph, a high weight for the eyes means that the model will pay more attention to eye-related information when processing that location.

[0084] S4, Calculate the self-attention output: The normalized attention weights and value vectors are weighted and summed to obtain the output of the self-attention encoding.

[0085] This output takes into account the correlation between various positions in the original image and the reference image, highlights the feature information that is highly related to the current position, and combines closely related parts to form a more meaningful whole.

[0086] Through steps S1-S4 above, self-attention encoding first calculates the correlation between different parts of the original image features, generating attention weights, which are then used to reweight the features. This highlights key information in the original image, allowing the model to focus on key points, enhancing feature expressiveness, and providing crucial foundational information for subsequent cross-attention encoding with style reference image features, thereby improving the accuracy of the subsequent style transfer process.

[0087] Furthermore, the cross-attention encoding process includes the following steps:

[0088] S1, the features of the original image after self-attention encoding are used as the query vector, and the features of the style reference image after being encoded by the image feature network are used as the key vector and value vector, respectively.

[0089] S2, Calculate the degree of association: Calculate the similarity between the query vector and the key vector, compare each positional feature of the original image with all positional features of the style reference image, and obtain a matrix. The values ​​in the matrix represent the degree of association between each position in the original image and each position in the style reference image.

[0090] S3, Determine Importance: The Softmax function is used to process the similarity results above, summing them to 1 to obtain the attention weights. This step clarifies the importance of information from different locations in the style reference image to various locations in the original image, and is a crucial step in achieving style fusion.

[0091] S4, Feature Fusion: The attention weights and value vectors are weighted and summed to incorporate information closely related to the original image from the style reference image, generating a cross-attention output, which provides an important basis for the subsequent generation of stylized images.

[0092] It should be noted that Transformer is a computationally intensive network structure. The computational performance of Transformer networks increases geometrically with the increase of image feature size, thereby enabling the use of high-resolution portrait images to meet computational requirements and improve inference efficiency.

[0093] Furthermore, in order to seek a balance between model inference performance and stylization effect, it is necessary to find a relatively reasonable feature cross scale. Based on the aforementioned embodiments, the stylization effect of portrait photography images is affected by two aspects: coarse-grained (overall level of color temperature, hue, exposure, contrast, etc.) and fine-grained (semantic level of portrait skin color, clothing, background elements, etc.). The inventors of this case reasonably assume that the target feature map scale for feature cross operation only needs to be able to clearly distinguish semantics. A larger scale increases the amount of computation but does not bring about an improvement in effect, while a smaller size reduces the amount of computation but will lead to a blurred mapping between the semantic relationship between the original image and the style reference image.

[0094] Furthermore, to verify the above hypothesis, the inventors performed feature crossing at scales of 1 / 4, 1 / 8, 1 / 16, 1 / 32, and 1 / 64. The final experimental results showed that the stylization effects obtained based on the cross features at the 1 / 4, 1 / 8, and 1 / 16 scales were almost identical. However, when the scale was further compressed to 1 / 32 and 1 / 64, the mapping relationship between the skin tone and background color in the final stylization effect gradually weakened. This conclusion is consistent with the aforementioned hypothesis to some extent. Therefore, in this embodiment, the Transformer attention feature crossing calculation is preferentially performed at the 1 / 16 scale to maintain a balance between inference performance and stylization effect to the greatest extent.

[0095] It is understandable that, through step S103 above, self-attention encoding first uncovers the connections between different parts of the original image, highlighting key features and enhancing feature expressiveness. Building on this, cross-attention encoding further integrates style reference image information, ensuring that the features output by the model retain both the content of the original image and the stylistic characteristics of the style reference image. The synergistic effect of these two methods allows the model to accurately grasp the features of both images during the stylization of portrait photography, thereby aligning fine-grained semantic features of the images.

[0096] S104, through a style mapping network, fuses global features and style cross-features to output a stylized image.

[0097] Specifically, the style mapping network in this embodiment is built based on the Unet network structure. The network takes an original image with dimensions (B,3,w_x,h_x) as input. In addition to encoding and decoding the original image, more importantly, it integrates the global features and style cross features output by the global information feature extraction network and the style feature cross attention network. The network finally outputs the original image with dimensions (B,3,w_x,h_x) after reference image stylization.

[0098] in addition, Figure 5 This is a network structure diagram of a portrait photography digital image stylization method according to an embodiment of this application, such as... Figure 5 As shown, the semantic segmentation model processes the original image and style reference image as input, generating semantic segmentation maps to provide semantic priors for subsequent processing. Next, the image feature encoding network concatenates the RGB image with the semantic segmentation image and encodes it, outputting a feature map that fuses color and semantic information. Simultaneously, the global information feature extraction network extracts global features such as color temperature from the RGB data of the original and reference images. Then, the style feature cross-attention network performs self-attention and cross-attention encoding on the encoded features, calculating at a 1 / 16 scale to achieve semantic-level feature alignment. Finally, the style mapping network, built based on UNet, takes the original image as input and fuses global and style cross-features, outputting a stylized result image after encoding and decoding, thus projecting the style of the reference image onto the original image, completing the stylization task.

[0099] The above steps S101 to S103 describe a complete operation process of the model. It can be understood that in order to obtain a model that can be practically deployed, it is necessary to use a large amount of data, repeatedly execute the above process, and continuously adjust the model for multiple iterations of training to obtain a trained model.

[0100] Specifically, in this embodiment, the model training process is based on PyTorch 1.12, the optimizer is Adam, the learning rate is set to 1e-4, the model is first trained for 400 epochs on a dataset of 10,000 pairs of 512*512 size, with a batch size of 256, and then trained for 300 epochs on a dataset of 50,000 pairs of 1024*1204 size, with a batch size of 64.

[0101] The learning rate of 1e - 4 was determined after balancing training speed and convergence stability, so that the model parameters can be updated in the direction of decreasing loss function value with an appropriate step size. An appropriate learning rate can enable the model to converge to a better combination of parameters in a reasonable time, thereby improving model performance.

[0102] In addition, to improve data diversity and reduce overfitting, data augmentation can be performed on the original image. Specific methods include, but are not limited to, random horizontal flipping, random brightness adjustment, and random contrast adjustment. The style reference image only uses random horizontal flipping data augmentation.

[0103] Furthermore, the loss function for training the portrait photography image stylization model consists of three parts: mean squared error loss, portrait-weighted perceptual loss, and style loss. The calculation formula is shown in the following equation.

[0104]

[0105] in, The model input is the original image. The output stylized image, This represents a stylized image that has undergone genuine human processing. Style reference image These represent the weights of each loss, specifically set as follows: , Indicates batch size. Describing the 2-norm, Represents the 1-norm, in the perceptual loss of a human face-weighted image. , This represents the output of the j-th layer of a Vgg16 network pre-trained on ImageNet. Indicates weight, specifically set as follows: , , , It is the original image. The corresponding single-channel dimension is In the image weighting map, regions identified as human figures in the semantic segmentation map are assigned a value of 10, while other regions are assigned a value of 1. This weighting plays a role in the perceptual loss calculation of human figure regions. In style loss... This represents the average pixel value of the image. This represents the standard deviation of the image pixels.

[0106] The stylized mean squared error loss (Content Loss) measures the difference between the stylized content and the original content, ensuring that the stylized image does not lose key information from the original. Perceptual loss is used to weight the loss calculation for the portrait region. Furthermore, style loss measures the style difference between the stylized image and the style reference image. When calculating style loss, attention is paid to style-related features such as texture and color distribution. By minimizing style loss, the style features (such as hue, color temperature, and contrast) of the stylized image can be made as close as possible to the style reference image. This helps to accurately transfer the style of the style reference image to the original image, giving the generated stylized image the desired style characteristics and improving the quality of style transfer.

[0107] Finally, the overall loss function in this application is a weighted sum of mean squared error loss, perceptual loss, and style loss. By adjusting the weight coefficients, the relative importance of content and style in the stylization process can be flexibly controlled. This allows for the generation of stylized images that retain both the original image content and the desired style, based on specific application scenarios and requirements, achieving a good balance between style transfer and content preservation.

[0108] After obtaining the trained model, a method for color grading of portrait photographs based on deep neural networks is also provided, which includes the following steps:

[0109] S201, Receive the image to be processed and the reference image;

[0110] S202, Based on the image to be processed and the reference image, the color model trained through the above steps is used to stylize the image to be processed, resulting in a stylized image corresponding to the image to be processed.

[0111] Figure 6 This is a schematic diagram illustrating the processing effect of a solid-color indoor scene image in a wedding photography setting according to the solution of this application. Figure 7 This is a schematic diagram illustrating the image processing effect of outdoor greenery in a wedding photography scene according to the solution proposed in this application. For ease of demonstration, the original image has been compressed to a size of 512*512. Figure 6 and Figure 7From left to right, the images are: the original image, the style reference image, the stylization result output by this method, the semantic segmentation visualization output by the self-trained model corresponding to the original image, and the semantic segmentation visualization output by the self-trained model corresponding to the style reference image. Figure 6 and Figure 7 As can be seen from the results, the results obtained by this application have a similar style to the reference image, and the "color imitation" operation is completed very well.

[0112] Figure 8 This is a schematic diagram illustrating the effectiveness of the style feature cross-module constructed based on the Transformer network in this application, such as... Figure 8 As shown, in this example, a warmer skin tone is a prominent feature of the style reference image. The output of this application's solution accurately reproduces the skin tone characteristics, verifying the effectiveness of the module in semantic feature alignment.

[0113] In summary, it can be seen that the portrait photography stylization results obtained by the method of this application maintain a consistent level with the style reference image not only in terms of overall image factors such as color temperature, hue, exposure, and contrast, but also in terms of semantic elements of the portrait and background.

[0114] Through the above steps, existing techniques for style transfer of portrait photographs suffer from poor inference efficiency and difficulty in guaranteeing image texture. This application's solution utilizes a feature cross-attention network built on a Transformer network to perform feature cross-attention encoding of the original and reference images. This enables efficient feature encoding even for high-resolution portrait photographs. Furthermore, semantic segmentation maps are used to assist in solving the fine-grained element style mapping problem, allowing the model to better handle elements such as skin tone and clothing. Further, a cross-attention network is combined to align the semantic features of the original and reference images. Finally, the original image, the global information features output from the global information feature extraction network, and the semantic cross-information features output from the style feature cross-attention network are fused. This ensures consistency with the reference image in overall image factors such as color temperature, hue, exposure, and contrast, as well as in each fine-grained semantic element, thereby achieving high-efficiency and high-quality portrait stylization processing.

[0115] This embodiment also provides a portrait photography image color grading system based on deep neural networks. Figure 9 This is a structural block diagram of a portrait photography image color correction system based on a deep neural network, according to an embodiment of this application. Figure 9 As shown, the system includes: an acquisition module 90 and a stylization processing module 91, wherein:

[0116] The acquisition module 90 is used to receive the image to be processed and the reference image;

[0117] The stylization processing module 91 is used to stylize the image to be processed based on the image to be processed and the reference image, using the color model trained through the above steps S101 to S104, to obtain a stylized image corresponding to the image to be processed.

[0118] This system utilizes a feature cross-attention network built upon the Transformer network to perform feature cross-attention encoding on the original and reference images. This enables efficient feature encoding even for high-resolution portrait photographs. Furthermore, it leverages semantic segmentation maps to address fine-grained element style mapping, allowing the model to better handle elements such as skin tone and clothing. The cross-attention network is further integrated to align semantic features between the original and reference images. Finally, the system fuses the original image, global information features output from the global information feature extraction network, and semantic cross-information features output from the style feature cross-attention network. This ensures consistency with the reference image across overall image factors such as color temperature, hue, exposure, and contrast, as well as various fine-grained semantic elements, thereby achieving highly efficient and high-quality portrait stylization.

[0119] In one embodiment, Figure 10 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 10 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 10 As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores an operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network, the internal memory provides an environment for the operating system, the computer programs are executed by the processor to implement a deep neural network-based color grading method for portrait photography images, and the database stores data.

[0120] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. Specifically, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0121] Furthermore, in conjunction with the deep neural network-based portrait image color grading method described in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the deep neural network-based portrait image color grading methods described in the above embodiments.

[0122] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the deep neural network-based portrait photography image color grading methods described above.

[0123] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0124] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for training a color model for portrait photography images based on deep neural networks, characterized in that, Any training step in the method includes: The system receives high-resolution portrait photographs with prominent visual effects of human elements, concrete semantic correspondence, and constructs a stylization training set based on the portrait photographs. The stylization training set includes: an original image, a reference image, and an artificially stylized image corresponding to the original image, as well as semantic segmentation maps of the original image and the reference image. The reference image is used to provide reference information for stylizing the original image, and the artificially stylized image is the result of stylizing the original image based on the reference image. Image features and global features are extracted from the stylized training set. The image features are obtained by encoding based on the concatenation result of the original image and its corresponding semantic segmentation map, or the concatenation result of the reference image and its corresponding semantic segmentation map, through an image feature encoding network. By constructing a feature cross-attention network based on the Transformer network, self-attention encoding and cross-attention encoding are performed based on the image features of the original image and the image features of the reference image to obtain aligned fine-grained image semantic level style cross features, including: performing self-attention encoding on the image features of the original image to obtain the self-attention encoding result; The self-attention encoding result and the image features of the reference image are input into the cross-attention module for cross-attention encoding, and the style cross features are output; wherein, the self-attention encoding process includes the following steps: The system receives the original image features output by the image feature encoding network, performs linear transformations on the original image features through different matrices, captures the relationships between features from different perspectives, and obtains query vectors, key vectors, and value vectors that re-express the original features from different perspectives. Calculate the similarity between the query vector and the key vector to obtain an attention score. Use the Softmax function to process the attention score to obtain a normalized attention weight that reflects the importance of each position relative to other positions. The normalized attention weights and value vectors are weighted and summed to obtain the self-attention encoding result that comprehensively considers the correlation between various positions in the original image and the reference image; wherein, the cross-attention encoding uses the features of the original image after self-attention encoding as the query vector, and the features of the style reference image after being encoded by the image feature network as the key vector and value vector, respectively; A style mapping network based on the Unet architecture projects the style of the reference image onto the original image based on the global features of the original image, the reference image, and the style cross features, outputting a stylized image corresponding to the portrait photograph image. Specifically, this includes: The original image is encoded and decoded at different levels to obtain the feature representation of the original image; The global features and the style cross features are fused, and the fusion result is integrated with the original image feature representation to obtain the stylized image.

2. The method according to claim 1, characterized in that, The method further includes: Based on the stylized training set, the Adam optimizer is used to iteratively run the training steps and optimize the model parameters. When the model converges, the trained image color model is obtained.

3. The method according to claim 2, characterized in that, Extracting image features and global features from the stylized training set includes: The image features are obtained by encoding each original image and each reference image using an image feature encoding network with the semantic segmentation map as prior information. The global features are obtained by extracting global information from each original image and each reference image through a global feature extraction network.

4. The method according to claim 2, characterized in that, The loss function used in training the image color model is determined by a weighted combination of mean squared error loss, portrait-weighted perceptual loss, and style loss, based on their respective weight coefficients, where: The mean squared error loss is used to measure the difference between the stylized content of the original image and the original content, ensuring that the stylized image retains the key content information of the original image; The perceptual loss is used to weight the loss calculation for the human portrait region; The style loss is used to measure the style difference between the stylized image and the style reference image. By minimizing the style loss, the style features of the stylized image are made closer to those of the style reference image.

5. A method for color grading of portrait photographs based on deep neural networks, characterized in that, The method includes: Receive the image to be processed and the reference image; Based on the image to be processed and the reference image, the image to be processed is stylized using a color model trained by the method described in any one of claims 1 to 4, to obtain a stylized image corresponding to the image to be processed.

6. A portrait photography image color grading system based on deep neural networks, characterized in that, The system includes an acquisition module and a stylization processing module, wherein: The acquisition module is used to receive the image to be processed and the reference image; The stylization processing module is used to perform stylization processing on the image to be processed based on the image to be processed and the reference image, using a color model trained by the method described in any one of claims 1 to 5, to obtain a stylized image corresponding to the image to be processed.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images