Intelligent color chasing method based on deep learning

Abstract

The invention discloses an intelligent color chasing method based on deep learning. The method comprises the following steps: carrying out image feature extraction and semantic feature extraction on an original image to obtain a first image feature and a first semantic feature; performing image feature extraction and semantic feature extraction on the reference image to obtain a second image feature and a second semantic feature; performing feature fusion on the first image feature and the first semantic feature and the second image feature and the second semantic feature to obtain a fusion feature; on the basis of the reference image, through the fusion features, performing movie processing and hue processing on the original image; and generating a target image with the same color, movie and style as the reference image based on a processing result. According to the method, various limitations of a traditional color matching mode and a style migration technology are solved.

Description

Technical Field

[0001] The present invention relates to the field of portrait photography post-production technology, and in particular to an intelligent color tracking method based on deep learning. Background Art

[0002] At present, image coloring is a core demand in the fields of photography, film and television post-production, and visual creativity, because color and tone adjustment are the most critical factors in determining visual effects. However, traditional image coloring methods usually rely on complex professional software and a large amount of manual operations, which not only has a high threshold but also takes time. Especially in the face of scenes where the color tone requires precision and large-scale processing (such as commercial photography or film and television production), manual coloring is not only inefficient, but also requires the colorist to have a keen artistic perception of color. In order to achieve the ideal visual effect, photographers or designers often spend a lot of time fine-tuning the color and adjusting every detail in the photo frame by frame. Especially when you want a series of photos or works to maintain a unified color style, or when the customer photos and sample photos taken are highly consistent, this task becomes more cumbersome and time-consuming, requiring people to perceive the subtle differences in light, saturation, and hue of different images, which is highly professional. In this context, the field of photography urgently needs to change the coloring to lower the threshold of coloring and improve the overall coloring quality and efficiency. Before the present invention, there was no mature algorithm for AI color tracking, and a similar technical method was style transfer. The basis of style transfer algorithms is neural network feature extraction and image reconstruction, that is, transferring the overall style of one image to another. However, these methods have the following problems:

[0003] 1. Limited effect and unprofessional: Existing methods tend to achieve overall style transfer and reconstruct the overall color. Generally, the color with a larger proportion will dominate the whole. At the same time, the effect only performs simple color processing, without analyzing and matching details such as color, tone, contrast, etc., especially unable to handle complex scenes.

[0004] 2. Lack of semantic level alignment: Style transfer is often a global process, and cannot be independently adjusted for different semantic regions in the image (such as people, background, and objects). For example, if the proportion of green trees in the reference image is too large, the sky in the original image will turn green, resulting in unreasonable mapping.

[0005] 3. Consistency issue: In batch processing or video sequences, it is difficult for style transfer algorithms to maintain consistency in batch processing, resulting in a lack of uniformity in the results required for professional applications. Summary of the invention

[0006] In view of this, the purpose of the present invention is to propose an intelligent color tracking method based on deep learning. This method solves the various limitations of traditional color adjustment methods and style transfer technologies. Users can upload a reference image and a picture to be adjusted, and the model intelligently analyzes the semantics, color, tone, etc. of the reference image and the picture to be adjusted. The model matches the analyzed semantics, color, tone, etc. one by one, and intelligently assigns the information of the reference image to the picture to be adjusted, so as to achieve the purpose of intelligent color tracking.

[0007] According to one aspect of the present invention, there is provided an intelligent color tracking method based on deep learning, the method comprising:

[0008] Performing image feature extraction and semantic feature extraction on the original image to obtain a first image feature and a first semantic feature;

[0009] Performing image feature extraction and semantic feature extraction on the reference image to obtain a second image feature and a second semantic feature;

[0010] Fusing the first image feature, the first semantic feature, the second image feature, and the second semantic feature to obtain a fused feature;

[0011] Based on the reference image and the fusion feature, the original image is subjected to tone processing and color tone processing respectively;

[0012] Based on the processing result, a target image having the same color, tone and style as the reference image is generated.

[0013] In the above technical solution, multi-dimensional feature sets are extracted from the original image and the reference image. These feature sets include basic visual features such as color, tone, texture and shape, as well as higher-level semantic features involving object category recognition and scene understanding. Specifically, a first set of image features and semantic features are extracted from the original image, while a second set of corresponding features are extracted from the reference image. These features can capture the underlying visual information and high-level semantic information of the image. Further, the two sets of features are fused to obtain a comprehensive feature representation, which not only includes the underlying visual features of the image, but also covers the high-level semantic information. The feature fusion process is the key to realizing intelligent color tracking, which ensures that the model can consider both visual information and semantic information during the conversion process, thereby ensuring semantic accuracy while maintaining content consistency. Based on the fused features and the reference image, the tone and hue of the original image are adjusted. Tone adjustment mainly involves the optimization of contrast, black, white, exposure, highlights, and shadows, while hue adjustment focuses on the precise adjustment of color. These adjustments are intended to make the original image match the reference image in visual style while retaining its original content. Finally, a target image with the same style as the reference image is generated.

[0014] In some embodiments, the first image feature, the first semantic feature, and the second image feature and the second semantic feature are fused to obtain a fused feature, including:

[0015] Mapping the first image feature and the first semantic feature to the same dimensional space, and adding them element by element to obtain the original feature;

[0016] Mapping the second image feature and the second semantic feature to the same dimensional space, and adding them element by element to obtain a reference feature;

[0017] The original features are connected with the reference features to obtain the fused features.

[0018] In the above technical solution, the image features and semantic features extracted from the original image are first dimensionally mapped to ensure that they can be compared and merged in a unified feature space. This mapping process may involve scaling, normalization or other necessary transformation operations of the features, aiming to achieve consistency and comparability of the feature set. Through this preprocessing, the feature set of the original image can be mapped to the same dimensional space as the feature set of the reference image. After the mapping is completed, the feature set of the original image is added element by element to the feature set of the reference image to obtain a comprehensive feature set that combines the visual and semantic information of the original image. This process is also applicable to the reference image, and a comprehensive feature set that combines the visual and semantic information of the reference image is obtained by the same method. Subsequently, we perform a feature concatenation operation, that is, splicing the original feature set and the reference feature set along the feature dimension to form a longer feature vector. Feature concatenation is a common feature fusion technique that can merge the information of two feature sets for subsequent processing steps. Through this feature fusion method, a fused feature set containing the complete information of the original image and the reference image can be constructed. This fused feature set not only enhances the model's ability to understand image content, but also provides a richer information basis for tasks such as style transfer and image enhancement, thereby achieving better processing results.

[0019] In some embodiments, based on the reference image and through the fusion feature, the original image is subjected to tone processing and color tone processing respectively, including:

[0020] Based on the reference image, the tone information and color tone information corresponding to each semantic area are adjusted respectively through the fusion feature, and the adjusted tone information and color tone information are spliced ​​to obtain the color tracking feature.

[0021] In the above technical solution, by adjusting the tone and color for each semantic region separately, the style of the reference image can be more accurately imitated. This method can ensure the consistency and naturalness of style transfer in different regions, especially when dealing with complex scenes. Since the tone and color information is adjusted separately for each semantic region, the final spliced ​​result appears natural and realistic visually. This method allows different style transfer strategies to be applied to different semantic regions, providing higher flexibility. This flexibility allows the method to adapt to a variety of different images and style transfer requirements.

[0022] According to another aspect of the present invention, a training method for an image color tracking neural network is provided, wherein the neural network includes a semantic segmentation network, a first image feature extraction network, a second image feature extraction network, a feature fusion module, a tone module, a hue module, and a generation module; the method includes:

[0023] After training the semantic segmentation network until the network converges, the semantic segmentation network is frozen;

[0024] Collecting a number of data pairs to construct a training set; wherein the data pairs include: a sample image and reference images of a number of style types generated based on the sample image, each style type includes a number of reference images, and a true label corresponding to the reference image;

[0025] Inputting the sample image into the semantic segmentation network and the first image feature extraction network respectively to obtain the first image feature and the first semantic feature;

[0026] Select a style image from the data pair and input it into the semantic segmentation network and the second image feature extraction network respectively to obtain the second image feature and the second semantic feature;

[0027] Inputting the first image feature, the first semantic feature, the second image feature, and the second semantic feature into a feature fusion module for feature fusion to obtain a fusion feature;

[0028] Based on the style image, the fusion features are respectively input into a tone module and a color tone module to perform tone processing and color tone processing on the sample image;

[0029] Based on the processing result, the result is input into a generation module to generate a target image with the same style as the style image;

[0030] Calculate the loss value based on the target image and the true label, and

[0031] Based on the loss value, the neural network after freezing the semantic segmentation network is trained.

[0032] In the above technical scheme, in order to better use the above method, the present application proposes a training method for an image color tracking neural network, and each step corresponds to each step of the above method. The specific principles have been described above and will not be repeated here.

[0033] According to another aspect of the present invention, there is provided an image color tracking neural network, comprising

[0034] A semantic segmentation network is configured to extract semantic features from the original image and the reference image respectively to obtain a first semantic feature and a second semantic feature;

[0035] A first image feature extraction network is configured to extract image features from the original image to obtain a first image feature;

[0036] A second image feature extraction network is configured to extract image features from the reference image to obtain second image features;

[0037] A feature fusion module is configured to fuse the first image feature, the first semantic feature, the second image feature, and the second semantic feature to obtain a fused feature;

[0038] A tone module, configured to perform tone processing on the original image based on the reference image through the fusion feature;

[0039] A tone module, configured to perform tone processing on the original image based on the reference image through the fusion feature;

[0040] The generation module is configured to generate a target image having the same color, tone and style as the reference image based on the processing result.

[0041] In the above technical scheme, in order to better use the above method, the present application proposes an image color tracking neural network, and each module and network corresponds to each step of the above method. The specific principles have been described above and will not be repeated here.

[0042] According to another aspect of the present invention, there is provided an intelligent color tracking system based on deep learning, based on the above method; comprising:

[0043] A first extraction module, used for performing image feature extraction and semantic feature extraction on the original image to obtain a first image feature and a first semantic feature;

[0044] A second extraction module is used to extract image features and semantic features from the reference image to obtain second image features and second semantic features;

[0045] A feature fusion module, used for fusing the first image feature, the first semantic feature, the second image feature, and the second semantic feature to obtain a fusion feature;

[0046] A processing module, used for performing tone processing and color tone processing on the original image based on the reference image through the fusion feature;

[0047] A generation module is used to generate a target image with the same color, tone and style as the reference image based on the processing result.

[0048] In the above technical scheme, in order to better use the above method, the present application proposes an intelligent color tracking system based on deep learning, and each step corresponds to each step of the above method. The specific principles have been described above and will not be repeated here.

[0049] According to another aspect of the present invention, there is provided an intelligent color tracking device based on deep learning, comprising:

[0050] at least one processor and a memory communicatively coupled to the at least one processor;

[0051] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above method.

[0052] In the above technical solution, in order to better run and process the method, the above method is stored in a memory, and the stored method is executed by a processor. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.

[0053] According to another aspect of the present invention, a computer-readable storage medium is provided, storing a computer program, wherein the computer program implements the above method when executed by a processor.

[0054] In the above technical solution, in order to better run and use the method, the above method is stored in a computer-readable storage medium, and the above method is implemented by a processor. It should be noted that the principle and effect of each step have been described above and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 It is a schematic diagram of the reasoning process of an embodiment of an intelligent color tracking method based on deep learning of the present invention;

[0057] Figure 2 It is a training process diagram of an embodiment of an intelligent color tracking method based on deep learning of the present invention;

[0058] Figure 3 It is a schematic diagram of a model structure of an embodiment of an intelligent color tracking method based on deep learning of the present invention;

[0059] Figure 4 It is a schematic diagram of the model structure of the prior art;

[0060] Figure 5 It is a structural schematic diagram of an embodiment of an intelligent color tracking system based on deep learning of the present invention;

[0061] Figure 6 is a result comparison diagram of an embodiment of an intelligent color tracking method based on deep learning of the present invention; wherein, in the figure Figure 6 A represents the original image, Figure 6 B represents the color tracking result diagram, Figure 6 C represents a reference figure. DETAILED DESCRIPTION

[0062] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present invention, but are not intended to limit the scope of the present invention. Similarly, the following examples are only partial embodiments of the present invention rather than all embodiments, and all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0063] The present invention provides an intelligent color tracking method based on deep learning, which solves the various limitations of traditional color adjustment methods and style transfer technologies. Users can upload a reference picture and a picture to be adjusted, and the model intelligently analyzes the semantics, color, tone and other contents of the reference picture and the picture to be adjusted. The model matches the analyzed semantics, color, tone and other contents one by one, and intelligently assigns the information of the reference picture to the picture to be adjusted, so as to achieve the purpose of intelligent color tracking.

[0064] Embodiment 1

[0065] See also Figure 1 , an intelligent color tracking method based on deep learning, the method comprising:

[0066] A1. Perform image feature extraction and semantic feature extraction on the original image to obtain a first image feature and a first semantic feature;

[0067] A2, performing image feature extraction and semantic feature extraction on the reference image to obtain a second image feature and a second semantic feature;

[0068] A3, fusing the first image feature, the first semantic feature, the second image feature, and the second semantic feature to obtain a fused feature;

[0069] A4, performing tone processing and color tone processing on the original image based on the reference image through the fusion feature;

[0070] A5. Generate a target image based on the processing result, which has the same color, tone and style as the reference image.

[0071] In this embodiment, multi-dimensional feature sets are extracted from the original image and the reference image. These feature sets include basic visual features such as color, tone, texture, and shape, as well as higher-level semantic features, involving object category recognition and scene understanding. Specifically, a first set of image features and semantic features are extracted from the original image, and a second set of corresponding features are extracted from the reference image. These features can capture the underlying visual information and high-level semantic information of the image. Further, the two sets of features are fused to obtain a comprehensive feature representation, which not only includes the underlying visual features of the image, but also covers high-level semantic information. The feature fusion process is the key to achieving style transfer, which ensures that the model can consider both visual information and semantic information during the conversion process, thereby ensuring semantic accuracy while maintaining content consistency. Based on the fused features and the reference image, the tone and hue of the original image are adjusted. Tone adjustment mainly involves the optimization of contrast, black, white, exposure, highlights, and shadows, while hue adjustment focuses on the precise adjustment of color. These adjustments are intended to make the original image match the reference image in visual style while retaining its original content. Finally, a target image with the same style as the reference image is generated.

[0072] In this embodiment, A3, fusing the first image feature, the first semantic feature, and the second image feature and the second semantic feature to obtain a fused feature, includes:

[0073] A31, mapping the first image feature and the first semantic feature to the same dimensional space, and adding them element by element to obtain the original feature;

[0074] A32, mapping the second image feature and the second semantic feature to the same dimensional space, and adding them element by element to obtain a reference feature;

[0075] A33. Connect the original features with the reference features to obtain fused features.

[0076] In this embodiment, the image features and semantic features extracted from the original image are first dimensionally mapped to ensure that they can be compared and merged in a unified feature space. This mapping process may involve scaling, normalization or other necessary transformation operations of the features, aiming to achieve consistency and comparability of the feature set. Through this preprocessing, the feature set of the original image can be mapped to the same dimensional space as the feature set of the reference image. After the mapping is completed, the feature set of the original image is added element by element to the feature set of the reference image to obtain a comprehensive feature set that combines the visual and semantic information of the original image. This process is also applicable to the reference image, and a comprehensive feature set that combines the visual and semantic information of the reference image is obtained by the same method. Subsequently, we perform a feature connection operation, that is, splicing the original feature set and the reference feature set along the feature dimension to form a longer feature vector. Feature connection is a common feature fusion technique that can merge the information of two feature sets for subsequent processing steps. Through this feature fusion method, a fused feature set containing complete information of the original image and the reference image can be constructed. This fused feature set not only enhances the model's ability to understand image content, but also provides a richer information basis for tasks such as style transfer and image enhancement, thereby achieving better processing results.

[0077] In this embodiment, A4, based on the reference image through the fusion feature, the original image is subjected to tone processing and color tone processing respectively, including: based on the reference image through the fusion feature, the tone information and color tone information corresponding to each semantic area are adjusted respectively, and the adjusted tone information and color tone information are spliced ​​to obtain the color tracking feature. By performing separate tone and color tone adjustments for each semantic area, the style of the reference image can be more accurately imitated. This method can ensure the consistency and naturalness of style transfer in different areas, especially when dealing with complex scenes. Since the tone and color tone information are adjusted separately for each semantic area, the final splicing result appears natural and realistic visually. This method allows different style transfer strategies to be applied to different semantic areas, providing greater flexibility. This flexibility allows the method to adapt to a variety of different image and style transfer requirements.

[0078] Embodiment 2

[0079] See also Figure 2 According to another aspect of the present invention, a training method for an image color tracking neural network is provided, wherein the neural network comprises a semantic segmentation network, a first image feature extraction network, a second image feature extraction network, a feature fusion module, a tone module, a hue module, and a generation module; the method comprises:

[0080] B1. After training the semantic segmentation network until the network converges, freeze the semantic segmentation network;

[0081] B2. Collect several data pairs to construct a training set; wherein the data pairs include: a sample image and reference images of several style types generated based on the sample image, each style type includes several reference images, and a true label corresponding to the reference image;

[0082] B3, inputting the sample image into the semantic segmentation network and the first image feature extraction network respectively to obtain the first image feature and the first semantic feature;

[0083] B4, select a style image from the data pair, and input it into the semantic segmentation network and the second image feature extraction network respectively to obtain the second image feature and the second semantic feature;

[0084] B5, inputting the first image feature, the first semantic feature, the second image feature, and the second semantic feature into a feature fusion module for feature fusion to obtain a fusion feature;

[0085] B6, based on the style image, input the fusion features into a tone module and a color tone module respectively, and perform tone processing and color tone processing on the sample image respectively;

[0086] B7. Based on the processing result, the result is input into a generation module to generate a target image with the same style as the style image;

[0087] B8, calculating a loss value based on the target image and the true label, and,

[0088] B9. Based on the loss value, train the neural network after freezing the semantic segmentation network.

[0089] In this embodiment, B5, the first image feature, the first semantic feature, the second image feature, and the second semantic feature are input into a feature fusion module for feature fusion to obtain a fusion feature.

[0090] Among them, the feature fusion module is configured to map the first image feature and the first semantic feature to the same dimensional space, and add them element by element to obtain the original feature; map the second image feature and the second semantic feature to the same dimensional space, and add them element by element to obtain the reference feature; and feature connect the original feature and the reference feature to obtain the fused feature.

[0091] In this embodiment, B6, based on the reference image, the original image is subjected to tone processing and color tone processing respectively through the fusion feature.

[0092] The tone module is configured to adjust the tone information corresponding to each semantic area; and the color tone module is configured to adjust the color tone information corresponding to each semantic area.

[0093] In the above technical scheme, in order to better use the above method, the present application proposes a training method for an image color tracking neural network, and each step corresponds to each step of the above method. The specific principles have been described above and will not be repeated here.

[0094] Embodiment 3

[0095] See also Figure 3 , an image color tracking neural network, including

[0096] A semantic segmentation network is configured to extract semantic features from the original image and the reference image respectively to obtain a first semantic feature and a second semantic feature;

[0097] A first image feature extraction network is configured to extract image features from the original image to obtain a first image feature;

[0098] A second image feature extraction network is configured to extract image features from the reference image to obtain second image features;

[0099] A feature fusion module is configured to fuse the first image feature, the first semantic feature, the second image feature, and the second semantic feature to obtain a fused feature;

[0100] A tone module, configured to perform tone processing on the original image based on the reference image through the fusion feature;

[0101] A tone module, configured to perform tone processing on the original image based on the reference image through the fusion feature;

[0102] The generation module is configured to generate a target image having the same color, tone and style as the reference image based on the processing result.

[0103] In the above technical scheme, in order to better use the above method, the present application proposes an image color tracking neural network, and each module and network corresponds to each step of the above method. The specific principles have been described above and will not be repeated here.

[0104] In order to better explain the first to third embodiments, the following three embodiments will be specifically described:

[0105] 1) Model design

[0106] Conventional style transfer models usually consist of input images, reference images, and output parameters, and their architecture is usually as follows: Figure 4For general style transfer models, the input image and the reference image are usually Figure 1 The images are thrown into the same network, and feature fusion and style transfer are achieved within the network. This method is usually based on global feature matching and cannot accurately align different semantic areas in the image (such as sky, skin, trees, etc.). The feature fusion process within the model is often a black box operation, and it is difficult to control the intensity. In addition, the global migration method easily ignores the local details of the image, such as the fine transition of edges and textures, resulting in the generated results appearing blurred or unnatural in high-resolution scenes, which is difficult to meet professional-level color or tone requirements. Based on the above problems, the algorithm architecture of the color tracking model was redesigned. Please refer to Figure 3 .

[0107] The network is divided into several parts: graph feature extraction network, semantic extraction network, feature fusion module, tone module, color tone module, and generation module. By combining the technical advantages of Transformer and CNN, the model performs well in semantic understanding, color adjustment, and tone adjustment. The algorithm has both global semantic understanding capabilities and local detail processing capabilities. It also optimizes computing efficiency and supports large-scale batch tasks. The network needs to input the original image and the reference image. The purpose is to change the tone, color, and style of the original image to the same as the reference image. Figure 1 Like. First, there are two feature extraction networks, which are two fully convolutional networks, used to extract the features of the reference image and the input image. The input is the image, and the output is the feature of the image. The semantic extraction network is mainly used to extract the semantic information of the image. The input is the image, and the output is the vector of semantic information. The feature fusion module fuses the input image information and the corresponding semantic information, the reference image information and the corresponding semantic information. The fusion method is mainly to perform a linear layer on the original image and the original image semantic information to keep the dimensions consistent, and then add them. The same operation is also performed on the reference image and the semantic information corresponding to the reference image, and then the added features are spliced. The formula is as follows:

[0108] d = concatenate(a+c1, b+c2)

[0109] The entire feature fusion module is a self-attention module of the transformer. The tone module and the hue module adjust the tone and hue information corresponding to each semantic region respectively. The tone and hue separation architecture designed for color grading tasks is adopted. The tone and color are deeply analyzed and understood through the dedicated tone module and hue module, and the delicate color features and tone information are accurately extracted from the image. This refinement process ensures that the converted image highly restores the reference image in terms of color accuracy and tone performance, while maintaining the naturalness of texture details and transitions. In addition, the model fuses semantic information, color information, and tone information one-to-one to ensure that the corresponding object follows the tone and color of the corresponding object. Finally, the tone and hue information are spliced ​​together, and a result image with the same size as the input image is generated through a fully convolutional generation module. The result image is the tone and hue of the reference image. The tone and hue module is a self-attention structure. The corresponding functions of the tone module and the hue module are realized by innovatively training the two modules alternately and iteratively, and then fusing the training. First, alternate training is used. When training the tone module, the tone module parameters are frozen, and the paired data for training only adjusts the tone information such as exposure, brightness, highlight, shadow, black, and white. The tone module is guided to learn tone information. Next, the tone module parameters are frozen and the color module parameters are released. The paired data for training only modifies the color, and the color module is guided to learn color information. After training until the loss converges, the parameters of the two modules are released at the same time, and the paired data modifies the tone and color information at the same time.

[0110] 2) Training methods

[0111] First, we need to train a semantic segmentation network for all things. This network refers to the network of mask2former, integrates multiple open source data sets for joint training, and finally conducts continuous training based on our own high-precision mask of all things. Joint training can take advantage of the dataset to train the basic segmentation capabilities of the model, and then fine-tune the segmentation results through high-precision annotation results. Through the self-trained high-quality semantic segmentation model, the image is segmented at the level of all things, and different semantic areas are identified, such as human skin, hair, trees, sky, grass, etc. Through high-quality semantic segmentation data and segmentation models, high-quality semantic results of all things are obtained. Through semantic-level matching and color adjustment, ensure that the skin in the reference image corresponds to the skin in the original image, and the sky corresponds to the sky, avoiding the common "misalignment" problem in style transfer algorithms.

[0112] After training the semantic segmentation network for everything, you need to freeze the network and then start training several modules of the color tracking network.

[0113] First, we need the original image and the image of the same style after retouching the original image. From these images, we randomly select an original image and a retouched image as the input image, as well as the paired data of the ground truth, and then randomly select one from the other retouched images as the reference image. The output image and the reference image must be of the same style. Then the original image and the reference image are simultaneously input into the semantic network and the feature extraction network. Finally, the generation module generates a color-tracked image of the same size as the original image. The color-tracked image and the ground truth results are supervised at the local level. That is, each semantic area is supervised separately. For example, the skin area of ​​the character is supervised separately, the hair area is supervised separately, the tree area is supervised separately, and so on. The model uses L1 loss, perceptual loss (vgg loss), and lab loss.

[0114]

[0115] Where, L1 loss, N is the number of samples. i The true value of the i-th sample, The predicted value of the i-th sample, |.| represents the absolute value.

[0116]

[0117] In the formula, VGG Loss is a perceptual loss, which is usually used in image generation tasks (such as image style transfer, image super-resolution reconstruction, etc.) to measure the difference between the generated image and the target image in high-level semantic features. It calculates the loss by extracting features in the pre-trained VGG network. The feature map extracted from the lth layer of the pre-trained VGG network. L: The set of layers in the VGG network used (such as layer 3, layer 5, etc.). l The weighting coefficient of each layer controls the contribution of different layers to the loss. ‖.‖ p Usually represents the p-norm.

[0118] The lab loss converts the RGB output to the lab space, and then uses the same l1 loss as formula (1) for supervision. At the same time, for inter-group consistency, the intra-group consistency loss is added to constrain the model to generate consistent output results for the same input image and different reference images. We use labeled grouped data to achieve this. A group of data refers to a specific color style with multiple images. During the training phase, we ensure that the data in a training iteration must appear in groups. The specific implementation is that the input image of the model remains unchanged, but the reference image is randomly obtained from the same group, and the output image must ensure the consistency of the results. The algorithm performs targeted optimization on batch images, and by adding a consistency constraint algorithm, it ensures the consistency of color and tone of the photo group, which is suitable for professional fields such as commercial photography and film and television post-production.

[0119] 3) Usage

[0120] When using it, just like when training, you need to input an original image and a reference image into the network. The network will analyze the color and tone relationship between the original image and the reference image, and then adjust the original image to match the reference image. Figure 1 See also Figure 6 , in the figure Figure 6 A represents the original image, Figure 6 B represents the color tracking result diagram, Figure 6 C represents the reference image

[0121] The present invention has the following advantages:

[0122] advantage:

[0123] 1. The overall model has a large number of parameters, reaching more than one billion. Compared with the traditional style transfer, the number of parameters is about 10 million, which is dozens of times larger, so it can learn massive amounts of data. Based on the large model, through massive data training, it can learn common semantic features, color features, and tone features from rich multimodal data. It not only improves the adaptability to various scenes and styles, but also significantly enhances the stability and generation effect of complex image processing.

[0124] 2. The technical solution breaks through the limitation of traditional style transfer that only considers the global style and ignores the semantic consistency of the image. By combining semantic segmentation information, semantic alignment can be achieved at the pixel level, for example: person to person, skin to skin, tree to tree. This semantic perception processing ensures that the adjustment of color and tone is more refined and meets the needs of the scene, greatly improving the user's perception quality and color accuracy.

[0125] 3. The solution divides image processing into a tone module and a color module. The modular structural design makes the processing more flexible and accurate. The tone module mainly adjusts light and shadow and contrast to ensure the layering of the overall picture; the color module is responsible for processing the migration and matching of color styles. This separate design can optimize modules separately according to different needs and avoid interference introduced during global adjustments.

[0126] 4. It breaks the defect that traditional style transfer only cares about color, but cannot handle tone, color transition and details. The generated results are excellent in style consistency and natural transition. Whether it is a single image or batch processing, it can maintain the consistency of style in different regions and semantics. For people and complex scenes, the generated images not only have beautiful visual effects, but also highly retain the details and semantic consistency of the original images.

[0127] 5. The large model is pre-trained with a large amount of data and can handle a variety of complex scenes (such as backlighting, color temperature imbalance, etc.) with extremely high robustness. The tone and color modules can expand more functions according to the scene, such as custom color adjustment for specific areas, local detail optimization, etc.

[0128] Embodiment 4

[0129] See also Figure 5 , an intelligent color tracking system based on deep learning, based on the methods described in Embodiments 1 to 3; comprising:

[0130] A first extraction module, used for performing image feature extraction and semantic feature extraction on the original image to obtain a first image feature and a first semantic feature;

[0131] A second extraction module is used to extract image features and semantic features from the reference image to obtain second image features and second semantic features;

[0132] A feature fusion module, used for fusing the first image feature, the first semantic feature, the second image feature, and the second semantic feature to obtain a fusion feature;

[0133] A processing module, used for performing tone processing and color tone processing on the original image based on the reference image through the fusion feature;

[0134] A generation module is used to generate a target image with the same color, tone and style as the reference image based on the processing result.

[0135] In this embodiment, in order to better use the methods described in Embodiments 1 to 3, the present application proposes an intelligent color tracking system based on deep learning, and each step corresponds to each step of the methods described in Embodiments 1 to 3. The specific principles have been described above and will not be repeated here.

[0136] Embodiment 5

[0137] An intelligent color tracking device based on deep learning, comprising:

[0138] at least one processor and a memory communicatively coupled to the at least one processor;

[0139] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above method.

[0140] In this embodiment, in order to better run and process the methods described in Embodiments 1 to 3, the methods described in Embodiments 1 to 3 are stored in a memory, and a processor is used to execute the stored methods described in Embodiments 1 to 3. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.

[0141] Embodiment 6

[0142] A computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the method described in any one of embodiments 1 to 3 is implemented.

[0143] In the above technical solution, in order to better run and use the methods described in Embodiments 1 to 3, the methods described in Embodiments 1 to 3 are stored in a computer-readable storage medium, and a processor is used to implement the methods described in Embodiments 1 to 3. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.

[0144] The above descriptions are only some embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent device or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An intelligent color tracking method based on deep learning, characterized in that: The method comprises: Performing image feature extraction and semantic feature extraction on the original image to obtain a first image feature and a first semantic feature; Performing image feature extraction and semantic feature extraction on the reference image to obtain a second image feature and a second semantic feature; Fusing the first image feature, the first semantic feature, the second image feature, and the second semantic feature to obtain a fused feature; Based on the reference image and the fusion feature, the original image is subjected to tone processing and color tone processing respectively; Based on the processing result, a target image having the same color, tone and style as the reference image is generated.

2. The intelligent color tracking method based on deep learning as claimed in claim 1, characterized in that: The first image feature, the first semantic feature, and the second image feature and the second semantic feature are fused to obtain a fused feature, including: Mapping the first image feature and the first semantic feature to the same dimensional space, and adding them element by element to obtain the original feature; Mapping the second image feature and the second semantic feature to the same dimensional space, and adding them element by element to obtain a reference feature; The original features are connected with the reference features to obtain the fused features.

3. The intelligent color tracking method based on deep learning as claimed in claim 1, characterized in that: Based on the reference image and the fusion feature, the original image is subjected to tone processing and color tone processing respectively, including: Based on the reference image, the tone information and color tone information corresponding to each semantic area are adjusted respectively through the fusion feature, and the adjusted tone information and color tone information are spliced ​​to obtain the color tracking feature.

4. A training method for an image color tracking neural network, characterized in that: The neural network includes a semantic segmentation network, a first image feature extraction network, a second image feature extraction network, a feature fusion module, a tone module, a color tone module, and a generation module; the method includes: After training the semantic segmentation network until the network converges, the semantic segmentation network is frozen; Collecting a number of data pairs to construct a training set; wherein the data pairs include: a sample image and reference images of a number of style types generated based on the sample image, each style type includes a number of reference images, and a true label corresponding to the reference image; Inputting the sample image into the semantic segmentation network and the first image feature extraction network respectively to obtain the first image feature and the first semantic feature; Select a style image from the data pair and input it into the semantic segmentation network and the second image feature extraction network respectively to obtain the second image feature and the second semantic feature; Inputting the first image feature, the first semantic feature, the second image feature, and the second semantic feature into a feature fusion module for feature fusion to obtain a fusion feature; Based on the style image, the fusion features are respectively input into a tone module and a color tone module to perform tone processing and color tone processing on the sample image respectively; Based on the processing result, the result is input into a generation module to generate a target image with the same style as the style image; Calculate the loss value based on the target image and the true label, and Based on the loss value, the neural network after freezing the semantic segmentation network is trained.

5. The method for training an image color tracking neural network according to claim 4, characterized in that: Inputting the first image feature, the first semantic feature, the second image feature and the second semantic feature into a feature fusion module for feature fusion to obtain a fusion feature, Among them, the feature fusion module is configured to map the first image feature and the first semantic feature to the same dimensional space, and add them element by element to obtain the original feature; map the second image feature and the second semantic feature to the same dimensional space, and add them element by element to obtain the reference feature; and feature connect the original feature and the reference feature to obtain the fused feature.

6. The method for training an image color tracking neural network according to claim 4, characterized in that: Based on the reference image, the original image is subjected to tone processing and color tone processing respectively through the fusion feature. The tone module is configured to adjust the tone information corresponding to each semantic area; and the color tone module is configured to adjust the color tone information corresponding to each semantic area.

7. An image color tracking neural network, characterized in that: include A semantic segmentation network is configured to extract semantic features from the original image and the reference image respectively to obtain a first semantic feature and a second semantic feature; A first image feature extraction network is configured to extract image features from the original image to obtain a first image feature; A second image feature extraction network is configured to extract image features from the reference image to obtain second image features; A feature fusion module is configured to fuse the first image feature, the first semantic feature, the second image feature, and the second semantic feature to obtain a fused feature; A tone module, configured to perform tone processing on the original image based on the reference image through the fusion feature; A tone module, configured to perform tone processing on the original image based on the reference image through the fusion feature; The generation module is configured to generate a target image having the same color, tone and style as the reference image based on the processing result.

8. An intelligent color tracking system based on deep learning, characterized in that: The method according to any one of claims 1 to 6; comprising: A first extraction module is used to extract image features and semantic features from the original image to obtain a first image feature and a first semantic feature; A second extraction module is used to extract image features and semantic features from the reference image to obtain second image features and second semantic features; A feature fusion module, used for fusing the first image feature, the first semantic feature, the second image feature, and the second semantic feature to obtain a fusion feature; A processing module, used for performing tone processing and color tone processing on the original image based on the reference image through the fusion feature; A generation module is used to generate a target image with the same color, tone and style as the reference image based on the processing result.

9. An intelligent color tracking device based on deep learning, characterized in that: include: at least one processor and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as claimed in any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images