Image color fusion method and device based on convolutional neural network and storage medium

Abstract

The present application relates to a convolutional neural network-based image color fusion method, device and storage medium, applied to the technical field of image processing, comprising: realizing color adaptation of foreground and background under the double constraints of foreground mask and skin segmentation mask through a color fusion model, optimizing the color naturalness of the skin area, effectively solving the problems of rigid fusion and strange skin color in the prior art, and improving the stability of the method; The high-resolution processing strategy of "low-resolution inference + detail preservation color migration" not only avoids the problems of large video memory consumption and long time consumption of direct high-resolution inference, but also solves the problems of loss of global color information in block inference and introduction of edge abnormalities in traditional high-frequency information recovery; The detail preservation color migration model can accurately extract the color features of the low-resolution fusion result and completely retain the texture details of the high-resolution original image, realizing the dual goals of color fusion and detail preservation; Automatic processing significantly reduces time and labor costs.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to an image color fusion method, apparatus, and storage medium based on convolutional neural networks. Background Technology

[0002] With the rapid development of modern digital image processing technology and the increasing diversification of image compositing needs, users have higher requirements for the naturalness, efficiency, and adaptability of image color grading and blending. They need precise color grading and blending techniques to harmoniously integrate the cut-out foreground area with the original background. However, limited by current technology, the color grading and blending effects cannot achieve high quality, high efficiency, stability, and flexibility, making it difficult to meet the diverse needs of professional commercial processing and everyday creative work.

[0003] Existing image color fusion techniques can be broadly categorized into three types: The first type involves manual color fusion based on traditional image processing software such as Photoshop. This method involves manually adjusting parameters such as brightness, contrast, color temperature, and color balance to achieve tonal matching between the foreground and background. The color fusion effect is highly dependent on the user's professional skills and experience, with cumbersome procedures, high time and labor costs, and the fusion effect is prone to problems such as tonal banding and abrupt light and shadow. The second type is automatic color fusion based on traditional algorithms such as histogram matching and Poisson fusion. While this method eliminates the need for extensive manual parameter adjustments, it can only achieve simple color mean or local edge matching, failing to accurately capture the global light and shadow distribution characteristics of the foreground and background. In complex scenes such as backlighting and multiple light sources, it is prone to problems such as image overexposure, reduced contrast, and strange skin tones in portraits. The third type is color fusion based on deep learning diffusion models. This method has certain advantages in color style diversity, but it is prone to altering the original local details of the image, is time-consuming, consumes a lot of video memory, and is difficult to efficiently process high-resolution images, thus limiting its practicality. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide an image color fusion method, apparatus and storage medium based on convolutional neural networks, which aims to solve the problems of high operation threshold, stiff fusion effect, low high resolution processing efficiency and easy loss of detail in existing image color fusion technology.

[0005] According to a first aspect of the present invention, an image color fusion method based on a convolutional neural network is provided, the method comprising:

[0006] Obtain the original image to be color-blended uploaded by the user, wherein the original image to be color-blended is the original image of the foreground and background that has not been blended; Mark the areas in the original image to be color-blended that require color blending processing to obtain a color blending area mask image; mark the skin areas in the original image to be color-blended to obtain a skin area mask image; The original image to be fused, the color fusion region mask image, and the skin region mask image are input into the pre-trained color fusion model; The pre-trained color fusion model, under the condition of downsampling and low-resolution inference of each input image, uses the color fusion region mask image and the skin region mask image as dual constraints. It extracts the color features of the foreground, background and skin color in the original image to be fused, adjusts the foreground color based on the extracted color features to blend it with the background, and outputs the color fusion result image; the resolution of the color fusion result image is lower than that of the original image to be fused. The original image to be color-blended and the resulting color-blended image are input into a pre-trained detail-preserving color transfer model. The pre-trained detail-preserving color transfer model adjusts the colors of the original image to match those of the resulting color-blended image. Figure 1 The color grading result image is output while preserving the details of the original image to be fused with the colors. The resolution of the color grading result image is greater than that of the color fusion result image.

[0007] Preferably, The training of the color fusion model includes: Randomly select a tone transfer original image and a tone reference original image. The tone transfer original image is the target image to be tone adjusted, and the tone reference original image is a reference image that provides a tone benchmark. The original tone transfer image and the original tone reference image are input into a pre-trained tone transfer model. The pre-trained tone transfer model extracts the overall tone features of the original tone reference image and performs differentiated color mapping on each local area of ​​the original tone transfer image, and outputs a tone transfer result image. Obtain the outline of the foreground region in the original tone-transfer image to obtain a foreground mask image; paste the region marked by the foreground mask image in the tone-transfer result image back to the corresponding position in the original tone-transfer image to obtain a composite image where the foreground and background are not blended. Obtain a composite image of the foreground and background that do not blend, the original image of tone transfer, the foreground mask image, and the skin region mask image that have matching correspondences. Specify the correspondences of the four sets of images through the configuration file. Use the four sets of images as training data and iteratively train the pre-built convolutional neural network model by fusing a complex lighting supplement dataset until the convolutional neural network model converges to obtain the color fusion model.

[0008] Preferably, The training of the tone transfer model includes: Obtain the original tone transfer image, the original tone reference image, and the tone transfer result image with matching correspondences. Specify the correspondence between the three sets of images through the configuration file to obtain tone transfer training data. The pre-built convolutional neural network model is iteratively trained using the tone transfer training data until the convolutional neural network model converges, thus obtaining the tone transfer model.

[0009] Preferably, The training of the detail-preserving color transfer model includes: Obtain the original image to be color fused, the color fused result image, and the color adjustment result image with matching correspondences. Specify the correspondences of the three sets of images through the configuration file to obtain detail-preserving color transfer training data. The pre-built convolutional neural network model is iteratively trained using the detail-preserving color transfer training data and supplementary datasets of diverse scenarios until the convolutional neural network model converges, thus obtaining the detail-preserving color transfer model.

[0010] Preferably, The foreground and background non-blended composite image is obtained through image linear blending calculation. The foreground and background non-blended composite image = tone transfer result image × alpha × foreground mask image + tone transfer original image × (1 - alpha × foreground mask image); where alpha is the intensity control parameter for tone feature application. The larger the alpha is, the less blended the foreground and background are in the composite image.

[0011] Preferably, The pre-trained tone transfer model extracts the overall tone features of the tone reference image, including: One or more of color temperature, hue distribution, and brightness range.

[0012] Preferably, The foreground mask image is obtained through a foreground segmentation model, and the skin region mask image is obtained through a skin segmentation model. The foreground segmentation model and skin segmentation model employ a semantic segmentation neural network model.

[0013] According to a second aspect of the present invention, an image color fusion apparatus based on a convolutional neural network is provided, the apparatus comprising: Image acquisition module: used to acquire the original image to be color-blended uploaded by the user, wherein the original image to be color-blended is the original image of the foreground and background that has not been blended; Image labeling module: used to label the areas in the original image to be color-blended that require color fusion processing, to obtain a color fusion area mask image; and to label the skin areas in the original image to be color-blended, to obtain a skin area mask image; Image input module: used to input the original image to be fused, the color fusion region mask image, and the skin region mask image into the pre-trained color fusion model; Color fusion module: This module is used by the pre-trained color fusion model to perform downsampling and low-resolution inference on each input image. Using the color fusion region mask image and the skin region mask image as dual constraints, it extracts color features from the foreground, background, and skin tone in the original image to be fused. Based on the extracted color features, it adjusts the foreground color to blend with the background, outputting a color fusion result image. The resolution of the color fusion result image is lower than that of the original image to be fused. Detail-preserving color transfer module: This module takes the original image to be color-blended and the resulting color-blended image as inputs to a pre-trained detail-preserving color transfer model. The pre-trained model adjusts the colors of the original image to match those of the resulting color-blended image. Figure 1 The color grading result image is output while preserving the details of the original image to be fused with the colors. The resolution of the color grading result image is greater than that of the color fusion result image.

[0014] According to a third aspect of the present invention, a storage medium is provided, the storage medium storing a computer program, which, when executed by a host controller, implements the steps of the above-described method.

[0015] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: This application achieves foreground and background color adaptation under the dual constraints of foreground mask and skin segmentation mask through a color fusion model. It specifically optimizes the color naturalness of the skin region, effectively solving the problems of stiff fusion and strange skin color in existing technologies. Furthermore, by combining training with a supplementary dataset for complex lighting, it can adapt to various complex lighting scenarios such as backlighting and multiple light sources, improving the stability of the method. It adopts a high-resolution processing strategy of "low-resolution inference + detail-preserving color transfer", which avoids the problems of high memory consumption and long time consumption of direct high-resolution inference, and also solves the problems of loss of global color information by block inference and edge anomalies introduced by traditional high-frequency information recovery. The detail-preserving color transfer model can accurately extract the color features of the low-resolution fusion result and completely preserve the texture details of the high-resolution original image, achieving the dual goals of color fusion and detail preservation. This application achieves fully automated processing, which can adapt to various image compositing scenarios such as portrait background replacement, product scene implantation, and film and television post-production. It is compatible with image fusion needs of different resolutions and types, and can significantly reduce time and labor costs and improve processing efficiency in large-scale image processing scenarios.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0018] Figure 1 This is a flowchart illustrating an image color fusion method based on a convolutional neural network according to an exemplary embodiment; Figure 2 This is a system schematic diagram of an image color fusion apparatus based on a convolutional neural network, according to another exemplary embodiment; In the attached diagram: 1-Image acquisition module, 2-Image marking module, 3-Image input module, 4-Color fusion module, 5-Detail preservation color transfer module. Detailed Implementation

[0019] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0020] Example 1 Figure 1This is a flowchart illustrating an image color fusion method based on a convolutional neural network according to an exemplary embodiment, such as... Figure 1 As shown, the method includes: S1, Obtain the original image to be fused by color uploaded by the user, wherein the original image to be fused by color is the original image of the foreground and background that has not been fused; S2, mark the areas in the original image to be color-blended that require color blending processing to obtain a color blending area mask image; mark the skin areas in the original image to be color-blended to obtain a skin area mask image; S3, input the original image to be fused, the color fusion region mask image, and the skin region mask image into the pre-trained color fusion model; S4, the pre-trained color fusion model, under the condition of downsampling and low-resolution inference of each input image, uses the color fusion region mask image and the skin region mask image as dual constraints. It extracts the color features of the foreground, background and skin color in the original image to be fused, adjusts the foreground color based on the extracted color features to blend it with the background, and outputs the color fusion result image; the resolution of the color fusion result image is lower than that of the original image to be fused. S5, input the original image to be fused and the color fusion result image into the pre-trained detail-preserving color transfer model. The pre-trained detail-preserving color transfer model adjusts the colors of the original image to be fused to match the colors of the color fusion result. Figure 1 The original image to be fused with color is rendered in a color-corrected image, while retaining its details. The color-corrected image has a higher resolution than the color-fused image. It is understandable that the high-resolution original image to be color-blended uploaded by the user is the original image of the foreground and background that has not been blended. Mark the skin regions in the high-resolution original image to be color fused to assist in targeted optimization of the skin region fusion effect. The skin region mask map is generated by inference from the skin segmentation model. Mark the areas in the high-resolution original image that need color fusion processing to be processed, and obtain the color fusion area mask image; The high-resolution original image to be fused, the color fusion region mask image, and the skin region mask image are input into the pre-trained color fusion model. Under the condition of downsampling and low-resolution inference of each input image, the color fusion model uses the color fusion region mask image and the skin region mask image as dual constraints. By extracting the color features of the foreground, background, and skin color in the high-resolution original image to be fused, the foreground is color-adjusted based on the extracted features to blend with the background, while ensuring that the skin color is natural and beautiful, and finally a low-resolution color fusion result image is obtained. The high-resolution original image to be color-fused and the low-resolution color-fused result image are input into a pre-trained detail-preserving color transfer model. The detail-preserving color transfer model adjusts the colors of the high-resolution original image to match those of the low-resolution color-fused result. Figure 1 It achieves color matching and detail preservation while retaining the details of the original high-resolution image to be color-blended, thus outputting a high-resolution color-grading result image. The following uses specific data to illustrate the model training process, including: Taking the image synthesis scenario of replacing the background of a portrait as an example, this embodiment provides a detailed explanation of the image color fusion method based on convolutional neural networks. In this scenario, the foreground of the cut-out portrait needs to be color-fused with the new background to ensure that the tones and lighting of the portrait and the background are harmonious, while preserving details such as the skin texture and hair of the portrait. In this embodiment, the alpha parameter is set to 0.9, the downsampling ratio is 1 / 4, and the training convergence index of all convolutional neural networks is that the loss value is less than 0.01 and there is no significant decrease for 10 consecutive epochs.

[0021] Preprocessing preparation for the example: Building an image library: Collecting image data containing different portraits, backgrounds, and lighting scenarios (front lighting, backlighting, side lighting, and multiple light sources), with resolutions ranging from 512×512 to 4096×4096, divided into training and testing sets, with the training set accounting for 80% and the testing set accounting for 20%. Preparing the segmentation model: The existing semantic segmentation model is selected as the foreground segmentation model and the skin segmentation model, which can accurately segment the portrait foreground and skin regions in the image and generate the corresponding binarized mask image; Configure the training data mapping file: Number all training data into groups, generate a mapping file in txt format, and specify the one-to-one matching relationship between the images in each group.

[0022] Training the tone transfer model: Select training data: Randomly select 100,000 sets of matching tone transfer original images, tone reference original images, and tone transfer result images (transfer the tone of the tone reference original image to the standard image of the tone transfer original image) from the image library training set to construct tone transfer original image group, tone reference original image group, and tone transfer result image group. Model training: Input the above three sets of data into the initialized convolutional neural network, set the learning rate to 0.0001 and the batch size to 16, and train until the loss value is lower than 0.01 and there is no significant decrease for 10 consecutive epochs, thus obtaining a converged tone transfer model; This model can automatically extract global and local tone features such as color temperature, tone distribution, and brightness range of the tone reference image, and perform differentiated color mapping on the portrait area and background area of ​​the tone transfer image.

[0023] Generate training data for the color fusion model: Input acquisition: Randomly select the tone transfer original image from the above training set, and input it and the corresponding tone reference original image into the trained and converged tone transfer model to obtain the tone transfer result image; use the foreground segmentation model to perform segmentation inference on the tone transfer original image to generate a foreground mask image. Linear blending calculation: The foreground and background non-blended composite image is calculated according to the formula = tone transfer result image × 0.9 × foreground mask image + tone transfer original image × (1 - 0.9 × foreground mask image). In this embodiment, alpha is set to 0.9, so that the foreground and background of the composite image show obvious color discontinuity, providing effective samples for color fusion model training. Data set construction: Repeat the above operations to generate 100,000 composite images with no blending between foreground and background. Combine these with the original image set with foreground and background fusion (i.e., the original image set for tone transfer), the foreground mask image set, and the skin segmentation mask image set to construct four sets of training data for the color fusion model.

[0024] Training the color fusion model: Training data supplementation: The four sets of training data constructed were fused with the complex lighting supplementary dataset (containing 20,000 sets of image data of complex lighting scenes such as backlighting and multiple light sources) to obtain a total of 120,000 sets of training data. Model Training: The fused training data is input into the initialized convolutional neural network with a learning rate of 0.0002 and a batch size of 8. The model is trained until the loss value is below 0.01 and there is no significant decrease for 10 consecutive epochs, resulting in a converged color fusion model. Under the dual constraints of foreground mask and skin segmentation mask, the model can perform color adaptation on the stitched image where the foreground and background are not fused, and specifically optimize the color naturalness of the skin area of ​​the portrait to avoid the problem of strange skin color.

[0025] Training a detail-preserving color transfer model: Training data selection: 100,000 sets of matching high-resolution original images to be color fused (4096×4096 unfused portrait background replacement images), low-resolution color grading result images (color fusion standard images with high-resolution images downsampled to 1024×1024), and high-resolution color grading result label images (high-resolution standard images with color fusion and complete details) were selected from the image library training set. Training data supplementation: The above three sets of data are merged with a diverse scene supplementary dataset (containing 20,000 sets of image data from different scenes such as indoor, outdoor, night scene, and daytime) to obtain a total of 120,000 sets of training data; Model training: The fused training data is input into the initialized convolutional neural network with a learning rate of 0.0001 and a batch size of 4. The model is trained until the loss value is below 0.01 and there is no significant decrease after 10 consecutive epochs, resulting in a converged detail-preserving color transfer model. This model can extract the color features of the low-resolution color-corrected image while fully preserving the texture details (such as hair strands, skin texture, and clothing wrinkles) of the high-resolution original image to be fused.

[0026] Image color fusion reasoning: In this embodiment, a 4096×4096 resolution unblended image of a portrait with a background swapped from the test set is selected as the high-resolution original image to be color-blended. This image is an unblended image of a portrait cut out and pasted onto a night scene background. There are obvious differences in tone and brightness between the portrait and the night scene background. The specific reasoning steps are as follows: Generate mask image: Use a foreground segmentation model to segment the high-resolution original image to be color fused and generate a color fusion region mask image; use a skin segmentation model to generate a skin region mask image; Downsampling low-resolution inference: Downsample the high-resolution original image to be color-blended to 1024×1024 (1 / 4 scale), and combine it with the color blending area mask image and the skin area mask. Figure 1 The color fusion model, which is trained to convergence with the same input, performs differential color adjustment on the foreground of the portrait under the constraint of double mask, adapts to the tone and light and shadow of the night scene background, and outputs a low-resolution color fusion result image of 1024×1024. In this image, the colors of the portrait and the background are naturally blended, and the skin tone is normal. Detail-preserving color transfer: Input a high-resolution original image of 4096×4096 to be color-fused and a low-resolution color-fused result image of 1024×1024 into the training convergent detail-preserving color transfer model. The model extracts the color features of the low-resolution fusion result and adjusts the color of the high-resolution original image to match it, while fully preserving the details of the high-resolution original image such as hair strands and skin texture. Finally, it outputs a high-resolution color-fused result image of 4096×4096. Results Verification: In the final high-resolution color fusion result image, the tones and light and shadow depths of the portrait foreground and the night scene background are well matched, without any harsh fusion or strange skin color issues. Moreover, details such as the hair and skin texture of the portrait are completely preserved, achieving a natural image fusion effect.

[0027] This embodiment uses a tone transfer model trained with a large amount of tone transfer color grading data based on reference images. This model can accurately capture the color and lighting features of the reference images and can implement differentiated color mapping for different local areas, rather than a global homogenization effect. This ensures that the color style of the color-graded image is highly consistent with that of the reference image, thus constructing high-quality training data for the image fusion model. It is also an efficient color grading scheme for transferring the color style of reference images, solving problems such as high time cost of manual color grading, reduced contrast caused by traditional global uniform color grading, and loss of details in bright and dark areas. The color fusion model, trained with a large amount of training data containing foreground and skin dual mask constraints, can perform targeted foreground and background color adaptation under the localization of foreground and skin masks, while optimizing the naturalness of skin color. It can achieve differentiated color mapping for different local areas of the foreground, rather than a global homogenization effect. This effectively solves the problems of stiff fusion and strange skin tone after foreground and background color adjustment, and achieves a harmonious visual effect. Compared with the complex parameter adjustment process required by traditional global color adjustment or manual fine color adjustment technology, this invention can significantly reduce the technical threshold of color fusion, improve the efficiency of image color adjustment processing, and at the same time ensure the precision and naturalness of the color adjustment result. The detail-preserving color transfer model, trained with a large amount of high / low resolution matched color grading data, can extract color features from the low-resolution fusion result and achieve accurate color matching while fully preserving the texture details of the high-resolution original image. It balances color grading effect and image detail integrity, and solves the following shortcomings of existing high-resolution image color grading techniques: The disadvantage of using high-resolution images directly for model inference is that it consumes a lot of video memory and takes a long time. Dividing a high-resolution image into multiple smaller images for inference has the disadvantage of losing global color style information, resulting in poor color fusion. Using low-resolution image inference, combined with traditional algorithms such as high-contrast preservation to restore high-frequency information, can improve resolution. However, when there is a large gradient change at the boundary between the foreground and background before and after color fusion, it can introduce incorrect gradient information before fusion, resulting in abnormal black / white edges at the foreground edges.

[0028] Example 2 Figure 2 This is a system schematic diagram of an image color fusion apparatus based on a convolutional neural network according to another exemplary embodiment, the apparatus comprising: Image acquisition module 1: used to acquire the original image to be color-fused uploaded by the user, wherein the original image to be color-fused is the original image of the foreground and background that has not been fused; Image labeling module 2: used to label the areas in the original image to be color-blended that require color fusion processing, to obtain a color fusion area mask image; and to label the skin areas in the original image to be color-blended, to obtain a skin area mask image; Image input module 3: used to input the original image to be fused, the color fusion region mask image, and the skin region mask image into the pre-trained color fusion model; Color fusion module 4: This module is used by the pre-trained color fusion model to perform downsampling and low-resolution inference on each input image. Using the color fusion region mask image and the skin region mask image as dual constraints, it extracts color features from the foreground, background, and skin tone in the original image to be fused. Based on the extracted color features, it adjusts the foreground color to blend with the background, outputting a color fusion result image. The resolution of the color fusion result image is lower than that of the original image to be fused. Detail-preserving color transfer module 5: This module inputs the original image to be color-blended and the resulting color-blended image into a pre-trained detail-preserving color transfer model. The pre-trained model adjusts the colors of the original image to match those of the resulting color-blended image. Figure 1 The color grading result image is output while preserving the details of the original image to be fused with the colors. The resolution of the color grading result image is greater than that of the color fusion result image.

[0029] Example 3 This embodiment provides a storage medium storing a computer program, which, when executed by a host controller, implements the various steps in the above method. It is understood that the storage medium mentioned above can be a read-only memory, a hard disk, or an optical disk, etc.

[0030] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0031] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0032] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0033] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0034] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0035] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0036] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0037] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0038] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An image color fusion method based on convolutional neural networks, characterized in that, The method includes: Obtain the original image to be color-blended uploaded by the user, wherein the original image to be color-blended is the original image of the foreground and background that has not been blended; Mark the areas in the original image to be color-blended that require color blending processing to obtain a color blending area mask image; mark the skin areas in the original image to be color-blended to obtain a skin area mask image; The original image to be fused, the color fusion region mask image, and the skin region mask image are input into the pre-trained color fusion model; The pre-trained color fusion model, under the condition of downsampling and low-resolution inference of each input image, uses the color fusion region mask image and the skin region mask image as dual constraints. It extracts the color features of the foreground, background and skin color in the original image to be fused, adjusts the foreground color based on the extracted color features to blend it with the background, and outputs the color fusion result image; the resolution of the color fusion result image is lower than that of the original image to be fused. The original image to be fused and the resulting color fusion image are input into a pre-trained detail-preserving color transfer model. The pre-trained detail-preserving color transfer model adjusts the color of the original image to be fused to match the color fusion result image while preserving the details of the original image and outputs a color adjustment result image. The resolution of the color adjustment result image is greater than that of the color fusion result image.

2. The method according to claim 1, characterized in that, The training of the color fusion model includes: Randomly select a tone transfer original image and a tone reference original image. The tone transfer original image is the target image to be tone adjusted, and the tone reference original image is a reference image that provides a tone benchmark. The original tone transfer image and the original tone reference image are input into a pre-trained tone transfer model. The pre-trained tone transfer model extracts the overall tone features of the original tone reference image and performs differentiated color mapping on each local area of ​​the original tone transfer image, and outputs a tone transfer result image. Obtain the outline of the foreground region in the original tone-transfer image to obtain a foreground mask image; paste the region marked by the foreground mask image in the tone-transfer result image back to the corresponding position in the original tone-transfer image to obtain a composite image where the foreground and background are not blended. Obtain a composite image of the foreground and background that do not blend, the original image of tone transfer, the foreground mask image, and the skin region mask image that have matching correspondences. Specify the correspondences of the four sets of images through the configuration file. Use the four sets of images as training data and iteratively train the pre-built convolutional neural network model by fusing a complex lighting supplement dataset until the convolutional neural network model converges to obtain the color fusion model.

3. The method according to claim 2, characterized in that, The training of the tone transfer model includes: Obtain the original tone transfer image, the original tone reference image, and the tone transfer result image with matching correspondences. Specify the correspondence between the three sets of images through the configuration file to obtain tone transfer training data. The pre-built convolutional neural network model is iteratively trained using the tone transfer training data until the convolutional neural network model converges, thus obtaining the tone transfer model.

4. The method according to claim 3, characterized in that, The training of the detail-preserving color transfer model includes: Obtain the original image to be color fused, the color fused result image, and the color adjustment result image with matching correspondences. Specify the correspondences of the three sets of images through the configuration file to obtain detail-preserving color transfer training data. The pre-built convolutional neural network model is iteratively trained using the detail-preserving color transfer training data and supplementary datasets of diverse scenarios until the convolutional neural network model converges, thus obtaining the detail-preserving color transfer model.

5. The method according to claim 2, characterized in that, The foreground and background non-blended composite image is obtained through image linear blending calculation. The foreground and background non-blended composite image = tone transfer result image × alpha × foreground mask image + tone transfer original image × (1 - alpha × foreground mask image); where alpha is the intensity control parameter for tone feature application. The larger the alpha is, the less blended the foreground and background are in the composite image.

6. The method according to claim 5, characterized in that, The pre-trained tone transfer model extracts the overall tone features of the tone reference image, including: One or more of color temperature, hue distribution, and brightness range.

7. The method according to claim 6, characterized in that, The foreground mask image is obtained through a foreground segmentation model, and the skin region mask image is obtained through a skin segmentation model. The foreground segmentation model and skin segmentation model employ a semantic segmentation neural network model.

8. An image color fusion device based on a convolutional neural network, characterized in that, The device includes: Image acquisition module: used to acquire the original image to be color-blended uploaded by the user, wherein the original image to be color-blended is the original image of the foreground and background that has not been blended; Image labeling module: used to label the areas in the original image to be color-blended that require color fusion processing, to obtain a color fusion area mask image; and to label the skin areas in the original image to be color-blended, to obtain a skin area mask image; Image input module: used to input the original image to be fused, the color fusion region mask image, and the skin region mask image into the pre-trained color fusion model; Color fusion module: This module is used by the pre-trained color fusion model to perform downsampling and low-resolution inference on each input image. Using the color fusion region mask image and the skin region mask image as dual constraints, it extracts color features from the foreground, background, and skin tone in the original image to be fused. Based on the extracted color features, it adjusts the foreground color to blend with the background, outputting a color fusion result image. The resolution of the color fusion result image is lower than that of the original image to be fused. Detail-preserving color transfer module: This module is used to input the original image to be color-fused and the color-fused result image into a pre-trained detail-preserving color transfer model. The pre-trained detail-preserving color transfer model adjusts the color of the original image to be color-fused to match the color-fused result image while preserving the details of the original image and outputs a color-adjusted result image. The resolution of the color-adjusted result image is greater than that of the color-fused result image.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by the main controller, implements each step of the image color fusion method based on a convolutional neural network as described in any one of claims 1-7.

Images