Migration model training method and device, migration method and device and electronic equipment

Abstract

The invention discloses a migration model training method and device, a migration method and device, electronic equipment and a computer program product. The method comprises the following steps: acquiring an image training set including style images and content images; inputting the style image and the content image into a to-be-trained migration model to obtain a migration image; inputting the style image, the content image and the migration image into a preset image feature extraction module to obtain style features, content features and migration features; calculating the model loss of a to-be-trained migration model based on the style features, the content features and the migration features; and optimizing the to-be-trained migration model according to the model loss until the model loss is converged to obtain a trained migration model which can be used for executing image color style migration processing. According to the scheme, the migration model with relatively high migration efficiency and relatively good migration effect can be obtained, so that the accurate, natural and controllable image color migration effect of various images can be realized through the migration model.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a migration model training method, a migration model training device, a migration method, a migration device, an electronic device, and a computer program product. Background Art

[0002] With the continuous development of digital image processing technology, image color transfer has shown widespread application demand in multiple fields. For example, in film and television post-production, image color transfer can quickly adjust the film's color tone to create a specific atmosphere and style. For another example, in object detection, image color transfer can generate data for difficult scenes without labeling costs, thereby enriching the dataset and optimizing model performance.

[0003] Currently, traditional image color transfer methods, such as those based on statistical features, mostly focus on simple statistical analysis of image color, directly applying statistics such as the source image's color mean and variance to the target image. While this approach can achieve basic color transfer effects in some simple scenarios, it still has significant limitations when it comes to complex images and diverse color requirements. Summary of the Invention

[0004] The present application provides a migration model training method, a migration model training device, a migration method, a migration device, an electronic device and a computer program product, which can obtain a migration model with high migration efficiency and good migration effect, so that accurate, natural and controllable image color migration effects can be achieved for various types of images through the migration model.

[0005] In a first aspect, the present application provides a migration model training method, comprising:

[0006] Obtain an image training set, which includes style images and content images;

[0007] Inputting the style image and the content image into the transfer model to be trained to obtain a transfer image output by the transfer model to be trained;

[0008] Input the style image, content image, and transfer image into a preset image feature extraction module to obtain the style features, content features, and transfer features output by the image feature extraction module, wherein the style features are image features of the style image, the content features are image features of the content image, and the transfer features are image features of the transfer image;

[0009] Calculate the model loss of the transfer model to be trained based on style features, content features, and transfer features;

[0010] The transfer model to be trained is optimized according to the model loss until the model loss has converged, thereby obtaining a trained transfer model. The trained transfer model is used to perform image color style transfer processing.

[0011] In a second aspect, the present application provides a migration method, comprising:

[0012] Determine the target style image and target content image;

[0013] The target style image and the target content image are input into the trained transfer model to obtain the target transfer image output by the trained transfer model, wherein the trained transfer model includes: a reversible neural flow module and a feature transfer module, the reversible neural flow module is used to extract the target style flow features of the target style image and the target content flow features of the target content image, the feature transfer module is used to fuse the target style flow features and the target content flow features to obtain the target transfer flow features, and the reversible neural flow module is also used to reconstruct the target transfer flow features to obtain the target transfer image.

[0014] In a third aspect, the present application provides a migration model training device, comprising:

[0015] An acquisition unit, configured to acquire an image training set, the image training set including: a style image and a content image;

[0016] a first processing unit, configured to input the style image and the content image into a transfer model to be trained, and obtain a transfer image output by the transfer model to be trained;

[0017] a second processing unit, configured to input the style image, the content image, and the transfer image into a preset image feature extraction module, and obtain style features, content features, and transfer features output by the image feature extraction module, wherein the style features are image features of the style image, the content features are image features of the content image, and the transfer features are image features of the transfer image;

[0018] A calculation unit, used to calculate the model loss of the transfer model to be trained based on the style features, content features, and transfer features;

[0019] The optimization unit is used to optimize the transfer model to be trained according to the model loss until the model loss has converged, thereby obtaining a trained transfer model. The trained transfer model is used to perform image color style transfer processing.

[0020] In a fourth aspect, the present application provides a migration device, comprising:

[0021] a determination unit, configured to determine a target style image and a target content image;

[0022] A migration unit is used to input the target style image and the target content image into the trained migration model to obtain the target migration image output by the trained migration model, wherein the trained migration model includes: a reversible neural flow module and a feature migration module, the reversible neural flow module is used to extract the target style flow features of the target style image and the target content flow features of the target content image, the feature migration module is used to fuse the target style flow features and the target content flow features to obtain the target migration flow features, and the reversible neural flow module is also used to reconstruct the target migration flow features to obtain the target migration image.

[0023] In a fifth aspect, the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method of the first aspect and / or the second aspect are implemented.

[0024] In a sixth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method of the first aspect and / or the second aspect are implemented.

[0025] In a seventh aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it implements the steps of the method of the first aspect and / or the second aspect.

[0026] Compared with the prior art, the beneficial effect of the present application is that the migration model constructed by the present application includes a reversible neural flow module and a feature migration module. The reversible neural flow module has a unique reversible transformation characteristic, which can establish an accurate bidirectional mapping between the image space and the latent space. It can not only efficiently learn the color distribution characteristics of the image, but also ensure the lossless transmission of information during the color migration process. Based on the above structure of the migration model, when training the migration model, the style image and the content image can be processed by the migration model first to obtain the migration image, and then the style features, content features and migration features corresponding to the above three types of images are obtained by the image feature extraction module, so as to calculate the model loss of the migration model to be trained, and optimize the migration model to be trained according to the model loss until the model loss has converged, and then the trained migration model can be obtained. The above training process can help obtain a migration model with high migration efficiency and good migration effect, so that the migration model can achieve accurate, natural and controllable image color migration effects for various images.

[0027] It can be understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0029] Figure 1 This is a schematic diagram of the implementation process of the migration model training method provided in the embodiment of the present application;

[0030] Figure 2 This is an example diagram of the training process of the migration model provided in the embodiment of the present application;

[0031] Figure 3 This is an example diagram of the architecture of the migration model provided in the embodiment of the present application;

[0032] Figure 4 This is a schematic diagram of the implementation process of the migration method provided in the embodiment of the present application;

[0033] Figure 5 This is a structural block diagram of the migration model training device provided in an embodiment of the present application;

[0034] Figure 6 This is a structural block diagram of the migration device provided in an embodiment of the present application;

[0035] Figure 7 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The following embodiments of the technical solution of the present application will be described in detail with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application and are therefore only examples and are not intended to limit the scope of protection of the present application.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned figure descriptions are intended to cover non-exclusive inclusions.

[0038] In the description of the embodiments of the present application, technical terms such as "first" and "second" are only used to distinguish different objects and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features.

[0039] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0040] In the description of the embodiments of this application, the term "and / or" is simply a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0041] In the description of the embodiments of the present application, the term "plurality" refers to two or more (including two), unless otherwise clearly and specifically defined.

[0042] With the continuous development of digital image processing technology, image color transfer has shown widespread application demand in multiple fields. For example, in film and television post-production, image color transfer can quickly adjust the film's color tone to create a specific atmosphere and style. For another example, in object detection, image color transfer can generate data for difficult scenes without labeling costs, thereby enriching the dataset and optimizing model performance.

[0043] Currently, traditional image color transfer methods, such as those based on statistical features, mostly focus on simple statistical analysis of image color, directly applying statistics such as the source image's color mean and variance to the target image. While this approach can achieve basic color transfer effects in some simple scenarios, it still has significant limitations when faced with complex images and diverse color requirements. On the one hand, relying solely on simple statistical features cannot accurately capture the rich color information and complex semantic structure of the image, resulting in color distortion in the transferred image. On the other hand, traditional methods struggle to adapt to the complex characteristics of different image types. When processing images with unique textures, lighting conditions, or scene content, problems such as unnatural color transitions and unsatisfactory results often occur.

[0044] Based on this, the embodiments of this application use a neural network structure to construct a transfer model and propose a transfer model training method. This method can obtain a transfer model with high transfer efficiency and good transfer effect, thereby achieving accurate, natural, and controllable image color transfer effects for various images. The following is an explanation and description of the specific embodiments.

[0045] The present application embodiment proposes a migration model training method. The migration model training method can be applied to electronic devices that have data processing capabilities. Figure 1 , Figure 1 The implementation process of the transfer model training method applied to the electronic device is given, and the details are as follows:

[0046] Step 101: Obtain an image training set.

[0047] Before formally training the migration model, the electronic device may first obtain an image sample set; wherein, the image sample set includes style images and content images. Style images refer to images with the color style expected by the user; content images refer to images other than style images. The electronic device may divide the image sample set to obtain an image training set, an image test set, and an image verification set. It can be understood that since the image training set, the image test set, and the image verification set are all subsets of the image sample set, in general, the image training set, the image test set, and the image verification set also include style images and content images. For the embodiments of the present application, the main focus is on the training process of the model, and thus the subsequent training of the migration model to be trained can be carried out based on the image training set.

[0048] In step 102 , the style image and the content image are input into the transfer model to be trained to obtain a transfer image output by the transfer model to be trained.

[0049] The electronic device can construct a migration model to be trained, which includes the following two modules: a reversible neural flow module and a feature migration module.

[0050] The reversible neural flow module has a unique reversible transformation property that enables it to establish a precise bidirectional mapping between the image space and the latent space. This not only allows for efficient learning of the image's color distribution characteristics, but also ensures lossless information transmission during color migration. It can be understood that for a reversible neural flow module, if its input is an image, the corresponding output can be the image's flow features; conversely, if its input is a flow feature, the corresponding output can be an image with that flow feature.

[0051] Among them, the feature migration module can fuse different features at the feature level.

[0052] During each training session, the electronic device randomly selects a style image and a content image from the image training set and inputs these two images into the transfer model to be trained. By processing these two images, the transfer model learns the color style of the style image and the image content of the content image, thereby generating a transfer image. This results in three images: the style image, the content image, and the transfer image. Users expect the image content of the transfer image to be as consistent as possible with that of the content image, and the color style of the transfer image to be as consistent as possible with that of the content image.

[0053] Step 103: Input the style image, content image, and transfer image into a preset image feature extraction module to obtain style features, content features, and transfer features output by the image feature extraction module.

[0054] In addition to the image feature extraction module to be trained, the electronic device can also construct an image feature extraction module. This image feature extraction module has been pre-trained and only participates in the training process of the migration model, but not in the application process of the migration model. The electronic device can use this image feature extraction module to extract image features from the style image, content image, and migration image, respectively, thereby obtaining image features of the style image, image features of the content image, and image features of the migration image. It is understood that this image feature can be used to express information such as the color, texture, brushstrokes, and semantics of the corresponding image, which is not limited here.

[0055] For ease of explanation, the image features of the style image are referred to as style features, the image features of the content image are referred to as content features, and the image features of the transfer image are referred to as transfer features.

[0056] Step 104 : Calculate the model loss of the transfer model to be trained based on the style features, content features, and transfer features.

[0057] The electronic device may calculate the model loss of the transfer model to be trained using a preset loss calculation formula based on the obtained style features, content features, and transfer features. In some embodiments, because the user desires that the image content of the transfer image be as consistent as possible with that of the content image, and that the color style of the transfer image be as consistent as possible with that of the content image, the model loss may specifically be composed of two components: content loss and style loss.

[0058] Step 105 : Optimize the migration model to be trained according to the model loss until the model loss converges, thereby obtaining a trained migration model.

[0059] The electronic device can perform back propagation based on the calculated model loss, thereby updating the model parameters of the migration model to be trained, and optimizing the migration model to be trained. It should be noted that during this optimization process, the electronic device will not update the image feature extraction module, that is, the relevant parameters of the image feature extraction module are fixed. After the optimization is completed, the electronic device can return to execute step 102 and subsequent steps until the final model loss has converged, and the preliminary training can be considered to be completed, and the trained migration model can be obtained. The electronic device can also continue to use the image test set and the image verification set to test and verify the trained migration model, which will not be repeated here. Finally, the trained migration model that has passed the test and verification can be put into use and can perform image color style transfer processing.

[0060] Based on steps 101 to 105 above, please refer to Figure 2 , Figure 2 An example diagram of the training process of the migration model is given.

[0061] In some embodiments, taking the model training scenario as an example, the working process of the migration model constructed by the electronic device is as follows:

[0062] A1. Extract the style flow features of the style image through the reversible neural flow module, and extract the content flow features of the content image.

[0063] In the model training scenario, after the electronic device sequentially inputs the style image and content image into the transfer model, the reversible neural flow module will first process the style image and the content image. The basic functions of the reversible neural flow module have been described above. When the input is an image, the reversible neural flow module can output the flow features corresponding to the image. Therefore, for the style image, the reversible neural flow module can output the flow features of the style image, denoted as style flow features. For the content image, the reversible neural flow module can output the flow features of the content image, denoted as content flow features.

[0064] The reversible neural flow module is described in detail below: The reversible neural flow module can be composed of multiple mapping modules. As an example only, the number of mapping modules can be 8. Of course, the number of mapping modules can also be increased or decreased according to needs, which is not limited here. Among them, the mapping module consists of three parts: activation normalization layer, 1×1 reversible convolution layer, and additive coupling layer. Among them, the activation normalization layer is used to stabilize the distribution and introduce nonlinearity, laying the foundation for subsequent transformations; the 1×1 reversible convolution layer is used to mix channel information, break the independence between dimensions, and enhance the flexibility of transformation; the additive coupling layer achieves local reversibility through block transformation, thereby forming an efficient and reversible neural flow module; the above three (i.e., activation normalization layer, 1×1 reversible convolution layer, additive coupling layer) jointly ensure the reversibility of the reversible neural flow module, and enable the reversible neural flow module to have nonlinearity and channel interaction to improve data fitting capabilities.

[0065] The formula used to activate the normalization layer is as follows:

[0066] y i,j =w*x i,j +b

[0067] In the above formula, w is the scale of feature transformation and b is the bias parameter. Both can be learned in the model training scenario, that is, they are updated with the gradient during the round of training. i,j is the specific value of the pixel point of the image, y i,j is the output after the pixel is transformed.

[0068] The formula used for the additive coupling layer is as follows:

[0069] x a ,x b =split(x)

[0070] y b =Conv(x a )+x b

[0071] y=concat(x a +y b )

[0072] In the above formula, split() means splitting the features by channel dimension; Conv() means two convolutional layers with the same input and output dimensions; concat() means concatenating the features by channel dimension; x, x a 、x b and y b are all features of the neural network, where x is the original input feature, x a and x b are the features after segmentation, yb is x a After convolution with x b The added features, y is the final output feature.

[0073] A2. The style flow features and content flow features are integrated through the feature transfer module to obtain the transfer flow features.

[0074] The formula used by the feature transfer module is as follows:

[0075] style c =norm c *σ s +μ s

[0076] norm c =(x c -μ c ) / σ c

[0077] Among them, x c is the content flow feature, μ c is the mean of the content flow features, σ c is the standard deviation of content flow features, μ s is the mean of the style flow features, σ s is the standard deviation of the style flow feature, style c It is the migration flow feature obtained by fusing the style flow feature and the content flow feature.

[0078] A3. Reconstruct the migration flow features through the reversible neural flow module to obtain the migration image.

[0079] The basic functions of the reversible neural flow module have been described above. Given a flow feature as input, the module can output an image with that flow feature. Based on this, the electronic device inputs the migration flow feature obtained in step A2 into the reversible neural flow module, which then performs the corresponding inverse processing on the migration flow feature to reconstruct a migration image. This migration image is an image that has the color style of the style image transferred from the content image.

[0080] See also Figure 3 , Figure 3 An example of the architecture of the migration model is given.

[0081] In some embodiments, the model loss can be calculated as follows:

[0082] B1. Based on the style features and transfer features, the style loss is calculated using a preset first loss function.

[0083] B2. Based on the content features and migration features, the content loss is calculated using a preset second loss function.

[0084] The first loss function and the second loss function can be different or the same, and this embodiment of the application does not limit this. As an example only, the first loss function and the second loss function can both be mean square error loss functions, which will not be described here.

[0085] If the image feature extraction module constructed by the electronic device is capable of outputting multi-scale features, the style features, content features, and transfer features obtained in the model training scenario can all be multi-scale features, and the style features, content features, and transfer features can correspond to the same multiple scales. For example, when the image feature extraction module uses the VGG model, the style features, content features, and transfer features all include feature maps at four scales, which will not be described in detail here.

[0086] On this basis, when calculating the style loss, the electronic device can specifically calculate the style loss by: calculating the style loss through the first loss function based on the style features and the transfer features at each scale; that is, the style loss between the style features and the transfer features needs to be calculated at all scales.

[0087] In addition, when calculating content loss, the electronic device may specifically calculate the style loss based on the content features and the transfer features using a second loss function at the minimum scale; that is, the content loss between the content features and the transfer features is only calculated at the minimum scale.

[0088] The reason why content loss and style loss consider different feature scales is that: generally speaking, the features of convolutional neural networks in the shallow layer are more biased towards pixel color blocks, while the features in the deep layer are more biased towards the meaning of the entire image; specifically in this application, content loss only plays an auxiliary role. In fact, the unbiased transfer of image content can be guaranteed in the feature transfer module, so it is only necessary to calculate the content loss at the minimum scale; while style features are hierarchical, so it is necessary to calculate the style loss at each scale.

[0089] B3. Calculate the model loss based on the style loss, the first weight corresponding to the style loss, the content loss, and the second weight corresponding to the content loss, where the first weight is greater than the second weight.

[0090] The electronic device can perform weighted processing on the style loss and content loss to obtain the final model loss. The weight corresponding to the style loss can be recorded as the first weight, and the weight corresponding to the content loss can be recorded as the second weight. As described above, the content loss only plays a supporting role, so the weight corresponding to the content loss can generally be set to a smaller value, that is, the first weight can generally be greater than the second weight. As an example only, the first weight can be 0.9 and the second weight can be 0.1.

[0091] In some embodiments, the electronic device may construct an image sample set in the following manner:

[0092] C1. Collect environmental images under different weather conditions and lighting conditions.

[0093] The electronic device can receive environmental images captured by cameras installed at different locations. As an example only, the camera can be a vehicle-mounted camera, and the cameras installed at different locations can specifically be cameras installed on different vehicles, which will not be described in detail here. To ensure the richness and diversity of the collected environmental images, these cameras can be controlled to collect images in different weather and different lighting conditions (i.e., at different time points), where different weather conditions can specifically include sunny days, rainy days, cloudy days, and foggy days, and different lighting conditions can specifically include daytime, nighttime, and evening, which are not limited here.

[0094] C2. Based on the preset color style filtering conditions, filter out the style image from the environment image.

[0095] The electronic device can be pre-set with corresponding color style filtering conditions based on the user's needs, so that the style images filtered from the environmental image based on the color style filtering conditions can include all the color styles that the user wants to generate. The color style filtering conditions can be entered in the form of a drop-down box selection, and this embodiment of the application is not limited to this.

[0096] C3. Determine the other images in the environment image except the style image as content images.

[0097] The electronic device can determine all images in the environmental image except the style image as content images; or, the electronic device can set corresponding image content screening conditions in advance based on the user's needs, and based on the image content screening conditions, screen out content images from the other images, so that the content images can cover as many scenes as possible, such as various scenes such as rural areas, cities and highways.

[0098] Based on the determined content images and style images, an image sample set can be constructed; then the image sample set is divided according to the specified ratio to obtain the image training set, image test set and image verification set, which will not be described in detail here.

[0099] As can be seen from the above, the migration model constructed in the embodiment of the present application includes a reversible neural flow module and a feature migration module. The reversible neural flow module has a unique reversible transformation characteristic, which can establish an accurate bidirectional mapping between the image space and the latent space. It can not only efficiently learn the color distribution characteristics of the image, but also ensure the lossless transmission of information during the color migration process. Based on the above structure of the migration model, when training the migration model, the style image and the content image can be processed by the migration model first to obtain the migration image, and then the style features, content features and migration features corresponding to the above three types of images are obtained by the image feature extraction module, so as to calculate the model loss of the migration model to be trained, and optimize the migration model to be trained according to the model loss. After the model loss has converged, the trained migration model can be obtained. The above training process can help obtain a migration model with high migration efficiency and good migration effect, so that the migration model can achieve accurate, natural and controllable image color migration effects for various images.

[0100] The embodiment of the present application also proposes a migration method. Among them, the migration method can be applied to an electronic device that has the ability to process data. It can be understood that the electronic device that executes the migration method can be the same as or different from the electronic device that executes the migration model training method, and this is not limited here. For example, the migration model training method can be executed by electronic device 1, and after obtaining the trained migration model, the migration model is deployed on electronic device 2, and the migration method is executed by electronic device 2; or, the migration model training method can be executed by electronic device 1, and after obtaining the trained migration model, the migration method is continued to be executed by electronic device 1, and it will not be repeated here.

[0101] See also Figure 4 , Figure 4 The implementation process of the transfer model training method applied to electronic devices is given, and the details are as follows:

[0102] Step 401: Determine a target style image and a target content image.

[0103] The target style image and the target content image are specifically images specified by the user. In some examples, the user can actively capture the desired style image and content image, and then load or transfer the style image and the content image to the electronic device, so that the electronic device determines the style image as the target style image and the content image as the target content image. Alternatively, the user can also input a style image selection instruction and a content image selection instruction in a preset media library, so that the electronic device can determine the style image indicated by the style image selection instruction as the target style image, and the content image indicated by the content image selection instruction as the target content image. Of course, the electronic device can also determine the target style image and the target content image in other ways, and the embodiments of the present application are not limited to this.

[0104] In step 402 , the target style image and the target content image are input into the trained transfer model to obtain a target transfer image output by the trained transfer model.

[0105] The trained migration model may include the following two modules: a reversible neural flow module and a feature migration module. Figure 3 , Figure 3 An example of the architecture of the migration model has been given, which has been described in the previous embodiment and will not be repeated here.

[0106] The following briefly describes the specific functions of the reversible neural flow module and the feature transfer module in the application stage of this migration model:

[0107] A reversible neural flow module for extracting target style flow features of a target style image and target content flow features of a target content image;

[0108] The feature transfer module is used to fuse the target style flow features and the target content flow features to obtain the target transfer flow features;

[0109] The reversible neural flow module is also used to reconstruct the target migration flow features to obtain the target migration image.

[0110] It can be understood that this trained transfer model is actually obtained by training the transfer model training method proposed in the previous embodiment, and the embodiments of this application will not be repeated here. When applied, the electronic device can regard the trained transfer model as a black box. It only needs to input according to the given requirements (specifically input the target style image and the target content image) to obtain the target transfer image output by the trained transfer model. The image content of the target transfer image is the image content of the target content image, and the image style of the target transfer image is the image style of the target style image.

[0111] As can be seen from the above, in the embodiments of the present application, the migration model used includes a reversible neural flow module and a feature migration module. The reversible neural flow module has a unique reversible transformation property, which can establish an accurate bidirectional mapping between the image space and the latent space. It can not only efficiently learn the color distribution characteristics of the image, but also ensure lossless transmission of information during the color migration process. Based on this, through this migration model, accurate, natural, and controllable image color migration effects can be achieved for various images.

[0112] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0113] Corresponding to the migration model training method provided above, the present application embodiment also provides a migration model training device. Figure 5 , the migration model training device 5 in the embodiment of the present application includes:

[0114] An acquisition unit 501 is configured to acquire an image training set, where the image training set includes style images and content images.

[0115] A first processing unit 502 is configured to input the style image and the content image into the transfer model to be trained, and obtain a transfer image output by the transfer model to be trained;

[0116] A second processing unit 503 is configured to input the style image, the content image, and the transfer image into a preset image feature extraction module, and obtain style features, content features, and transfer features output by the image feature extraction module, wherein the style features are image features of the style image, the content features are image features of the content image, and the transfer features are image features of the transfer image;

[0117] A calculation unit 504 is used to calculate the model loss of the transfer model to be trained based on the style features, content features and transfer features;

[0118] The optimization unit 505 is used to optimize the transfer model to be trained according to the model loss until the model loss has converged, thereby obtaining a trained transfer model. The trained transfer model is used to perform image color style transfer processing.

[0119] In some embodiments, the migration model includes: a reversible neural flow module and a feature migration module; a first processing unit includes:

[0120] an extraction subunit, configured to extract style flow features of the style image and content flow features of the content image through a reversible neural flow module;

[0121] The fusion subunit is used to fuse the style flow features and the content flow features through the feature transfer module to obtain the transfer flow features;

[0122] The reconstruction subunit is used to reconstruct the migration flow features through the reversible neural flow module to obtain the migration image.

[0123] In some embodiments, the computing unit 504 includes:

[0124] A first calculation subunit, configured to calculate a style loss using a preset first loss function based on the style feature and the transfer feature;

[0125] A second calculation subunit is configured to calculate content loss using a preset second loss function based on content features and migration features;

[0126] The third calculation subunit is used to calculate the model loss based on the style loss, the first weight corresponding to the style loss, the content loss and the second weight corresponding to the content loss, wherein the first weight is greater than the second weight.

[0127] In some embodiments, the style feature, the content feature, and the migration feature are all multi-scale features, and the multiple scales corresponding to the style feature, the content feature, and the migration feature are the same.

[0128] In some embodiments, the first calculation subunit is specifically configured to calculate the style loss using a first loss function based on the style features and the transfer features at each scale.

[0129] In some embodiments, the second calculation subunit is specifically configured to calculate the style loss using a second loss function based on the content features and the transfer features at the minimum scale.

[0130] In some embodiments, the acquisition unit 501 includes:

[0131] The acquisition subunit is used to collect environmental images under different weather conditions and lighting conditions;

[0132] A screening subunit, configured to screen out a style image from the environment image based on a preset color style screening condition;

[0133] The determination subunit is configured to determine other images in the environment image except the style image as content images.

[0134] As can be seen from the above, the migration model constructed in the embodiment of the present application includes a reversible neural flow module and a feature migration module. The reversible neural flow module has a unique reversible transformation characteristic, which can establish an accurate bidirectional mapping between the image space and the latent space. It can not only efficiently learn the color distribution characteristics of the image, but also ensure the lossless transmission of information during the color migration process. Based on the above structure of the migration model, when training the migration model, the style image and the content image can be processed by the migration model first to obtain the migration image, and then the style features, content features and migration features corresponding to the above three types of images are obtained by the image feature extraction module, so as to calculate the model loss of the migration model to be trained, and optimize the migration model to be trained according to the model loss. After the model loss has converged, the trained migration model can be obtained. The above training process can help obtain a migration model with high migration efficiency and good migration effect, so that the migration model can achieve accurate, natural and controllable image color migration effects for various images.

[0135] Corresponding to the migration method provided above, the present application embodiment also provides a migration device. Figure 6 , the migration device 6 in the embodiment of the present application includes:

[0136] A determination unit 601 is used to determine a target style image and a target content image;

[0137] The migration unit 602 is used to input the target style image and the target content image into the trained migration model to obtain the target migration image output by the trained migration model, wherein the trained migration model includes: a reversible neural flow module and a feature migration module, the reversible neural flow module is used to extract the target style flow features of the target style image and the target content flow features of the target content image, the feature migration module is used to fuse the target style flow features and the target content flow features to obtain the target migration flow features, and the reversible neural flow module is also used to reconstruct the target migration flow features to obtain the target migration image.

[0138] As can be seen from the above, in the embodiments of the present application, the migration model used includes a reversible neural flow module and a feature migration module. The reversible neural flow module has a unique reversible transformation property, which can establish an accurate bidirectional mapping between the image space and the latent space. It can not only efficiently learn the color distribution characteristics of the image, but also ensure lossless transmission of information during the color migration process. Based on this, through this migration model, accurate, natural, and controllable image color migration effects can be achieved for various images.

[0139] Corresponding to the migration model training method provided above, the embodiment of the present application also provides an electronic device. Figure 7The electronic device 7 in the embodiment of the present application includes: a memory 701, one or more processors 702 ( Figure 7 Only one is shown) and a computer program stored in the memory 701 and executable on the processor. Specifically, when the electronic device is used in the training phase of the migration model, the processor 702 implements the following steps by running the computer program stored in the memory 701:

[0140] Obtain an image training set, which includes style images and content images;

[0141] Inputting the style image and the content image into the transfer model to be trained to obtain a transfer image output by the transfer model to be trained;

[0142] Input the style image, content image, and transfer image into a preset image feature extraction module to obtain the style features, content features, and transfer features output by the image feature extraction module, wherein the style features are image features of the style image, the content features are image features of the content image, and the transfer features are image features of the transfer image;

[0143] Calculate the model loss of the transfer model to be trained based on style features, content features, and transfer features;

[0144] The transfer model to be trained is optimized according to the model loss until the model loss has converged, thereby obtaining a trained transfer model. The trained transfer model is used to perform image color style transfer processing.

[0145] Assuming that the above is the first possible implementation, in a second possible implementation provided based on the first possible implementation, the transfer model includes: a reversible neural flow module and a feature transfer module; inputting the style image and the content image into the transfer model to be trained, and obtaining a transfer image output by the transfer model to be trained, including:

[0146] Extracting style flow features of the style image through a reversible neural flow module, and extracting content flow features of the content image;

[0147] The style flow features and content flow features are fused through the feature transfer module to obtain the transfer flow features;

[0148] The migration flow features are reconstructed through the reversible neural flow module to obtain the migration image.

[0149] In a third possible implementation provided as a basis for the first possible implementation, the model loss of the to-be-trained transfer model is calculated based on the style features, the content features, and the transfer features, including:

[0150] Based on the style features and the transfer features, the style loss is calculated using a preset first loss function;

[0151] Based on the content features and the migration features, the content loss is calculated using a preset second loss function;

[0152] The model loss is calculated based on the style loss, the first weight corresponding to the style loss, the content loss, and the second weight corresponding to the content loss, wherein the first weight is greater than the second weight.

[0153] In a fourth possible implementation provided on the basis of the third possible implementation, the style feature, the content feature, and the migration feature are all multi-scale features, and the multiple scales corresponding to the style feature, the content feature, and the migration feature are the same.

[0154] In a fifth possible implementation provided as a basis for the fourth possible implementation, the style loss is calculated using a preset first loss function based on the style features and the transfer features, including:

[0155] At each scale, the style loss is calculated using the first loss function based on the style features and transfer features.

[0156] In a sixth possible implementation provided on the basis of the fourth possible implementation, the content loss is calculated using a preset second loss function based on the content features and the migration features, including:

[0157] At the smallest scale, the style loss is calculated using the second loss function based on content features and transfer features.

[0158] In a seventh possible implementation provided on the basis of the first possible implementation, obtaining an image training set includes:

[0159] Collect environmental images under different weather conditions and lighting conditions;

[0160] Based on the preset color style screening conditions, the style image is screened out from the environment image;

[0161] The images other than the style image in the environment image are determined as content images.

[0162] Specifically, when the electronic device is used in the application stage of the migration model, the processor 702 implements the following steps by running the computer program stored in the memory 701:

[0163] Determine the target style image and target content image;

[0164] The target style image and the target content image are input into the trained transfer model to obtain the target transfer image output by the trained transfer model, wherein the trained transfer model includes: a reversible neural flow module and a feature transfer module, the reversible neural flow module is used to extract the target style flow features of the target style image and the target content flow features of the target content image, the feature transfer module is used to fuse the target style flow features and the target content flow features to obtain the target transfer flow features, and the reversible neural flow module is also used to reconstruct the target transfer flow features to obtain the target transfer image.

[0165] It should be understood that in the embodiment of the present application, the processor 702 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0166] The memory 701 may include a read-only memory and a random access memory, and provides instructions and data to the processor 702. A portion or all of the memory 701 may also include a non-volatile random access memory. For example, the memory 701 may also store device type information.

[0167] As can be seen from the above, the migration model constructed in the embodiment of the present application includes a reversible neural flow module and a feature migration module. The reversible neural flow module has a unique reversible transformation characteristic, which can establish an accurate bidirectional mapping between the image space and the latent space. It can not only efficiently learn the color distribution characteristics of the image, but also ensure the lossless transmission of information during the color migration process. Based on the above structure of the migration model, when training the migration model, the style image and the content image can be processed by the migration model first to obtain the migration image, and then the style features, content features and migration features corresponding to the above three types of images are obtained by the image feature extraction module, so as to calculate the model loss of the migration model to be trained, and optimize the migration model to be trained according to the model loss. After the model loss has converged, the trained migration model can be obtained. The above training process can help obtain a migration model with high migration efficiency and good migration effect, so that after the migration model is put into use, it can achieve accurate, natural and controllable image color migration effects for various images through the migration model.

[0168] An embodiment of the present application further provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in the above-mentioned various method embodiments.

[0169] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0170] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0171] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of external device software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0172] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the above modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0173] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the associated hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program, when executed by the processor, can implement the steps of the above-mentioned various method embodiments. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The above-mentioned computer-readable storage medium may include: any entity or device that can carry the above-mentioned computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer-readable memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable storage media does not include electrical carrier signals and telecommunication signals.

[0175] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A migration model training method, characterized in that: include: Obtaining an image training set, wherein the image training set includes: a style image and a content image; Inputting the style image and the content image into a transfer model to be trained to obtain a transfer image output by the transfer model to be trained, wherein the transfer model includes: a reversible neural flow module and a feature transfer module; Inputting the style image, the content image, and the transition image into a preset image feature extraction module to obtain style features, content features, and transition features output by the image feature extraction module, wherein the style features are image features of the style image, the content features are image features of the content image, and the transition features are image features of the transition image; Calculating a model loss of the to-be-trained transfer model based on the style feature, the content feature, and the transfer feature; The transfer model to be trained is optimized according to the model loss until the model loss has converged, thereby obtaining a trained transfer model, and the trained transfer model is used to perform image color style transfer processing.

2. The migration model training method according to claim 1, wherein: Inputting the style image and the content image into the transfer model to be trained to obtain a transfer image output by the transfer model to be trained includes: extracting style flow features of the style image and content flow features of the content image through the reversible neural flow module; fusing the style flow features and the content flow features through the feature migration module to obtain migration flow features; The migration flow features are reconstructed by the reversible neural flow module to obtain a migration image.

3. The migration model training method according to claim 1, wherein: The calculating the model loss of the to-be-trained migration model based on the style feature, the content feature, and the migration feature includes: Calculating a style loss using a preset first loss function based on the style feature and the transfer feature; Calculating content loss using a preset second loss function based on the content feature and the migration feature; The model loss is calculated based on the style loss, a first weight corresponding to the style loss, the content loss, and a second weight corresponding to the content loss, wherein the first weight is greater than the second weight.

4. The migration model training method according to claim 3, wherein: The style feature, the content feature, and the migration feature are all multi-scale features, and the multiple scales corresponding to the style feature, the content feature, and the migration feature are the same.

5. The migration model training method according to claim 4, wherein: The calculating the style loss based on the style feature and the transfer feature by using a preset first loss function includes: At each scale, a style loss is calculated using the first loss function based on the style feature and the transfer feature.

6. The migration model training method according to claim 4, wherein: The calculating the content loss by using a preset second loss function based on the content feature and the migration feature includes: At the minimum scale, a style loss is calculated using the second loss function based on the content feature and the transfer feature.

7. The migration model training method according to claim 1, wherein: The obtaining of the image training set comprises: Collect environmental images under different weather conditions and lighting conditions; Based on a preset color style screening condition, screening out a style image from the environment image; The other images in the environment image except the style image are determined as content images.

8. A migration method, characterized in that: include: Determine the target style image and target content image; The target style image and the target content image are input into a trained transfer model to obtain a target transfer image output by the trained transfer model, wherein the trained transfer model includes: a reversible neural flow module and a feature transfer module, the reversible neural flow module is used to extract the target style flow features of the target style image and the target content flow features of the target content image, the feature transfer module is used to fuse the target style flow features and the target content flow features to obtain target transfer flow features, and the reversible neural flow module is also used to reconstruct the target transfer flow features to obtain the target transfer image.

9. A migration model training device, characterized in that: include: An acquisition unit, configured to acquire an image training set, wherein the image training set includes: a style image and a content image; a first processing unit, configured to input the style image and the content image into a transfer model to be trained, and obtain a transfer image output by the transfer model to be trained; a second processing unit, configured to input the style image, the content image, and the transition image into a preset image feature extraction module, and obtain style features, content features, and transition features output by the image feature extraction module, wherein the style features are image features of the style image, the content features are image features of the content image, and the transition features are image features of the transition image; a calculation unit, configured to calculate a model loss of the transfer model to be trained based on the style feature, the content feature, and the transfer feature; An optimization unit is used to optimize the transfer model to be trained according to the model loss until the model loss has converged, thereby obtaining a trained transfer model, and the trained transfer model is used to perform image color style transfer processing.

10. A migration device, characterized in that: include: a determination unit, configured to determine a target style image and a target content image; A migration unit is used to input the target style image and the target content image into a trained migration model to obtain a target migration image output by the trained migration model, wherein the trained migration model includes: a reversible neural flow module and a feature migration module, the reversible neural flow module is used to extract target style flow features of the target style image and extract target content flow features of the target content image, the feature migration module is used to fuse the target style flow features and the target content flow features to obtain target migration flow features, and the reversible neural flow module is also used to reconstruct the target migration flow features to obtain the target migration image.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented; and / or when the processor executes the computer program, the method according to claim 8 is implemented.

12. A computer program product, characterized in that The computer program product comprises a computer program, which implements the method according to any one of claims 1 to 7 when executed by one or more processors; and / or, which implements the method according to claim 8 when executed by one or more processors.

Images