Model training methods, image processing methods, and electronic devices

Abstract

This application discloses a model training method, an image processing method, and an electronic device. It belongs to the field of image processing technology. An embodiment of the method includes: acquiring a training dataset; wherein the training dataset includes multiple triples, each triple including a source image, a style reference image, and a ground truth image, the style reference image and the ground truth image having the same color tone style; training an initial model based on the training dataset to obtain a style transfer model; wherein, during the training process, a first loss function, a second loss function, and a third loss function are used for supervision, the first loss function indicating pixel-level reconstruction loss, the second loss function indicating style-aware loss, and the third loss function indicating key region preservation loss.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to model training methods, image processing methods, and electronic devices. Background Technology

[0002] In the fields of image processing and artificial intelligence, color tone transfer and enhancement technology aims to apply the visual style of a reference image to a source image to improve its aesthetic performance. This technology is widely used in post-production photography, film and television production, and creative media. Its core objective is to change the overall look of an image while maintaining the authenticity and naturalness of the visual content.

[0003] In existing technologies, image color and tone enhancement is typically achieved using rule-based methods or color mapping. However, these methods only allow for global style adjustments and are prone to problems such as localized distortion, blurred details, or color distortion in key areas, resulting in low-quality images. Summary of the Invention

[0004] The purpose of this application is to provide a model training method, an image processing method, and an electronic device that can generate images with globally consistent style, clear details, and natural colors in key areas, thereby improving the quality of the output images of the style transfer model.

[0005] In a first aspect, embodiments of this application provide a model training method, the method comprising: acquiring a training dataset; wherein the training dataset includes multiple triples, each triple including a source image, a style reference image, and a ground truth image, the style reference image having a consistent color tone style with the ground truth image; training an initial model based on the training dataset to obtain a style transfer model; wherein, during the training process, a first loss function, a second loss function, and a third loss function are used for supervision, the first loss function indicating pixel-level reconstruction loss, the second loss function indicating style-aware loss, and the third loss function indicating critical region preservation loss.

[0006] Secondly, embodiments of this application provide an image processing method, the method comprising: acquiring an image to be processed and metadata corresponding to the image to be processed; determining style parameters based on the image to be processed and the metadata; generating a target style reference image based on the style parameters and the image to be processed; inputting the image to be processed and the target style reference image into a style transfer model to obtain a target image; wherein the style transfer model is trained by the model training method described in the first aspect.

[0007] Thirdly, embodiments of this application provide a model training apparatus, comprising: an acquisition unit for acquiring a training dataset; wherein the training dataset includes multiple triples, each triple including a source image, a style reference image, and a ground truth image, the style reference image having a consistent color tone style with the ground truth image; and a training unit for training an initial model based on the training dataset to obtain a style transfer model; wherein, during the training process, a first loss function, a second loss function, and a third loss function are used for supervision, the first loss function indicating pixel-level reconstruction loss, the second loss function indicating style-aware loss, and the third loss function indicating critical region protection loss.

[0008] Fourthly, embodiments of this application provide an image processing apparatus, comprising: an acquisition unit for acquiring an image to be processed and metadata corresponding to the image to be processed; a first determination unit for determining style parameters based on the image to be processed and the metadata; a generation unit for generating a target style reference image based on the style parameters and the image to be processed; and a processing unit for inputting the image to be processed and the target style reference image into a style transfer model to obtain a target image; wherein the style transfer model is trained by the model training apparatus described in the first aspect.

[0009] Fifthly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing the steps of the methods described in the first or second aspect.

[0010] In a sixth aspect, embodiments of this application provide a readable storage medium on which a computer program is stored, and when executed by a processor, the computer program implements the steps of the method as described in the first or second aspect above.

[0011] In a seventh aspect, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the steps of the methods described in the first or second aspect.

[0012] Eighthly, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the steps of the method as described in the first or second aspect.

[0013] In this embodiment, during model training, ground truth images are introduced to provide the model with precise detail reconstruction targets. Combined with pixel-level reconstruction loss, this ensures the detail fidelity of the generated images. Style reference images are introduced to provide the model with clear style learning targets. Combined with style-aware loss, this drives the model to learn and transfer color tonal distribution, ensuring the accuracy of style transfer. Key region protection loss is introduced to guide the model in protecting the naturalness of key region colors during stylization, effectively avoiding local distortion caused by global adjustments. These three losses work together to supervise and guide the model to learn to simultaneously optimize content detail reconstruction, global style matching, and local color protection when transferring color tonal values. This results in high-quality images with harmonious global style, clear details, and natural colors in key regions, improving the quality of the style transfer model's output images. Attached Figure Description

[0014] Figure 1 This is a flowchart of the model training method provided in the embodiments of this application; Figure 2 This is a schematic diagram illustrating the processing of the basic model in the model training method provided in the embodiments of this application; Figure 3 This is a flowchart of the image processing method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the training process of the feature extraction model in the image processing method provided in the embodiments of this application; Figure 5 This is a schematic diagram of the style library construction process in the image processing method provided in the embodiments of this application; Figure 6 This is a schematic diagram of the index generation process in the image processing method provided in the embodiments of this application; Figure 7 This is a schematic diagram of the image processing process in the image processing method provided in the embodiments of this application; Figure 8 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application; Figure 10 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application; Figure 11 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0016] The terms "first," "second," etc., used in this application's specification are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0017] The model training method and apparatus provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0018] Please refer to Figure 1 This document illustrates one of the flowcharts of the model training method provided in this application embodiment. The model training method provided in this application embodiment can be applied to electronic devices. In practice, the aforementioned electronic devices can be smartphones, tablets, laptops, wearable devices, etc.

[0019] The model training method provided in this application includes the following steps: Step 101: Obtain the training dataset; wherein the training dataset includes multiple triples, each triple including a source image, a style reference image and a ground truth image, and the style reference image and the ground truth image have the same color tone style.

[0020] In this embodiment, the training dataset is a collection of data used to train a machine learning model, which may include multiple sets of training data. Each set of training data is a triple consisting of three images. Each triple includes a source image, a style reference image, and a ground truth image.

[0021] In this context, the source image refers to the original input image to be enhanced in color and tone, denoted as I_src, whose content, composition, and details need to be preserved. The ground truth image refers to the image that has undergone ideal color and tone processing and serves as the final learning target, denoted as Igt. It shares the same style attributes as the style reference image in the same triplet and is highly semantically corresponding to the source image, providing pixel-level supervision signals for the model. The style reference image refers to an image with the same color and tone style as the ground truth image, but it does not need to be aligned with the source and ground truth images in terms of spatial content; for example, object positions and compositions can be different. The style reference image can be denoted as Iref.

[0022] In practice, a new image, known as a style reference image, can be generated by applying a series of spatial geometric transformations to the ground truth image, such as random cropping, rotation, scaling, and slight perspective distortion. This new image is no longer aligned with the ground truth image in pixel coordinates, but its color and tonal style are consistent. By introducing a style reference image, the model is forced to learn semantic correspondences rather than simply copying pixels.

[0023] Step 102: Train the initial model based on the training dataset to obtain the style transfer model; during the training process, a first loss function, a second loss function, and a third loss function are used for supervision. The first loss function indicates pixel-level reconstruction loss, the second loss function indicates style-aware loss, and the third loss function indicates key region protection loss.

[0024] In this embodiment, the initial model refers to the neural network model architecture that has not yet been optimized using the training dataset before the training process begins. The initial model may have processing capabilities such as feature extraction, feature matching and fusion, and image reconstruction. The style transfer model refers to the final, usable model obtained after optimizing the network parameters of the initial model through the training process. This model can receive a source image and a style reference image as input and output an image that fuses the content of the source image and the color tones of the style reference image.

[0025] In this embodiment, machine learning methods, such as supervised learning, can be used to train the initial model to obtain the style transfer model. During the training process, a first loss function, a second loss function, and a third loss function can be used to supervise the initial model.

[0026] The first loss function indicates the pixel-level reconstruction loss. For example, the first loss function can be a loss function used to measure the difference in pixel values ​​between the model's output image and the ground truth image. Its core function is to provide strong supervision, forcing the model to learn an accurate pixel-level mapping from the source image to the target style image, thereby ensuring the content fidelity and structural stability of the enhancement result and avoiding structural distortion or severe distortion in the generated image.

[0027] The second loss function can indicate style-aware loss. For example, the second loss function can be a loss function that measures the similarity between the model's output image and the style reference image in high-level feature statistics. It does not require spatial alignment, but instead drives the model to capture and transfer the overall color atmosphere and texture style of the style reference image by comparing statistical moments, such as Gram matrices, between feature maps.

[0028] The third loss function can indicate the loss in protecting critical regions. For example, the third loss function can be a loss function used to constrain the degree to which the model modifies the colors of critical regions in an image. It can prevent unnatural color casts in areas that are abnormally sensitive to color by comparing the color statistical features of the source image and the model output image in these critical regions, such as histograms and means, thus ensuring the naturalness and visual comfort of the image enhancement results.

[0029] During training, after each input of training data into the initial model, the loss value of the initial model can be obtained by calculating pixel-level reconstruction loss, style-aware loss, and key region preservation loss based on the output of the initial model using the first loss function, the second loss function, and the third loss function, respectively. Then, backpropagation and gradient descent algorithms can be used to calculate the gradient based on the loss value and update the parameters in the initial model.

[0030] The method provided in the above embodiments of this application uses a triplet consisting of a source image, a ground truth image, and a style reference image with the same color tone style as the ground truth image but different content as a training dataset. It then uses a first loss function indicating pixel-level reconstruction loss, a second loss function indicating style-aware loss, and a third loss function indicating key region preservation loss for multi-objective supervision, thereby training a style transfer model. In this process, introducing the ground truth image provides the model with a precise detail reconstruction target, and combined with pixel-level reconstruction loss, ensures the detail fidelity of the model-generated image. Introducing the style reference image provides the model with a clear style learning target, and combined with style-aware loss, drives the model to learn and transfer color tone distribution, ensuring the accuracy of style transfer. Introducing key region preservation loss guides the model to protect the color naturalness of key regions during stylization, effectively avoiding local distortion caused by global adjustments. The three losses mentioned above work together to supervise and guide the model to learn to optimize the three objectives of content detail reconstruction, global style matching, and local color preservation when transferring color tones. This enables the generation of high-quality images with global style coordination, clear details, and natural colors in key areas, thus improving the quality of the output images of the style transfer model.

[0031] In some optional embodiments, step 101 above may further include the following steps: Step S11: Input the source image and style reference image into the initial model to obtain the predicted image output by the initial model. Specifically, the initial model can be an encoder-decoder structure and includes a cross-image semantic feature matching module, such as one based on a cross-attention mechanism. Internally, the model performs a series of nonlinear transformations, including feature extraction, feature matching and fusion, and image reconstruction. The source image and style reference image can be input into the initial model, and the initial model can output a predicted image with the same resolution as the source image, denoted as I_out.

[0032] Step S12: Determine the first loss value based on the first loss function, the predicted image, and the ground truth image. The first loss value is a scalar value calculated by comparing the predicted image and the ground truth image based on the first loss function. This value quantifies the degree to which the predicted image deviates from the ground truth image at the pixel level; the smaller the value, the higher the fidelity of the content reconstruction. The first loss value can be denoted as L_rec.

[0033] Specifically, the first loss function can be either L1 loss (mean absolute error loss) or L2 loss (mean squared error loss). The first loss value is obtained by inputting the pixel value matrix of the predicted image I_out and the ground truth image Igt into the first loss function. For example, the calculation formula can be: L_rec = ||I_out - I_gt||1.

[0034] Step S13: Determine the second loss value based on the second loss function, the predicted image, and the style reference image. The second loss value is a scalar value calculated by comparing the predicted image and the style reference image based on the second loss function. This value quantifies the degree to which the predicted image deviates from the style reference image in terms of overall style features such as color distribution and texture statistics; the smaller the value, the higher the accuracy of style transfer. The second loss value can be denoted as L_percep.

[0035] As an example, the second loss function can also be L1 loss or L2 loss. First, deep features of the predicted image and the style reference image can be extracted, and then these deep features are input into the second loss function to obtain the second loss value.

[0036] Step S14: Determine the third loss value based on the third loss function, the source image, and the predicted image. The third loss value is a scalar value calculated by comparing the key regions in the source image and the predicted image based on the third loss function. This value quantifies the degree of unexpected color distortion occurring in the key regions; the smaller the value, the better the protection effect on the key regions. The third loss value can be denoted as L_semantic.

[0037] As an example, key regions in the source image I_src and the predicted image I_out can first be identified, which can be obtained through a pre-trained segmentation network or detector. Then, based on the difference between the two key regions using a third loss function, a third loss value can be obtained.

[0038] Step S15: Based on the first loss value, the second loss value, and the third loss value, determine the total loss value of the initial model. Specifically, the total loss value can be obtained by weighted summation of the first loss value, the second loss value, and the third loss value. The total loss value reflects the overall error of the model's current output in three dimensions: content fidelity, style transfer accuracy, and local preservation effect. The total loss value can be denoted as L_total.

[0039] As an example, the total loss value can be calculated using the following formula: L_total=λ1L_rec+λ2L_percep+λ3L_semantic Wherein, λ1, λ2, and λ3 are weighting coefficients, which can be preset as needed and are not limited here.

[0040] Step S16: Adjust the network parameters of the initial model based on the total loss value to obtain the style transfer model.

[0041] Specifically, it refers to the process of calculating the gradient of the loss with respect to each network parameter using the backpropagation algorithm based on the total loss value, and then updating the network parameters along the reverse direction of the gradient using optimization algorithms, such as stochastic gradient descent and its variants. The aim is to iteratively reduce the total loss value, thereby improving the model's predictive output.

[0042] Traditional style transfer algorithms typically only employ style-aware loss, leading to poor style transfer performance or even destruction of image content structure. In this embodiment, by combining triplet data and pixel-level reconstruction loss, strong supervised constraints are applied to the style transfer effect. This allows the final trained style transfer model to collaboratively optimize multiple performance metrics, thereby significantly enhancing the artistic expressiveness of images while maintaining content authenticity and local naturalness, achieving high-quality, controllable color enhancement.

[0043] In some alternative embodiments, see Figure 2 The initial model includes a first encoder, a second encoder, an attention network, and a decoder. Based on this, step S11 may further include the following steps: inputting the source image into the first encoder to obtain first image features; inputting the style reference image into the second encoder to obtain second image features; inputting the first and second image features into the attention network to obtain fused features; and inputting the fused features into the decoder to obtain the predicted image.

[0044] The first encoder refers to the neural network structure in the initial model used to extract features from the source image. It can be composed of a series of convolutional layers, normalization layers, and activation functions stacked together. Its function is to map the input source image from a high-dimensional pixel space to a low-dimensional, dense deep semantic feature space, and output the first image features of the source image. The first image features can characterize the content structure, semantic information, and basic visual attributes of the source image; the first image features can also be called content features.

[0045] The second encoder refers to the neural network structure in the initial model used to extract features from the style reference image. Its network structure can be the same as or similar to the first encoder, and the weights can be shared or independent. Its function is to extract deep features from the style reference image and output second image features. These second image features not only contain the semantic content of the reference image, but more importantly, they encode target style information such as color distribution, tonal style, and texture. These second image features can also be called style features.

[0046] An attention network refers to the computational unit built on a cross-attention mechanism in the initial model. It receives first and second image features as input, dynamically generating a set of attention weights by calculating their similarity in the feature space. These weights are then used to selectively and adaptively fuse the second image features into the first image features, outputting a fused feature. The fused feature retains the main structural information of the first image features while incorporating style information extracted from the second image features and semantically aligned. The attention network achieves semantic alignment and local fusion across images. By introducing an attention network into color transformation, for a specific region in the source image, a semantically corresponding region in the style reference image can be found, thus transferring the brightness, color, and other attributes of key regions in the style reference image to the source image. Because local mapping is used for transfer, it avoids the key regions from migrating colors from other regions, preventing them from deviating from their intended colors.

[0047] The decoder refers to the neural network structure in the initial model responsible for decoding the fused features into a pixel-level image. It can consist of a series of upsampling layers and convolutional layers, and its function is to progressively increase the spatial resolution of the fused features and reduce the number of channels, ultimately generating a predicted image with the same size as the source image.

[0048] By employing an initial model that includes an attention network, a semantically guided and locally adaptive color tone transfer mechanism was implemented. This design ensures that style information can be accurately applied to the corresponding areas according to the semantics of the image content, thereby fundamentally avoiding the color distortion problem in sensitive areas such as skin and sky caused by global style application in traditional methods. This significantly improves the accuracy of color enhancement, local naturalness, and the realism of the overall visual effect.

[0049] In some optional embodiments, step S12 above may further include the following steps: Step S31: Extract deep features from the predicted image and the style reference image respectively.

[0050] Here, a feature extraction network can be used to extract deep features from the predicted image and the style reference image, respectively. The feature extraction network refers to a deep convolutional neural network whose parameters have been pre-trained on a large-scale image dataset (such as ImageNet) and frozen. This network is not part of the trained base model, but rather serves as a fixed tool for extracting high-level abstract features of the image. For example, convolutional neural network structures such as VGG (Visual Geometry Group Network) and ResNet (Residual Network) can be used.

[0051] Deep features refer to the feature maps output by the intermediate layers of a feature extraction network. Deep features ignore precise spatial details but encode statistical distribution information of image content and style, making them suitable for measuring the similarity of two images in overall style and texture.

[0052] Step S32: Determine the Gram matrix of the deep features of the predicted image and the deep features of the style reference image, respectively.

[0053] A Gram matrix is ​​a matrix used to calculate the correlation between channels of a feature map. For a feature map of dimension C×H×W, it is first reshaped into a two-dimensional matrix F of dimension C×(H×W), and then its Gram matrix G = F·F^T is calculated. Each element Gij of the Gram matrix represents the mean of the inner product of the i-th feature channel and the j-th feature channel over the entire feature map space. It captures the co-occurrence relationship between the activation modes of different feature channels, effectively characterizing the style information of the image such as texture and color distribution, and effectively encoding the overall texture pattern, brushstroke style, and color matching rules of the image.

[0054] Step S33: Determine the second loss value based on the second loss function and the Gram matrix.

[0055] The second loss function can be a function used to calculate the difference between two Gram matrices, such as a function to calculate their Euclidean distance or mean squared error. The two Gram matrices can be input into the second loss function to obtain the second loss value.

[0056] By extracting deep features from the predicted image and the style reference image and calculating their Gram matrices, the measurement of style similarity can be transformed from an indefinable intuitive feeling into a computable mathematical indicator based on feature statistics. By determining the second loss value based on the second loss function and the Gram matrix, the model can minimize the difference in the Gram matrix between its output image and the style reference image during training. This guides the model to learn to mimic the inherent texture generation rules and color distribution structure of the style reference image, rather than simply replacing pixel values ​​or global tone mapping when transferring color tones. This improves and ensures the accuracy and robustness of the style transfer model in style restoration, resulting in images that not only change colors but also highly match the user's desired target style in terms of overall visual atmosphere and artistic texture.

[0057] In some optional embodiments, step S13 above may further include the following steps: Step S41: Determine key regions in the source image and the prediction image; wherein the key regions include at least one of the following: face region, skin region.

[0058] Key regions refer to specific semantic areas in an image that are highly visually sensitive to color changes, and whose color abnormalities can easily lead to an unnatural or even distorted overall image appearance. For example, these may include, but are not limited to, at least one of the following: face regions and skin regions. The face region is the entire area encompassing facial features and contours. The skin region is the skin color area of ​​the face and exposed parts of the body. The colors in these regions have a recognized baseline range in human visual perception and image aesthetic evaluation; abnormal modifications to them will affect the realism of the image.

[0059] In practice, a pre-trained detection model can be used to detect key regions in both the source and predicted images. This detection model does not participate in the training and parameter updates of the style transfer model; it serves only as a fixed preprocessing tool. For example, a lightweight face detector can be used to locate face bounding boxes and use their interior regions as key regions, or a skin semantic segmentation model can be used to detect skin regions in the image. No specific limitations are imposed here.

[0060] Step S42: Determine the color statistical features of the key regions in the source image and the predicted image, respectively.

[0061] Color statistical features refer to mathematical features used to quantitatively describe the color distribution patterns within a key area. They are not the color values ​​of individual pixels, but rather a statistical summary of the color information of all pixels within the entire key area. For example, color statistical features may include, but are not limited to, color histograms, color moments, and color covariance matrices.

[0062] Step S43: Determine the third loss value based on the third loss function and color statistical features.

[0063] The third loss function can be defined as the distance between these two statistical features, such as the histogram intersection distance or the Mahalanobis distance. The color statistical features of both features can be input into the third loss function to obtain the third loss value. The smaller this value, the closer the two key regions are in the overall color distribution, meaning the color of the key regions is effectively protected.

[0064] By explicitly introducing protective constraints based on the color statistical features of key regions into the loss function, abnormal color shifts that are prone to occur in sensitive areas such as faces and skin during color migration can be effectively curbed. This significantly improves the local naturalness, visual comfort, and overall aesthetic reliability of the final enhanced image, making the personalized style enhancement results both artistic and realistic.

[0065] Further reference Figure 3 This document illustrates a flowchart of the image processing method provided in an embodiment of this application. The model training method provided in this application includes the following steps: Step 301: Obtain the image to be processed and the corresponding metadata.

[0066] In this embodiment, the image to be processed refers to the original image input by the user that requires personalized enhancement of color and tone. It is usually a high-resolution natural scene photograph, whose existing color and lighting effects may not meet the user's expectations.

[0067] Metadata refers to auxiliary data associated with the shooting process or content of the image to be processed. It can be embedded in the image file or provided by the user / electronic device to provide context for the image. Metadata may include, but is not limited to, at least one of the following: shooting time, shooting location, weather conditions, device model, scene tags, etc. This information can indicate the shooting context of the image to be processed.

[0068] Step 302: Determine style parameters based on the image to be processed and metadata.

[0069] In this embodiment, the style parameter is a parameterized representation used to control and define the final color and tonal presentation of an image. It can be determined by analyzing the image to be processed and metadata, and is used to accurately guide the personalized enhancement style applicable to the image to be processed.

[0070] Specifically, the image to be processed and its metadata can be analyzed, including the current scene. The current scene is then matched against a pre-built user-personalized style library. This style library records the user's preferred color and tonal tendencies in different scene types. By analyzing the match between the current scene and the styles of each scene recorded in the library, the most suitable set of style control parameters for the current input can be determined.

[0071] As an example, the image to be processed is a mountain forest image. Photos saved by users in the past under the themes of "autumn," "afternoon," and "mountain forest" tend to have a style of "vibrant autumn colors" and "clear light and shadow." Therefore, it can be determined that the image to be processed should be adapted to this style, and the parameters corresponding to these two styles are fused to obtain the style parameters.

[0072] Step 303: Generate a target style reference image based on style parameters and the image to be processed.

[0073] In this embodiment, the target style reference image refers to an image whose color tone style highly conforms to the above style parameter requirements and whose semantic content and composition structure are basically corresponding to the above image to be processed, and can be used to provide a reference for subsequent color tone transfer.

[0074] In practice, style parameters can be used as conditions, and the image to be processed can be used as content guidance to call an image generation model to synthesize a new image. During the generation process, the model tends to maintain the general scene type and overall composition of the input image, but will freely reshape details, textures, and even change the shape of non-critical elements to a certain extent in order to better fit and reflect the target style.

[0075] As an example, the image to be processed is a forest image. A diffusion model can be used, taking style parameters as conditional input and guided by the image to be processed, to generate a new forest image as the target style reference image. In the target style reference image, the colors of the leaves are enhanced to more vibrant reds and yellows, and the overall lighting appears more transparent.

[0076] Optionally, the image to be processed can first be denoised to obtain a noisy image. Then, the style parameters and the noisy image are input into a diffusion model. The diffusion model denoises the noisy image based on the style parameters to obtain a target style reference image. The diffusion model is a type of generative artificial intelligence model based on a denoising diffusion probability model, such as the stable diffusion model. Its core principle is to learn an inverse process that progressively removes noise from an image, thereby generating high-quality, diverse images from pure noise. In this embodiment, this model is used as a stylized reference image generator.

[0077] Specifically, the image to be processed can first be encoded into a latent space code Z0 using a variational autoencoder (VAE). Then, random Gaussian noise of a certain intensity is added to Z0. This yields the noisy latent vector ZT. The denoising strength can be set between 0.4 and 0.6; too low a strength will prevent style generation, while too high a strength will destroy the original image's content structure. Next, the noisy image and style parameters are input into the diffusion model. The diffusion model, combined with the style parameters, progressively denoises the ZT, ultimately decoding and outputting a lower-resolution target style reference image whose style highly matches the user's preferences.

[0078] By utilizing a conditionally guided diffusion model to generate a process, personalized style parameters are transformed into specific style reference images that are highly adapted to the input content, thereby providing high-quality, highly relevant, and diverse style targets for subsequent color tone transfer steps.

[0079] Step 304: Input the image to be processed and the target style reference image into the style transfer model to obtain the target image.

[0080] In this embodiment, the style transfer model can be trained using any of the model training methods described in the above embodiments. The image to be processed and the target style reference image can be input into the style transfer model. Through its internal mechanism, the model intelligently maps and fuses style features from different regions of the target style reference image, such as the blue hue of the sky, the saturation of the grass, and the warmth of skin tones, onto the corresponding semantic regions in the image to be processed, thereby outputting the target image. The target image is a high-quality image that has undergone personalized color and tone enhancement processing. It retains the original clarity and detail of the image to be processed while incorporating artistic colors and lighting styles that conform to the user's specific preferences.

[0081] As an example, the image to be processed is of children playing in a forest. The style transfer model automatically identifies the children's facial skin areas in the image and transfers a "rosy and warm" skin tone from the corresponding area in the target style reference image; it also identifies the "background forest" area and transfers a "bright and emerald green" color. The final generated high-resolution target image perfectly preserves the original details such as facial expressions, hair, and clothing textures, but the overall color and lighting have been updated to a fresh and warm style that the user prefers.

[0082] The method provided in the above embodiments of this application first obtains the image to be processed and its corresponding metadata, then determines style parameters based on the image to be processed and the metadata, then generates a target style reference image based on the style parameters and the image to be processed, and finally inputs the image to be processed and the target style reference image into a style transfer model to obtain the target image. On the one hand, by determining the target style parameters based on the image to be processed and its metadata, and generating a target style reference image based on these parameters and the image to be processed, the style selection can be associated with the specific content and shooting context of the image to be processed. This provides a visual reference whose content is compatible with the image to be processed and accurately reflects the target style, thus providing an optimal style example for any image and helping to improve the quality of the generated target image. On the other hand, since the style transfer model can achieve localized and precise color and tone transfer, it can effectively avoid the global coloring problem caused by traditional global filter methods, and can generate high-quality images with global style coordination, clear details, and natural colors in key areas, thus further improving the quality of the target image.

[0083] In some optional embodiments, step 302 above may further include the following steps: Step S51: Generate style features based on the image to be processed and metadata.

[0084] Style features are feature vectors that comprehensively represent the visual content and metadata of an image to be processed, and can be used for subsequent retrieval. Specifically, visual features can be extracted from the image to be processed, and environmental features can be extracted from the metadata. The two are then fused to obtain style features. Style features can be denoted as V_query.

[0085] Step S52: Determine the similarity between style features and multiple indices; wherein each of the multiple indices is associated with a style model in a style library, and the style model is obtained by low-rank adaptive training of a pre-trained model based on a set of user-preferred images.

[0086] An index can be a fixed-dimensional feature vector. Each index uniquely associates with a style model in the style library and represents the core features of a specific user preference style represented by that style model. Indices can be obtained offline by averaging the multimodal features of a certain type of user preference image. For example, there are N indices, which can be denoted as {KeyVector_1, KeyVector_2, ..., KeyVector_N}, where N is a positive integer. The style library is a collection used to store style models and their indices.

[0087] A style model refers to a parameterized module used for lightweight, personalized style modulation of pre-trained models such as Stable Diffusion. Specifically, it can be obtained by performing low-rank adaptive training on a pre-trained model. It can be a LoRA (Low-Rank Adaptation) model. Each style model encodes a color tone style. The correspondence between indices and style models can be denoted as {KeyVector_1: LoRA model 1}, {KeyVector_2: LoRA model 2}, ..., {KeyVector_N: LoRA model N}.

[0088] In practice, the style features can be compared one by one with all indices {KeyVector_1, KeyVector_2, ..., KeyVector_N}, and the cosine similarity S_i = cos(V_query, KeyVector_i) can be calculated. i is a positive integer less than or equal to N.

[0089] Step S53: Based on similarity, perform weighted fusion of the parameters of at least two style models to obtain style parameters.

[0090] Specifically, the top k indices with the highest similarity and their corresponding style models can be selected. These k similarity scores are then Softmax normalized to obtain a set of weights {w_1, w_2, ..., w_k}. Then, the trainable low-rank parameters of these k style models are weighted and summed to obtain the style parameters.

[0091] As an example, multiple indices correspond to style models such as "cool snow scene", "heavy ancient style", and "warm dusk". The similarity between V_query and each index can be calculated. If it is found that the similarity between V_query and the index keys of "cool snow scene" and "heavy ancient style" is high, the parameter matrix of the style model corresponding to "cool snow scene" and the parameter matrix of the style model corresponding to "heavy ancient style" can be weighted and summed to obtain the style parameters.

[0092] By generating style features based on the image to be processed and its metadata, and performing similarity matching with multiple indexes associated with style models, the target style is ensured to be highly semantically and contextually relevant to the image to be processed, greatly improving the accuracy and scene adaptability of the target style. Furthermore, by weighted fusion of parameters from at least two style models based on similarity to obtain the final style parameters, flexible style synthesis and expression capabilities are achieved. The sparse injection strategy makes each basic style model extremely lightweight, greatly reducing the storage overhead and computational burden of a single style model. Thus, a rich variety of style combinations can be supported with ultra-low storage and computational costs, achieving a balance between effectiveness and efficiency.

[0093] In some optional embodiments, step S51 may further include the following steps: Step S61: Extract visual features of the image to be processed using a feature extraction model.

[0094] The feature extraction model is a model used to extract visual features from an image, and may include, but is not limited to, the CLIP (Contrastive Language-Image Pre-training) model. Visual features can be represented as 512-dimensional feature vectors, denoted as F_vis.

[0095] See Figure 4 The feature extraction model includes a visual encoder and a style classifier. The style classifier includes at least two fully connected layers. During training, the feature extraction model is supervised by a fourth loss function and a fifth loss function. The fourth loss function indicates the image-text contrast loss, and the fifth loss function indicates the style classification loss.

[0096] The visual encoder is a fundamental part of the feature extraction model. For example, it can be a model pre-trained on large-scale image-text pairs, whose function is to map the input image into an initial, high-dimensional visual feature representation that primarily encodes the semantic content of the image.

[0097] A style classifier is a small neural network module appended to the output of a visual encoder, and may include at least two fully connected layers. Its function is to receive features from the visual encoder output and further map them to a feature space that can more clearly distinguish different image styles.

[0098] The fourth loss function indicates the image-text contrast loss, which aims to narrow the distance between image features and their corresponding textual description features in the joint embedding space, while widening the feature distance between unrelated image-text pairs. This loss is inherited from the training objective of contrastive learning models such as CLIP, and is used to maintain the general semantic understanding ability of the features. The fourth loss function can be denoted as L_clip. The fifth loss function indicates the style classification loss, which measures the difference between the style label and the style type output by the style classifier. The style label indicates the true style type of the image. The fifth loss function can specifically use cross-entropy loss, denoted as L_style. The total loss of the feature extraction model can be denoted as L_total. L_total = L_clip + λL_style, where λ is a balance coefficient, for example, 0.5.

[0099] Understandably, the features extracted by the conventional CLIP model only contain object classification information, such as "a beach" or "a dog," lacking information related to color style and lighting atmosphere. Therefore, they cannot be directly used to construct visual features due to their training process. Here, building upon the conventional CLIP model, the output features of the visual encoder are not directly used; instead, a finely tuned style classifier is connected. See [link to relevant documentation]. Figure 4 During the training phase, in addition to retaining the original image-text contrast learning loss L_clip from CLIP, a style classification loss L_style is added. This allows the extracted visual features to contain not only the semantic information of the image content but also the style information of the image to be processed, without increasing computational cost. This solves the problem of insufficient style representation capabilities of visual features extracted using traditional methods, providing higher-quality and more comprehensive visual feature inputs for subsequent content-based style matching.

[0100] Step S62: Encode the metadata to obtain environmental characteristics.

[0101] Environmental features refer to features obtained through structured encoding of metadata, which quantitatively represent the contextual information at the time of image capture. Specifically, different fields in the metadata can be encoded separately, and then the encodings can be merged, as shown in the following process: First, the time information in the metadata can be encoded. The shooting timestamp is converted into a periodic feature. For example, the hour h and the day of the year d can be converted to: F_time=[sin(2πh / 24),cos(2πh / 24),sin(2πd / 365),cos(2πd / 365)]. This ensures that similar times are close in the feature space.

[0102] Next, information such as location and weather in the metadata can be encoded. For example, the location coordinates can be converted into semantic text, such as "beach" or "shopping mall," through geocoding, and then combined with the weather label and input into a text encoder, such as CLIP's text encoder, to obtain the text feature F_env_txt.

[0103] Finally, F_time and F_env_txt are concatenated (Concat) and mapped to the same dimension as the visual features through a linear projection layer to obtain the environmental features F_env.

[0104] Step S63: Based on the cross-attention mechanism, visual features and environmental features are fused to obtain style features.

[0105] Because visual and environmental information have different modalities and densities, simply concatenating or adding visual and environmental features is not optimal. This embodiment employs a cross-attention fusion module, using visual features Fvis as the query and environmental features Fenv as the key and value. Through attention calculation, the visual features actively query and fuse the most relevant environmental context information. The calculation formula is: V_query = Attention(Q = F_vis, K = F_env, V = F_env), where Attention is standard scaled dot product attention.

[0106] As an example, when the visual feature F_vis indicates that the image contains "water" and "sunset," a cross-attention mechanism is used to extract and fuse information on "sunset" and "seaside" with high weight from the environmental features F_env indicating "summer," "dusk," and "seaside." The resulting V_query will be a highly unified feature representing "an image taken at a seaside at dusk in summer, featuring sunset and water, with a certain stylistic tendency," thus laying the foundation for subsequent accurate style retrieval.

[0107] By employing a feature extraction model composed of a visual encoder and a style classifier, and jointly trained using image-text contrast loss and style classification loss, this approach ensures that the extracted visual features not only contain traditional object semantic information but also style type information. This addresses the limitation of general visual features in style representation, providing higher-quality and more comprehensive visual feature input for subsequent content-based style matching. Furthermore, by fusing visual features with environmental features using a cross-attention mechanism, a deep association between visual content and context is achieved. This enables the generated style features to accurately represent "what content was captured in what environment," significantly improving the discriminative power and accuracy of style features in expressing complex user preferences.

[0108] In some optional embodiments, a style library may be built before step 302 is executed. The process of building and using the style library can be found in [reference needed]. Figure 5 As shown. The process of building a style library may include the following steps: Step S71: Obtain the user preference image set and the metadata corresponding to each image in the user preference image set.

[0109] User preference image sets refer to the collection of images selected from a user's stored historical images. These images are typically obtained through aesthetic rating models and user behavior analysis, representing the user's style preferences. These user behaviors may include, but are not limited to, saving, liking, and saving after multiple edits.

[0110] Step S72: Based on the user preference image set and metadata, extract the multimodal features of each image in the user preference image set.

[0111] Multimodal features refer to comprehensive features that can simultaneously characterize the visual content of an image and its shooting environment context, and can be formed by fusing visual features and environmental features. The extraction method of multimodal features is the same as that of style features in step S51 above, and will not be repeated here. Multimodal features can be denoted as F_unified.

[0112] Step S73: Cluster the user preference image set based on multimodal features to obtain at least one style cluster.

[0113] Clustering is an unsupervised machine learning method used to group the multimodal features extracted in step S71. In practice, clustering algorithms such as DBSCAN can be used to divide images in a user's preferred image set into different groups based on the similarity of their multimodal features. Each group is called a style cluster. Images corresponding to multimodal features within the same style cluster have a high degree of similarity in visual style and shooting environment, collectively representing a user's specific color and tonal preferences.

[0114] Step S74: Based on the images corresponding to each style cluster in at least one style cluster, perform low-rank adaptive training on the pre-trained model to obtain at least one style model that corresponds one-to-one with at least one style cluster.

[0115] Low-rank adaptive training is a technique for efficient and lightweight fine-tuning of large pre-trained models. Specifically, it involves adding and training a set of low-rank decomposable matrices, such as matrices A and B in LoRA, next to specific weight matrices of the pre-trained model, so that the model can learn new tasks without updating or storing the entire large original model parameters.

[0116] A style model refers to a lightweight parameter module, such as a set of LoRA weights, obtained by performing low-rank adaptive training on a pre-trained model. Each style model corresponds to a style cluster, which can control the base model to generate images with the specific style represented by that cluster.

[0117] Optionally, for each style cluster, the corresponding style model can be obtained by low-rank adaptive training of the U-Net part of a pre-trained model such as Stable Diffusion. The training process employs a core sparse injection strategy, with the specific steps as follows: The first step is to identify the Decoder (Up-blocks) and Mid-blocks of the pre-trained U-Net model as regions where style information can be injected. These parts have greater control over the texture and color of the generated image.

[0118] The second step involves performing a low-rank adaptation only on the Value projection matrix W_v of the self-attention layers within the locked Up-block and Mid-block. The weights of other linear or convolutional layers remain unchanged.

[0119] The third step is to introduce a pair of trainable low-rank matrices A and B for each selected W_v matrix. Let the original dimension of W_v be d_out × d_in, then the dimension of the introduced matrix A is r × d_in, and the dimension of matrix B is d_out × r, where the rank r is a value much smaller than d_in and d_out, such as 4 or 8.

[0120] In the fourth step, during forward propagation, the actual weights for the Value projection in the modified attention layer become: W_v' = W_v + α·(B×A). Here, α is a scaling factor controlling the adaptation strength. During training, all images within this cluster are used as targets, and the optimization objective is to enable the pre-trained model to generate images with a style consistent with those in this cluster after injecting these LoRA parameters. After training, the LoRA weights {A, B} corresponding to this cluster are saved as an independent style model.

[0121] By employing the sparse injection strategy described above, the images generated by the model can closely approximate the user's preferred style in terms of color and lighting, while preserving the original geometric structure generation capability to the greatest extent possible.

[0122] Step S75: Determine the average value of the multimodal features in each style cluster within at least one style cluster, and use the average value as the index of the corresponding style model.

[0123] An index refers to the index of a style model in the feature space. The index is composed of the average of all multimodal features within the corresponding style cluster; see [link to relevant documentation]. Figure 6 As shown, during online inference, the best-matching style model can be quickly found by calculating the similarity between the style features of the input image to be processed and each index.

[0124] Step S76: Store at least one style model and the indexes of each style model within the at least one style model to obtain a style library. Here, at least one style model and the indexes of each style model within the at least one style model can be stored in a designated storage space, which is the style library.

[0125] By acquiring user-preferred image sets and metadata, and extracting their multimodal features for clustering, this system automates and unsupervisedly summarizes and structures users' complex and diverse historical style preferences, laying a data foundation for subsequent personalized services. By performing low-rank adaptive training on pre-trained models based on each style cluster, corresponding style models are obtained, ensuring that each style model is extremely lightweight with minimal storage and loading overhead, facilitating deployment on various electronic devices. By determining the average multimodal feature value of each style cluster as the index of the corresponding style model, online inference can quickly find several best-matching style models for any input image through simple vector similarity calculations, achieving a fast and accurate mapping from image content and context to personalized style models.

[0126] In some optional embodiments, after step 304 is performed, the following steps may also be performed: Step S81: Obtain the edited image after the user has processed the target image.

[0127] Here, the user's processing of the target image may include, but is not limited to, at least one of the following: color processing and tonal processing. In practice, users can process the target image manually or using other editing tools, and the final saved image after processing can be identified as the retouched image. The retouched image reflects the difference between the result of the target image and the user's final expectation; it represents the user's explicitly expressed new preference or a modification of the original preference.

[0128] Step S82: Use the retouched image and its corresponding metadata as new training data. If the number of new training data of the same style is greater than the threshold, perform low-rank adaptive training on the pre-trained model based on the new training data of the same style to obtain the new style model, and determine the index of the new style model based on the new training data.

[0129] Specifically, multimodal features of the newly added training data can be extracted periodically or when the amount of newly added training data reaches a certain scale, and clustering algorithms can be used to cluster these features. Each cluster represents a new style. If the amount of newly added training data corresponding to a certain cluster exceeds a preset threshold, a new style can be considered to have been stably formed. At this point, an offline training task can be started. This task uses a low-rank adaptive training method, employing the newly added training data corresponding to this cluster, to train a new style model based on a pre-trained diffusion model, thus obtaining the new style model. The average value of the multimodal features in this cluster is used as the index of the new style model.

[0130] Step S83: Store the newly added style model and its index in the style library.

[0131] By acquiring user-edited images as new training data and triggering the training of new style models after the data accumulates to a threshold, the system can proactively discover and learn new style preferences generated by users or not covered in the original library through continuous interaction with users. This ensures that the system can continuously adapt to the latest user needs and meet users' personalized requirements as they use the system over a long period of time.

[0132] Based on one or more of the above embodiments, see [link to relevant documentation]. Figure 7 Upon receiving the user-input image to be processed, the system first extracts visual features from the image and environmental features from its metadata. These visual and environmental features are then fused using a cross-attention mechanism to obtain style features. Next, the similarity between the style features and multiple indices is determined. Several style models with high similarity are selected from the style library, and their parameters are weighted and fused to obtain style parameters. Noise is then added to the VAE-processed image to create a noisy image. The style parameters and the noisy image are then input into a diffusion model, which denoises the noisy image based on the style parameters to obtain a target style reference image. Finally, the image to be processed and the target style reference image are input into a style transfer model to obtain the target image.

[0133] By determining target style parameters based on the image to be processed and its metadata, and generating a target style reference image based on these parameters and the image to be processed, the style selection can be associated with the specific content and shooting context of the image to be processed. This provides a visual reference whose content is compatible with the image to be processed and accurately reflects the target style, thus providing an optimal style example for any image and helping to improve the quality of the generated target image. Furthermore, because the style transfer model can achieve localized and precise color and tone transfer, it can effectively avoid the global coloring problem caused by traditional global filtering methods, generating high-quality images with globally harmonious styles, clear details, and natural colors in key areas, thus further improving the quality of the target image.

[0134] It should be noted that the model training method provided in this application embodiment can be executed by a model training device. This application embodiment uses the execution of the model training method by a model training device as an example to illustrate the model training device provided in this application embodiment.

[0135] like Figure 8As shown, the model training device 800 of this embodiment includes: an acquisition unit 801, used to acquire a training dataset; wherein the training dataset includes multiple triples, each triple including a source image, a style reference image, and a ground truth image, the style reference image and the ground truth image having the same color tone style; and a training unit 802, used to train an initial model based on the training dataset to obtain a style transfer model; wherein, during the training process, a first loss function, a second loss function, and a third loss function are used for supervision, the first loss function indicating pixel-level reconstruction loss, the second loss function indicating style-aware loss, and the third loss function indicating critical region protection loss.

[0136] In some optional implementations of this embodiment, the training unit 802 is further configured to: input the source image and the style reference image into the initial model to obtain a predicted image output by the initial model; determine a first loss value based on the first loss function, the predicted image, and the ground truth image; determine a second loss value based on the second loss function, the predicted image, and the style reference image; determine a third loss value based on the third loss function, the source image, and the predicted image; determine the total loss value of the initial model based on the first loss value, the second loss value, and the third loss value; and adjust the network parameters of the initial model based on the total loss value to obtain the style transfer model.

[0137] In some optional implementations of this embodiment, the initial model includes a first encoder, a second encoder, an attention network, and a decoder; the training unit 802 is further configured to: input the source image into the first encoder to obtain a first image feature; input the style reference image into the second encoder to obtain a second image feature; input the first image feature and the second image feature into the attention network to obtain a fused feature; and input the fused feature into the decoder to obtain the predicted image.

[0138] In some optional implementations of this embodiment, the training unit 802 is further configured to: extract deep features of the predicted image and the style reference image respectively; determine the Gram matrix of the deep features of the predicted image and the style reference image respectively; and determine the second loss value based on the second loss function and the Gram matrix.

[0139] In some optional implementations of this embodiment, the training unit 802 is further configured to: determine key regions in the source image and the prediction image, the key regions including at least one of the following: face region, skin region; determine the color statistical features of the key regions in the source image and the prediction image respectively; and determine the third loss value based on the third loss function and the color statistical features.

[0140] The apparatus provided in the above embodiments of this application, by introducing ground truth images, provides the model with precise detail reconstruction targets. Combined with pixel-level reconstruction loss, it ensures the detail fidelity of the images generated by the model. By introducing style reference images, it provides the model with clear style learning targets. Combined with style-aware loss, it drives the model to learn and transfer color tonal distribution, ensuring the accuracy of style transfer. By introducing key region protection loss, it guides the model to protect the color naturalness of key regions during stylization, effectively avoiding local distortion caused by global adjustments. The above three losses work together to supervise and guide the model to learn to simultaneously optimize the three targets of content detail reconstruction, global style matching, and local color protection when transferring color tonal values. This enables the generation of high-quality images with global style coordination, clear details, and natural colors in key regions, improving the quality of the output images of the style transfer model.

[0141] It should be noted that the image processing method provided in this application embodiment can be executed by an image processing device. This application embodiment uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application embodiment.

[0142] like Figure 9 As shown, the image processing apparatus 900 of this embodiment includes: an acquisition unit 901, configured to acquire an image to be processed and metadata corresponding to the image to be processed; a first determination unit 902, configured to determine style parameters based on the image to be processed and the metadata; a generation unit 903, configured to generate a target style reference image based on the style parameters and the image to be processed; and a processing unit 904, configured to input the image to be processed and the target style reference image into a style transfer model to obtain a target image; wherein the style transfer model is trained by the model training apparatus according to any one of claims 1-5.

[0143] In some optional implementations of this embodiment, the first determining unit 902 is further configured to: generate style features based on the image to be processed and the metadata; determine the similarity between the style features and multiple indices; wherein each of the multiple indices is associated with a style model in a style library, and the style model is obtained by low-rank adaptive training of a pre-trained model based on a user-preferred image set; and perform weighted fusion of the parameters of at least two style models based on the similarity to obtain the style parameters.

[0144] In some optional implementations of this embodiment, the first determining unit 902 is further configured to: extract visual features of the image to be processed through a feature extraction model; wherein the feature extraction model includes a visual encoder and a style classifier, the style classifier includes at least two fully connected layers, and the feature extraction model is supervised during training by a fourth loss function and a fifth loss function, the fourth loss function indicating image-text contrast loss and the fifth loss function indicating style classification loss; encode the metadata to obtain environmental features; and fuse the visual features and the environmental features based on a cross-attention mechanism to obtain the style features.

[0145] In some optional implementations of this embodiment, the apparatus further includes a second determining unit, configured to: acquire the user preference image set and metadata corresponding to each image in the user preference image set; extract multimodal features of each image in the user preference image set based on the user preference image set and the metadata; cluster the user preference image set based on the multimodal features to obtain at least one style cluster; perform low-rank adaptive training on a pre-trained model based on the images corresponding to each style cluster in the at least one style cluster to obtain at least one style model corresponding to the at least one style cluster; determine the average value of the multimodal features in each style cluster in the at least one style cluster, and determine the average value as the index of the corresponding style model; store the at least one style model and the index of each style model in the at least one style model to obtain the style library.

[0146] In some optional implementations of this embodiment, the device further includes a storage unit, configured to: acquire a retouched image after the user has processed the target image; use the retouched image and the corresponding metadata as new training data; when the number of new training data of the same style is greater than a threshold, perform low-rank adaptive training on the pre-trained model based on the new training data of the same style to obtain a new style model, and determine the index of the new style model based on the new training data; and store the new style model and the index of the new style model in the style library.

[0147] In some optional implementations of this embodiment, the generation unit 903 is further configured to: add noise to the image to be processed to obtain a noisy image; input the style parameters and the noisy image into a diffusion model, and perform denoising on the noisy image based on the style parameters through the diffusion model to obtain the target style reference image.

[0148] By determining target style parameters based on the image to be processed and its metadata, and generating a target style reference image based on these parameters and the image to be processed, the style selection can be associated with the specific content and shooting context of the image to be processed. This provides a visual reference whose content is compatible with the image to be processed and accurately reflects the target style, thus providing an optimal style example for any image and helping to improve the quality of the generated target image. Furthermore, because the style transfer model can achieve localized and precise color and tone transfer, it can effectively avoid the global coloring problem caused by traditional global filtering methods, generating high-quality images with globally harmonious styles, clear details, and natural colors in key areas, thus further improving the quality of the target image.

[0149] The model training device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0150] The model training device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0151] The model training device provided in this application embodiment can achieve... Figure 1or Figure 3 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0152] Optionally, such as Figure 10 As shown, this application embodiment also provides an electronic device 1000, including a processor 1001 and a memory 1002. The memory 1002 stores a program or instructions that can run on the processor 1001. When the program or instructions are executed by the processor 1001, they implement the various steps of the above-described model training method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0153] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0154] Figure 11 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application. The electronic device 1100 includes, but is not limited to, components such as: radio frequency unit 1101, network module 1102, audio output unit 1103, input unit 1104, sensor 1105, display unit 1106, user input unit 1107, interface unit 1108, memory 1109, and processor 1110.

[0155] Those skilled in the art will understand that the electronic device 1100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 11 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here. The processor 1110 is used to acquire a training dataset. The training dataset includes multiple triples, each triple including a source image, a style reference image, and a ground truth image. The style reference image and the ground truth image have the same color tone style. An initial model is trained based on the training dataset to obtain a style transfer model. During the training process, a first loss function, a second loss function, and a third loss function are used for supervision. The first loss function indicates pixel-level reconstruction loss, the second loss function indicates style-aware loss, and the third loss function indicates key region protection loss.

[0156] By introducing ground truth images, the model is provided with precise detail reconstruction targets. Combined with pixel-level reconstruction loss, this ensures the detail fidelity of the generated images. Introducing style reference images provides a clear style learning objective. Combined with style-aware loss, this drives the model to learn and transfer color tonal distribution, ensuring the accuracy of style transfer. Introducing key region preservation loss guides the model to protect the naturalness of key region colors during stylization, effectively avoiding local distortion caused by global adjustments. These three losses work together to supervise and guide the model to learn to simultaneously optimize content detail reconstruction, global style matching, and local color preservation when transferring color tonal values. This results in high-quality images with harmonious global style, clear details, and natural colors in key regions, improving the quality of the style transfer model's output images.

[0157] In some optional implementations of this embodiment, the processor 1110 is further configured to input the source image and the style reference image into the initial model to obtain a predicted image output by the initial model; determine a first loss value based on the first loss function, the predicted image, and the ground truth image; determine a second loss value based on the second loss function, the predicted image, and the style reference image; determine a third loss value based on the third loss function, the source image, and the predicted image; determine the total loss value of the initial model based on the first loss value, the second loss value, and the third loss value; and adjust the network parameters of the initial model based on the total loss value to obtain the style transfer model.

[0158] In some optional implementations of this embodiment, the initial model includes a first encoder, a second encoder, an attention network, and a decoder; the processor 1110 is further configured to input the source image into the first encoder to obtain a first image feature; input the style reference image into the second encoder to obtain a second image feature; input the first image feature and the second image feature into the attention network to obtain a fused feature; and input the fused feature into the decoder to obtain the predicted image.

[0159] In some optional implementations of this embodiment, the processor 1110 is further configured to extract deep features of the predicted image and the style reference image respectively; determine the Gram matrix of the deep features of the predicted image and the deep features of the style reference image respectively; and determine the second loss value based on the second loss function and the Gram matrix.

[0160] In some optional implementations of this embodiment, the processor 1110 is further configured to determine key regions in the source image and the prediction image, the key regions including at least one of the following: face region, skin region; determine the color statistical features of the key regions in the source image and the prediction image respectively; and determine the third loss value based on the third loss function and the color statistical features.

[0161] In addition, the processor 1110 can also be used to acquire the image to be processed and the metadata corresponding to the image to be processed; determine style parameters based on the image to be processed and the metadata; generate a target style reference image based on the style parameters and the image to be processed; input the image to be processed and the target style reference image into the style transfer model to obtain the target image; wherein the style transfer model is trained by the model training method of any one of claims 1-5.

[0162] By determining target style parameters based on the image to be processed and its metadata, and generating a target style reference image based on these parameters and the image to be processed, the style selection can be associated with the specific content and shooting context of the image to be processed. This provides a visual reference whose content is compatible with the image to be processed and accurately reflects the target style, thus providing an optimal style example for any image and helping to improve the quality of the generated target image. Furthermore, because the style transfer model can achieve localized and precise color and tone transfer, it can effectively avoid the global coloring problem caused by traditional global filtering methods, generating high-quality images with globally harmonious styles, clear details, and natural colors in key areas, thus further improving the quality of the target image.

[0163] In some optional implementations of this embodiment, the processor 1110 is further configured to generate style features based on the image to be processed and the metadata; determine the similarity between the style features and multiple indices; wherein each of the multiple indices is associated with a style model in a style library, and the style model is obtained by low-rank adaptive training of a pre-trained model based on a user-preferred image set; and based on the similarity, perform weighted fusion of the parameters of at least two style models to obtain the style parameters.

[0164] In some optional implementations of this embodiment, the processor 1110 is further configured to extract visual features of the image to be processed through a feature extraction model; wherein the feature extraction model includes a visual encoder and a style classifier, the style classifier includes at least two fully connected layers, and the feature extraction model is supervised during training by a fourth loss function and a fifth loss function, the fourth loss function indicating image-text contrast loss and the fifth loss function indicating style classification loss; the metadata is encoded to obtain environmental features; and the visual features and the environmental features are fused based on a cross-attention mechanism to obtain the style features.

[0165] In some optional implementations of this embodiment, the processor 1110 is further configured to: acquire the user preference image set and metadata corresponding to each image in the user preference image set; extract multimodal features of each image in the user preference image set based on the user preference image set and the metadata; cluster the user preference image set based on the multimodal features to obtain at least one style cluster; perform low-rank adaptive training on the pre-trained model based on the images corresponding to each style cluster in the at least one style cluster to obtain at least one style model corresponding to the at least one style cluster; determine the average value of the multimodal features in each style cluster in the at least one style cluster, and determine the average value as the index of the corresponding style model; and store the at least one style model and the index of each style model in the at least one style model to obtain the style library.

[0166] In some optional implementations of this embodiment, the processor 1110 is further configured to acquire the retouched image after the user has processed the target image; use the retouched image and the corresponding metadata as new training data; when the number of new training data of the same style is greater than a threshold, perform low-rank adaptive training on the pre-trained model based on the new training data of the same style to obtain a new style model; and determine the index of the new style model based on the new training data; and store the new style model and the index of the new style model in the style library.

[0167] In some optional implementations of this embodiment, the processor 1110 is further configured to add noise to the image to be processed to obtain a noisy image; input the style parameters and the noisy image to a diffusion model, and perform denoising on the noisy image based on the style parameters through the diffusion model to obtain the target style reference image.

[0168] It should be understood that, in this embodiment, the input unit 1104 may include a graphics processing unit (GPU) 11041 and a microphone 11042. The GPU 11041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1106 may include a display panel 11061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1107 includes at least one of a touch panel 11071 and other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0169] The memory 1109 can be used to store software programs and various data. The memory 1109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1109 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0170] Processor 1110 may include one or more processing units; optionally, processor 1110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 1110.

[0171] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described model training method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0172] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0173] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described model training method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0174] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0175] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the model training method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0176] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0177] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0178] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A model training method, characterized in that, The method includes: Obtain a training dataset; wherein the training dataset includes multiple triples, each triple including a source image, a style reference image and a ground truth image, the style reference image and the ground truth image having the same color tone style; The initial model is trained based on the training dataset to obtain a style transfer model. During the training process, a first loss function, a second loss function, and a third loss function are used for supervision. The first loss function indicates pixel-level reconstruction loss, the second loss function indicates style-aware loss, and the third loss function indicates key region protection loss.

2. The method according to claim 1, characterized in that, The process of training the initial model based on the training dataset to obtain the style transfer model includes: The source image and the style reference image are input into the initial model to obtain the predicted image output by the initial model; Based on the first loss function, the predicted image, and the ground truth image, a first loss value is determined; A second loss value is determined based on the second loss function, the predicted image, and the style reference image; Based on the third loss function, the source image, and the predicted image, a third loss value is determined; Based on the first loss value, the second loss value, and the third loss value, the total loss value of the initial model is determined; The network parameters of the initial model are adjusted based on the total loss value to obtain the style transfer model.

3. The method according to claim 2, characterized in that, The initial model includes a first encoder, a second encoder, an attention network, and a decoder; the step of inputting the source image and the style reference image into the initial model to obtain the predicted image output by the initial model includes: The source image is input into the first encoder to obtain the first image features; The style reference image is input into the second encoder to obtain the second image features; The first image features and the second image features are input into the attention network to obtain fused features; The fused features are input into the decoder to obtain the predicted image.

4. The method according to claim 2, characterized in that, The step of determining the second loss value based on the second loss function, the predicted image, and the style reference image includes: Deep features are extracted from the predicted image and the style reference image, respectively; The Gram matrices of the deep features of the predicted image and the deep features of the style reference image are determined respectively. The second loss value is determined based on the second loss function and the Gram matrix.

5. The method according to claim 2, characterized in that, The step of determining the third loss value based on the third loss function, the source image, and the predicted image includes: Determine key regions in the source image and the predicted image, wherein the key regions include at least one of the following: face region, skin region; Determine the color statistical features of the key region in the source image and the predicted image, respectively; The third loss value is determined based on the third loss function and the color statistical features.

6. An image processing method, characterized in that, The method includes: Obtain the image to be processed and the metadata corresponding to the image to be processed; Based on the image to be processed and the metadata, determine the style parameters; Based on the style parameters and the image to be processed, a target style reference image is generated; The image to be processed and the target style reference image are input into the style transfer model to obtain the target image; wherein the style transfer model is trained by the model training method of any one of claims 1-5.

7. The method according to claim 6, characterized in that, The process of determining style parameters based on the image to be processed and the metadata includes: Based on the image to be processed and the metadata, style features are generated; Determine the similarity between the style features and multiple indices; wherein each of the multiple indices is associated with a style model in a style library, the style model being obtained by low-rank adaptive training of a pre-trained model based on a set of user-preferred images; Based on the similarity, the parameters of at least two style models are weighted and fused to obtain the style parameters.

8. The method according to claim 7, characterized in that, The generation of style features based on the image to be processed and the metadata includes: Visual features of the image to be processed are extracted using a feature extraction model; wherein, the feature extraction model includes a visual encoder and a style classifier, the style classifier includes at least two fully connected layers, and the feature extraction model is supervised during training by a fourth loss function and a fifth loss function, the fourth loss function indicating image-text contrast loss, and the fifth loss function indicating style classification loss; The metadata is encoded to obtain environmental characteristics; The style features are obtained by fusing the visual features and the environmental features based on the cross-attention mechanism.

9. The method according to claim 7, characterized in that, Before determining the style parameters based on the image to be processed and the metadata, the method further includes: Obtain the user preference image set and the metadata corresponding to each image in the user preference image set; Based on the user preference image set and the metadata, extract multimodal features of each image in the user preference image set; Based on the multimodal features, the user preference image set is clustered to obtain at least one style cluster; Based on the images corresponding to each style cluster in the at least one style cluster, the pre-trained model is subjected to low-rank adaptive training to obtain at least one style model that corresponds one-to-one with the at least one style cluster. Determine the average value of the multimodal features in each style cluster within the at least one style cluster, and use the average value as the index of the corresponding style model; The style library is obtained by storing the at least one style model and the index of each style model in the at least one style model.

10. The method according to claim 9, characterized in that, After obtaining the target image, the method further includes: Obtain the edited image after the user has processed the target image; The retouched image and its metadata are used as new training data. When the number of new training data of the same style is greater than a threshold, the pre-trained model is subjected to low-rank adaptive training based on the new training data of the same style to obtain a new style model. The index of the new style model is determined based on the new training data. The newly added style model and its index are stored in the style library.

11. The method according to claim 6, characterized in that, The step of generating a target style reference image based on the style parameters and the image to be processed includes: The image to be processed is subjected to noise processing to obtain a noisy image; The style parameters and the noisy image are input into a diffusion model, and the diffusion model performs denoising on the noisy image based on the style parameters to obtain the target style reference image.

12. A model training device, characterized in that, The device includes: An acquisition unit is used to acquire a training dataset; wherein the training dataset includes multiple triples, each triple including a source image, a style reference image and a ground truth image, and the style reference image and the ground truth image have the same color tone style; The training unit is used to train the initial model based on the training dataset to obtain the style transfer model. During the training process, a first loss function, a second loss function, and a third loss function are used for supervision. The first loss function indicates pixel-level reconstruction loss, the second loss function indicates style-aware loss, and the third loss function indicates key region protection loss.

13. An image processing apparatus, characterized in that, The device includes: The acquisition unit is used to acquire the image to be processed and the metadata corresponding to the image to be processed; The first determining unit is used to determine style parameters based on the image to be processed and the metadata; The generation unit is used to generate a target style reference image based on the style parameters and the image to be processed; A processing unit is configured to input the image to be processed and the target style reference image into a style transfer model to obtain a target image; wherein the style transfer model is trained by the model training device according to any one of claims 1-5.

14. An electronic device, characterized in that, It includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the steps of the model training method as claimed in any one of claims 1-5 or the steps of the image processing method as claimed in any one of claims 6-11.

Images