Model training method, image processing method and device

Abstract

The invention discloses a model training method and device and an image processing method and device, and belongs to the field of image processing. The model training method comprises the steps of obtaining a training sample set; fusing the content image and the sample text by using a multi-modal fusion module of the first image processing model to obtain a hidden space vector of the multi-modal fusion module; decoding the hidden space vector by using an image decoder of the first image processing model to obtain a first emotion image output by the image decoder; the image decoder is obtained by training by taking a first vector corresponding to the content image as input and taking the reference emotion image as output; according to the first image feature of the first emotion image, determining an emotion distribution feature of the first emotion image; determining a first loss function value of an emotional distribution loss function according to the emotional distribution characteristics of the first emotional image and the emotional distribution characteristics of the reference emotional image; and training a multi-modal fusion module according to the first loss function value to obtain a second image processing model.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to a model training method, an image processing method and a device thereof. Background Art

[0002] With the development of technology, social media platforms have become a ubiquitous tool for people to build and deepen emotional connections. Users can share their experiences, thoughts, and emotions through posting. While text can directly convey emotion, images are more effective in capturing audiences' attention and stimulating deeper emotional resonance. Therefore, driven by both technological advancements and user needs, how to translate the emotions expressed in text into images has become a crucial research topic worthy of attention.

[0003] The current solutions cannot accurately map the emotions in text to images, and it is difficult to meet users' emotion mapping needs. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a model training method, an image processing method and a device thereof, which can accurately map the emotions in the text to the image.

[0005] In a first aspect, an embodiment of the present application provides a model training method, comprising:

[0006] Obtaining a training sample set, the training sample set including at least two groups of training samples, each group of training samples including a content image, a sample text, and a reference emotion image corresponding to the sample text, the reference emotion image being used to reflect the emotion in the sample text;

[0007] Using the multimodal fusion module of the first image processing model to fuse the content image and the sample text, and obtain a latent space vector of the multimodal fusion module;

[0008] Decoding the latent space vector using the image decoder of the first image processing model to obtain a first emotion image output by the image decoder; wherein the image decoder is trained using the first vector corresponding to the content image as input and the reference emotion image as output;

[0009] determining an emotion distribution feature of the first emotion image based on the first image feature of the first emotion image;

[0010] Determining a first loss function value of an emotion distribution loss function according to the emotion distribution characteristics of the first emotion image and the emotion distribution characteristics of the reference emotion image;

[0011] The multimodal fusion module is trained according to the first loss function value to obtain a second image processing model.

[0012] In a second aspect, an embodiment of the present application provides an image processing method, comprising:

[0013] Acquire a first image and a first text;

[0014] Encoding the first image to obtain an encoding vector;

[0015] Processing the first text according to the sentiment in the first text to obtain a second text;

[0016] The encoding vector and the second text are input into a second image processing model to obtain a second image; wherein the second image processing model is trained according to the model training method of the first aspect.

[0017] In a third aspect, an embodiment of the present application provides a model training device, comprising:

[0018] An acquisition module is used to acquire a training sample set, where the training sample set includes at least two groups of training samples, each group of training samples includes a content image, a sample text, and a reference emotion image corresponding to the sample text, where the reference emotion image is used to reflect the emotion in the sample text;

[0019] A multimodal fusion module is used to fuse the content image and sample text and output a latent space vector. The multimodal fusion module belongs to the first image processing model.

[0020] an image decoder, configured to decode the latent space vector and output a first emotion image; wherein the image decoder is trained using the first vector corresponding to the content image as input and the reference emotion image as output, and the image decoder belongs to the first image processing model;

[0021] a determination module, configured to determine an emotion distribution feature of the first emotion image based on a first image feature of the first emotion image; and determine a first loss function value of an emotion distribution loss function based on the emotion distribution feature of the first emotion image and the emotion distribution feature of a reference emotion image;

[0022] A training module is used to train the multimodal fusion module according to the first loss function value to obtain a second image processing model.

[0023] In a fourth aspect, an embodiment of the present application provides an image processing device, including:

[0024] An acquisition module, configured to acquire a first image and a first text;

[0025] An encoding module, configured to encode the first image to obtain an encoding vector;

[0026] A processing module is used to process the first text according to the emotion in the first text to obtain a second text; input the encoding vector and the second text into a second image processing model to obtain a second image; wherein the second image processing model is trained according to the model training method of the first aspect.

[0027] In a fifth aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect or the steps of the method described in the second aspect are implemented.

[0028] In a sixth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect or the steps of the method described in the second aspect are implemented.

[0029] In the seventh aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the method described in the first aspect, or the steps of the method described in the second aspect.

[0030] In an eighth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect, or the steps of the method described in the second aspect.

[0031] Before training the first image processing model, this embodiment first trains the image decoder of the first image processing model based on the content image and the reference emotion image that can reflect the emotion, ensuring that the image decoder can accurately understand the relationship between emotions and improve the image decoder's ability to understand emotions. Then, based on the trained image decoder and the multimodal fusion module and training samples of the first image processing model, the loss function value of the emotion distribution of the first emotion image and the reference emotion image output by the first image processing model is determined, and the multimodal fusion module is trained based on the loss function value. That is, this embodiment combines the image decoder's understanding of emotion and the emotion distribution loss of the first emotion image and the reference emotion image to train the model from multiple levels, so that the model can more accurately understand the emotion in the text, and then more accurately map the emotion to the image to meet the user's emotional expression needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0033] Figure 2A schematic structural diagram of a first image processing model provided in an embodiment of the present application;

[0034] Figure 3 A schematic structural diagram of another first image processing model provided in an embodiment of the present application;

[0035] Figure 4 A flowchart of another model training method provided in an embodiment of the present application;

[0036] Figure 5 A flowchart of another model training method provided in an embodiment of the present application;

[0037] Figure 6 A flowchart of another model training method provided in an embodiment of the present application;

[0038] Figure 7 A schematic diagram of an emotion wheel provided in an embodiment of the present application;

[0039] Figure 8 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0040] Figure 9 A schematic structural diagram of an image processing device provided in an embodiment of the present application;

[0041] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0042] Figure 11 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0044] The terms "first," "second," and the like in the specification of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. Furthermore, the term "and / or" in this specification indicates at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0045] The model training method, image processing model and device provided in the embodiments of the present application are described in detail below with reference to specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0046] Figure 1 A flowchart of a model training method provided in an embodiment of the present application, which can be applied to electronic devices with data processing functions such as laptops, tablets, servers, etc.

[0047] like Figure 1 As shown, the model training method may include the following steps:

[0048] S110 , obtaining a training sample set, where the training sample set includes at least two groups of training samples, each group of training samples includes a content image, a sample text, and a reference emotion image corresponding to the sample text, where the reference emotion image is used to reflect the emotion in the sample text.

[0049] S120 , using the multimodal fusion module of the first image processing model to fuse the content image and the sample text, and obtain a latent space vector output by the multimodal fusion module.

[0050] S130. Decode the latent space vector using the image decoder of the first image processing model to obtain a first emotion image output by the image decoder.

[0051] The image decoder is trained by taking the first vector corresponding to the content image as input and the reference emotion image as output.

[0052] S140: Determine an emotion distribution feature of the first emotion image according to the first image feature of the first emotion image.

[0053] S150. Determine a first loss function value of the emotion distribution loss function according to the emotion distribution characteristics of the first emotion image and the emotion distribution characteristics of the reference emotion image.

[0054] S160. Train the multimodal fusion module according to the first loss function value to obtain a second image processing model.

[0055] Before training the first image processing model, this embodiment first trains the image decoder of the first image processing model based on the content image and the reference emotion image that can reflect the emotion, ensuring that the image decoder can accurately understand the relationship between emotions and improve the image decoder's ability to understand emotions. Then, based on the trained image decoder and the multimodal fusion module and training samples of the first image processing model, the loss function value of the emotion distribution of the first emotion image and the reference emotion image output by the first image processing model is determined, and the multimodal fusion module is trained based on the loss function value. That is, this embodiment combines the image decoder's understanding of emotion and the emotion distribution loss of the first emotion image and the reference emotion image to train the model from multiple levels, so that the model can more accurately understand the emotion in the text, and then more accurately map the emotion to the image to meet the user's emotional expression needs.

[0056] The above steps are explained in detail below:

[0057] In S110 , the content image may be any RGB image. This embodiment does not limit the objects included in the content image. For example, the content image may be a landscape image, a person image, or the like.

[0058] The sample text may be text used to train the first image processing model. The sample text may contain one or more emotions. For example, the emotion may be a concrete emotion, such as happiness, sadness, or joy. For example, a sample text may be: "Zhang San won first place in this competition and he is very happy." For example, the emotion may also be an abstract emotion, i.e., the sample text does not contain a clear emotion word, and the emotion needs to be determined based on the specific content of the sample text.

[0059] For example, a portion of the sample text may be selected from locally stored text, or a portion of the sample text may be selected from the Internet using a web crawler, or may be obtained in other ways. This embodiment does not limit the length of the sample text, and for example, it may include N sentences, where N is an integer greater than or equal to 1.

[0060] In order to more deeply understand the deep emotions in the text and improve the model's ability to understand emotions, for example, the initial text can be obtained first, and then the emotions expressed in the initial text can be identified using a large language model. The emotions identified by the large language model are then used as additional context information to process the initial text to obtain a text supplemented with emotional context cues as a sample text.

[0061] The reference emotion image may be an RGB image that accurately expresses the emotion in the sample text, and each sample text may correspond to a reference emotion image.

[0062] In S120, the first image processing model is used to map the emotion in the text to the image, thereby obtaining an image that can express the emotion in the text. For example, during the training phase, the first image processing model can map the emotion in the sample text to the content image to obtain a first emotion image. This embodiment does not limit the specific structure of the first image processing model, and for example, it may include a multimodal fusion module, an image decoder, an image encoder, a text encoder, etc.

[0063] Figure 2 A structural diagram of a first image processing model is exemplarily provided. Figure 2 Taking the first image processing model 200 including the multimodal fusion module 201 and the image decoder 202 as an example, the input of the multimodal fusion module 201 can be the content image and the sample text, or the result of encoding the content image and the sample text. In other words, the input of the multimodal fusion module 201 can be the result of encoding the content image and the result of encoding the sample text. In other words, the input of the multimodal fusion module 201 can also be the vectors corresponding to the content image and the sample text.

[0064] In some embodiments, the multimodal fusion module 201 may also first encode the content image and the sample text separately to obtain corresponding vectors, and then process the vectors corresponding to the content image and the vectors corresponding to the sample text to obtain the latent space vector z′0. The latent space vector z′0 is the representation of the fusion result of the content image and the sample text in the latent space, that is, the multimodal fusion module 210 in this embodiment can convert the result of the fusion of the content image and the sample text into a latent space representation. In this way, the first image processing model can learn the potential features and rules of the content image and the sample text, thereby improving the performance of the first image processing model. The latent space vector z′0 is an image vector containing emotional information.

[0065] For example, the multimodal fusion module 201 may be a deep neural network model such as a deep generative model, which may be a diffusion model, or other model structures.

[0066] The image decoder 202 is used to map the latent space vector z′0 to the pixel space to obtain the first emotion image. The image decoder 202 may include multiple convolutional layers and multiple pooling layers. The specific number of convolutional layers and pooling layers can be set according to actual needs.

[0067] The ability of the image decoder 202 to understand emotions directly affects the accuracy of the first emotional image. In order to make the image decoder output a more accurate first emotional image, this embodiment can first use the first vector corresponding to the content image as input and the reference emotional image as output to train the image decoder 202 separately to improve the image decoder 202's ability to understand emotions and ensure that the image decoder 202 can accurately understand the relationship between emotions. The specific training process of the image decoder 202 can be found in the following embodiments. After the training of the image decoder 202 is completed, its parameters are fixed. Further training the multimodal fusion module 201 based on the pre-trained image decoder 202 can improve the training effect of the first image processing model 200.

[0068] In S130 , the image decoder may decode the latent space vector z′0 and map it to the pixel space to obtain a first emotional image with an emotional filter.

[0069] In S140, the first image feature may be a feature of the extracted first emotional image, for example, it may include at least one of the following: color feature, texture feature, style feature, block feature, where the block feature is an image feature obtained by extracting features from a part of the first emotional image.

[0070] Exemplarily, the color histograms of the R channel, the G channel, and the B channel of the first emotion image may be calculated respectively, and the color histograms of the channels may be spliced to obtain the color features of the first emotion image.

[0071] For example, a gray-level co-occurrence matrix (GLCM) may be calculated based on the first emotional image, and then the GLCM may be normalized to obtain the angular second moment, which is used as the texture feature of the first emotional image.

[0072] The style features are used to characterize the visual artistic or aesthetic attributes of the first emotional image. For example, a pre-trained convolutional neural network can be used to extract the style features of the first emotional image. For example, the convolutional neural network can be a pre-trained deep convolutional neural network (Visual Geometry Group, VGG). Specifically, the first emotional image can be input into a VGG network to obtain candidate image features obtained by passing the first emotional image through the target network layer of the VGG network; and the style features of the first emotional image are determined based on the statistical characteristics of the candidate image features.

[0073] The target network layer can be any network layer included in a VGG network. Taking a VGG network including three layers as an example, for example, candidate image features obtained by each network layer of the first emotional image can be extracted. Statistical features can include, for example, mean and variance. Taking the example of statistical features including mean and variance, for example, the mean and variance of the candidate image features obtained by each network layer can be calculated, thereby obtaining three sets of means and variances, which can be used as style features of the first emotional image.

[0074] The image block feature is used to represent local image features of the first emotional image, and may include, for example, local color features, texture features, style features, etc. The image block feature can eliminate the influence of content.

[0075] For example, the first emotional image can be randomly cropped into multiple blocks, and for each block, the color features, texture features, and style features corresponding to each block are obtained according to the above method. In this way, the global image features and local image features of the first emotional image can be obtained.

[0076] This embodiment can determine the emotional distribution characteristics of the first emotional image based on image features of different dimensions, taking into account that different users may have multiple emotions when viewing the same image, that is, taking into account the perceptual preferences, cognitive styles and aesthetic tendencies of different users. This enables the first image processing model to more accurately analyze the emotions aroused by the first emotional image, thereby improving the training effect of the first image processing model.

[0077] The emotion distribution feature is used to characterize the emotion distribution characteristics of the first emotion image. For example, the emotion distribution characteristics of the first emotion image can be characterized by the emotion values of different emotions. For example, the emotion distribution characteristics of the first emotion image can be determined based on the first image features and an emotion prediction model. The emotion prediction model can be, for example, a classifier or a neural network model. Different first image features correspond to different emotion prediction models.

[0078] For example, based on the color features of the first emotional image and the emotion prediction model corresponding to the color features, the emotion distribution features of the first emotional image can be determined. Based on the texture features of the first emotional image and the emotion prediction model corresponding to the texture features, the emotion distribution features of the first emotional image can be determined. Based on the style features of the first emotional image and the emotion prediction model corresponding to the style features, the emotion distribution features of the first emotional image can be determined. Based on the block features of the first emotional image and the emotion prediction model corresponding to the block features, the emotion distribution features of the first emotional image can be determined. In this way, the emotion distribution features of the first emotional image can be predicted from different dimensions.

[0079] For example, the emotion distribution features obtained for different dimensions can be weighted and summed to obtain the final emotion distribution features of the first emotion image. For example, for a certain type of emotion, each dimension can predict the emotion value of that type of emotion. The weighted sum of the emotion values predicted for each dimension can be used to obtain the final emotion value of that type of emotion. The calculation process for other types of emotions is similar, thereby obtaining the final emotion distribution features of the first emotion image.

[0080] In S150, based on the emotion distribution characteristics of the first emotion image and the emotion distribution characteristics of the reference emotion image, the emotion distribution loss of the first emotion image can be determined to obtain a first loss function value It provides a basis for subsequent model training.

[0081] The specific form of the sentiment distribution loss function can be set or selected according to actual needs and is not limited in this embodiment. Based on the sentiment distribution loss function, the first image processing model can be trained to learn potential sentiment information, thereby improving the training effect of the first image processing model.

[0082] In S160, according to the first loss function value The multimodal fusion module is trained to obtain a second image processing model, which can accurately map the sentiment in any text to an RGB image.

[0083] For example, when the number of iterations reaches the threshold, or the first loss function value When it reaches the minimum, the training is terminated.

[0084] Figure 3 A structural diagram of a first image processing model is exemplarily provided. Figure 3 Taking the multimodal fusion module 201 as an example of a diffusion model, it includes a noise prediction unit 2011 and a denoising unit 2012. Based on the properties of the forward process in the diffusion model, this embodiment gradually adds noise to the first vector corresponding to the content image, and then gradually removes the noise, ultimately obtaining a latent space vector z′0 with clear content.

[0085] Based on this, in some embodiments, the above S120 can be refined into the following S1201-S1203.

[0086] S1201: Add first noise to a first vector to obtain a noise sample.

[0087] For example, the first noise may be Gaussian noise sampled with a mean of 0 and a variance of 1. The first vector may be a vector obtained by encoding the content image. For example, the content images used for model training may be batch processed in advance to obtain the first vector corresponding to each content image, facilitating subsequent direct use and thereby improving model training efficiency.

[0088] Exemplarily, the first vector and the first noise may be linearly combined using formula (1) to obtain a noise sample.

[0089]

[0090] Among them, z t represents the noise sample, z0 represents the first vector, ε t represents the first noise, α t is a hyperparameter that controls the noise intensity.

[0091] S1202: Input the second vector corresponding to the noise sample and the sample text into a noise prediction unit, and use the noise prediction unit to predict the noise in the noise sample to obtain predicted noise.

[0092] The second vector is obtained by encoding the sample text. For example, the sample text can be encoded using a multimodal pre-training model (Contrastive Language-Image Pre-training, CLIP) to obtain the second vector y tex .

[0093] For example, the noise sample z t and the second vector y tex Input the noise prediction unit, which performs noise prediction to obtain the predicted noise Among them, the second vector y tex When inputting the noise prediction unit, it can be injected into the noise prediction unit through the cross attention mechanism.

[0094] The noise prediction unit adds noise step by step and removes noise step by step, that is, according to the noise sample z in step t t , we can predict the noise added in step t-1 and get z t-1 Specifically, z t-1 It can be expressed by the following formula (2):

[0095]

[0096] Among them, ∈ θ (z t ,t,y tex ,f cnt) represents the prediction function corresponding to the noise prediction unit, f cnt represents the first vector, σ t is the standard deviation of the noise, Used to control the noise intensity, α s Indicates the noise intensity control parameter corresponding to each step. t is the number of iterations, for example, t = 50. When training the model, in order to speed up the training efficiency of the model, it is possible to directly t-1 Get the latent space vector z′0, no longer need to be based on z t-1 Get z t-2 , until a noise-free result is obtained.

[0097] S1203: Input the predicted noise and the noise sample into a denoising unit, and use the denoising unit to perform denoising on the noise sample to obtain a latent space vector.

[0098] According to the predicted noise, the noise samples are denoised to obtain the latent space vector, which provides a basis for the generation of the first emotional image.

[0099] This embodiment performs gradual denoising and gradual denoising on the content image based on the diffusion model, which can improve the quality of the generated image and maintain the content and structure of the content image as much as possible.

[0100] In some embodiments, before S160, the model training method may further include the following steps:

[0101] Determining a noise loss function value of a noise loss function according to the first noise and the predicted noise;

[0102] Accordingly, the above S160 may include the following steps:

[0103] The multimodal fusion module is trained according to the first loss function value and the noise loss function value to obtain a second image processing model.

[0104] For example, the noise loss function value can be calculated using a mean square error function, that is, the noise loss function can be a mean square error loss function. The specific calculation process of the noise loss function value can be referred to the following formula (3):

[0105]

[0106] in, is the noise loss function value.

[0107] Exemplarily, the first loss function value and the noise loss function value may be weightedly summed to obtain a target loss function value, and the multimodal fusion module may be trained based on the target loss function value.

[0108] For example, in, represents the target loss function value, λ dm and λ ed are the weights corresponding to the noise loss function value and the first loss function value respectively. Each weight can be determined according to the accuracy of the noise prediction result and the sentiment distribution feature. The higher the accuracy, the greater the corresponding weight. The weight can also be set according to experience or needs, such as λ dm =λ ed .

[0109] Based on the sentiment distribution loss, this embodiment further considers the noise prediction loss, and trains the first image processing model in combination with the sentiment distribution loss and the noise prediction loss, so that the first image processing model can not only ensure that the generated image can accurately express the emotion in the text, but also make the generated image clearer, that is, taking into account both the accuracy of the emotion and the clarity of the image.

[0110] Taking the example that the first image feature includes at least two, illustratively, the first image feature may include at least two of color feature, texture feature, style feature and block feature. Taking the example that the emotion distribution feature includes the emotion value of the target emotion, Figure 4 A flowchart of a model training method is provided as an example. Figure 4 and Figure 1 The difference is that Figure 1 The S140 in the Figure 4 S410-S420 in.

[0111] S410: Input the first image feature into the corresponding emotion prediction model to obtain a candidate emotion value of the target emotion in the first emotion image output by the emotion prediction model.

[0112] Among them, different first image features correspond to different emotion prediction models, and the emotion prediction model is trained with the reference image features of the reference emotion image as input and the true emotion value of the target emotion in the reference emotion image as output.

[0113] The reference emotion feature is an image feature extracted from the reference emotion image, corresponding to the first image feature. For example, if the first image includes a color feature, the reference emotion feature also includes a color feature. There can be one or more target emotions. This embodiment uses the example of multiple target emotions so that it can be applied to more scenarios and accurately map different emotions to images. For example, the target emotions can include the eight common categories: fear, excitement, surprise, satisfaction, happiness, anger, disgust, and sadness.

[0114] Each emotion prediction model corresponding to the first image feature can predict the emotion values of the above eight categories of emotions and obtain the emotion distribution characteristics. The training process of each emotion prediction model is similar. The training process of the emotion prediction model corresponding to the color feature is described below.

[0115] Exemplarily, the color features of the reference emotional image can be input into the corresponding emotion prediction model to obtain the predicted emotional values of each type of emotion. Then, the loss function value of the predicted emotional value and the true emotional value of each type of emotion is calculated, and the emotion prediction model is trained based on the loss function value. Exemplarily, the loss function value can be implemented based on the relative entropy (Kullback-Leibler divergence, KL divergence) loss function. The emotion prediction model after training can predict the emotional distribution characteristics of any image based on the color features of the image. The training process of the emotion prediction model corresponding to other first image features is similar and will not be repeated here.

[0116] Therefore, for each first image feature, the emotion values of various emotions of the first emotion image can be predicted based on the pre-trained emotion prediction model. In this embodiment, these emotion values are recorded as candidate emotion values.

[0117] S420. Determine the emotion value of the target emotion in the first emotion image based on the candidate emotion values corresponding to each first image feature and the confidence of each emotion prediction model.

[0118] Taking the case where the first image features include two as an example, two groups of candidate emotion values can be obtained, each group of candidate emotion values includes 8.

[0119] The confidence level of an emotion prediction model can be used to characterize the accuracy of the prediction results of the emotion prediction model. The confidence level can be determined in advance based on experiments or other methods. Based on the confidence level, the weights corresponding to each emotion prediction model can be determined. Based on the weights corresponding to the emotion prediction models and the prediction results of each emotion prediction model, the final emotion value of each emotion in the first emotion image can be determined.

[0120] For example, the final emotion value of each emotion in the first emotion image can be expressed as follows (4):

[0121]

[0122] Wherein, i represents the i-th emotion. In some embodiments, i=8, and j represents the number of first image features or emotion prediction models. For example, if the first image features include two, then j=2. out ) i represents the final sentiment value of the i-th category of the first sentiment image, φ j (I out ) irepresents the sentiment value of the i-th category of sentiment predicted by the j-th sentiment prediction model, w j Denotes the weight of the jth emotion prediction model. Based on the above formula, the final emotion distribution feature of the first emotion image can be obtained.

[0123] This embodiment simulates the process of people viewing images from different perspectives, predicts the emotional distribution characteristics of the first emotional image from different perspectives, and obtains the final emotional distribution characteristics based on the emotional distribution characteristics prediction results obtained from different perspectives. It takes into account that different users have different perceptual preferences, cognitive styles and aesthetic tendencies, which is more in line with people's understanding of emotions.

[0124] Taking the case where the target emotion includes at least two types of emotions as an example, based on the above embodiment, in some embodiments, S150 may include the following steps:

[0125] For each type of emotion, determining an emotion difference between the first emotion image and the reference emotion image according to the emotion value of the emotion in the first emotion image and the emotion value of the emotion in the reference emotion image;

[0126] The first loss function value of the emotion distribution loss function is determined according to the emotion difference values corresponding to each emotion and the emotion values of each emotion in the reference emotion image.

[0127] For example, the emotion difference between the first emotion image and the reference emotion image can be expressed by the following formula (5):

[0128]

[0129] Among them, d i Represents the emotion value of the i-th category emotion of the reference emotion image.

[0130] Based on the emotional differences corresponding to various emotions and the emotional values of various emotions in the reference emotional image, the first loss function value of the emotional distribution loss function can be determined, which can be specifically expressed by the following formula (6):

[0131]

[0132] in, Represents the first loss function value, N cat The number of categories representing sentiment.

[0133] This embodiment calculates the emotion distribution loss based on the emotion values of various emotions obtained from different perspectives and the emotion values of various emotions of the reference emotion image. It takes into account the subjectivity of emotions, improves the accuracy of the emotion distribution loss, and thus improves the training effect of the model.

[0134] In order to ensure that the image can accurately express the emotion in the text and be consistent with the input image in content and structure, Figure 5 A flowchart of a model training method is provided as an example. Figure 5 and Figure 1 The difference is that Figure 5 S510-S520 is also included, and the aforementioned S160 can be further refined into S530. In actual application, this embodiment does not limit the execution order of S510-S520 and S130-S150. For example, S510-S520 may be executed first, followed by S130-S150, or S130-S150 may be executed first, followed by S510-S520, or S510-S520 and S130-S150 may be executed in parallel.

[0135] S510: Acquire texture mapping features of the first emotional image.

[0136] The texture mapping feature is the feature obtained by passing the latent space vector z′0 through the i-th decoder layer of the image decoder, 1≤i≤N-1, and N is the number of decoder layers contained in the image decoder.

[0137] In this embodiment, it is understood that the later the decoder layer in the image decoder, the stronger the emotional or artistic characteristics of the generated image, but this is accompanied by increased content distortion. Therefore, to balance emotion and content consistency, this embodiment can extract the texture mapping features of the first emotional image output through the i-th decoder layer. From this, the texture mapping features output by multiple decoder layers can be obtained. Taking N = 3 as an example, the texture mapping features output by the first decoder layer and the texture mapping features output by the second decoder layer can be extracted separately.

[0138] S520: Determine a second loss function value of a texture mapping loss function according to the texture mapping features of the first emotion image and the texture mapping features of the reference emotion image.

[0139] This embodiment does not limit the specific form of the texture mapping loss function. For example, the mean function can be used as the texture mapping loss function, the variance function can be used as the texture mapping loss function, or the mean function and the variance function can be used as the texture mapping loss function.

[0140] Exemplarily, the texture mapping features of the first emotion image and the reference emotion image may be directly input into the texture mapping loss function to obtain the second loss function value.

[0141] For example, statistical features of the texture mapping feature may be first determined, and then input into the texture mapping loss function to obtain a second loss function value. The statistical features here may include, but are not limited to, mean and variance. The mean may characterize the global characteristics of the texture mapping feature, and the variance may characterize the local characteristics of the texture mapping feature.

[0142] S530: Train a multimodal fusion module according to the first loss function value and the second loss function value to obtain a second image processing model.

[0143] Exemplarily, the first loss function value and the second loss function value can be weighted and summed to obtain a target loss function value, and the multimodal fusion module can be trained based on the target loss function value. The weights of the first loss function value and the second loss function value can be determined based on the accuracy of the emotion distribution feature and the accuracy of the texture mapping feature, respectively.

[0144] Based on the sentiment distribution loss, this embodiment further considers the texture mapping loss of the image generated by the model, and trains the first image processing model by combining the sentiment distribution loss and texture mapping loss. This allows the first image processing model to ensure that the generated image can accurately express the emotions in the text, and to make the content and structure of the generated image consistent with the input image, that is, to take into account both sentiment accuracy and content consistency.

[0145] To determine the second loss function value, illustratively, the above S520 may include the following steps:

[0146] Determining a third statistical feature of a first texture mapping feature, where the first texture mapping feature is a feature extracted from the first emotion image through an i-th decoder layer;

[0147] Based on the third statistical feature and the fourth statistical feature, the second loss function value of the texture mapping loss function is determined; wherein the fourth statistical feature is the statistical feature of the second texture mapping feature, and the second texture mapping feature is the feature obtained by the reference emotion image through the i-th decoder layer.

[0148] For example, the third statistical feature may include, but is not limited to, a mean and a variance. The third statistical feature of the first texture mapping feature may be calculated based on the first texture mapping feature output by the i-th decoder layer. Thus, each decoder layer may obtain a third statistical feature. Similarly, for the reference emotional image, corresponding multiple fourth statistical features may be obtained.

[0149] According to the third statistical features and the fourth statistical features corresponding to the i-th decoder layer, the loss function value corresponding to the i-th decoder layer can be determined, and according to the loss function values corresponding to each decoder layer, the second loss function value can be obtained.

[0150] Exemplarily, the second loss function value can be determined in the following specific manner:

[0151] Determining a feature difference value between the first texture mapping feature and the second texture mapping feature based on the third statistical feature and the fourth statistical feature, where the feature difference value includes at least one of the following: a global feature difference value and a local feature difference value;

[0152] A second loss function value of the texture mapping loss function is determined according to the feature difference values corresponding to each decoder layer.

[0153] Taking the third statistical feature including the mean and variance as an example, illustratively, for the i-th decoder layer, the local feature difference value between the first texture mapping feature and the second texture mapping feature can be determined based on the mean of the first texture mapping feature and the mean of the second texture mapping feature. For the i-th decoder layer, the global feature difference value between the first texture mapping feature and the second texture mapping feature can be determined based on the variance of the first texture mapping feature and the variance of the second texture mapping feature. The local feature difference value is used to characterize the local texture mapping difference between the first emotion image and the reference emotion image after passing through i decoder layers, and the global feature difference value is used to characterize the global texture mapping difference between the first emotion image and the reference emotion image after passing through i decoder layers.

[0154] According to the local feature difference value and the global feature difference value corresponding to each decoder layer, the second loss function value can be obtained. For details, please refer to the following formula (7) and formula (8):

[0155]

[0156] in, is the loss function value corresponding to the i-th decoding layer, F dif (a,b)=|μ(a)-μ(b)| 2 +|σ(a)-σ(b)| 2 , μ and σ represent the mean function and variance function respectively, F dif (a, b) represents the sum of the local feature difference value and the global feature difference value corresponding to the i-th decoder layer, represents the texture map feature extracted from the i-th decoder layer in the image decoder, z acr represents the encoding result of the reference emotional image, z out represents the latent space vector after denoising, i.e. z′0. For example, is the value of the second loss function, γ is a hyperparameter, and illustratively, γ = 0.3.

[0157] This embodiment calculates the statistical features of texture mapping features, and based on the statistical features, determines the feature difference values of the first emotion image and the reference emotion image after passing through i decoder layers. In particular, when the statistical features are the mean and variance, the global texture mapping difference and local texture mapping difference of the first emotion image and the reference emotion image after passing through i decoder layers can be obtained. Training the first image processing model based on the global texture mapping difference and the local texture mapping difference can improve the effect of the first image processing model, so that the first image processing model can ensure the accuracy of emotion and the consistency of image content, and avoid the loss of content and structure in the generated image.

[0158] In some embodiments, the multimodal fusion module can be trained based on the first loss function value, the noise loss function value, and the second loss function value. For example, the first loss function value, the noise loss function value, and the second loss function value can be weighted and summed to obtain a target loss function value, and the multimodal fusion module can be trained based on the target loss function value. This allows the first image processing model to simultaneously consider the emotional accuracy, image clarity, and content and structural consistency of the generated image, further improving the training effect of the first image processing model.

[0159] To further enable the first affective image to evoke a specific affective response in the observer, Figure 6 A flowchart of a model training method is provided as an example. Figure 6 and Figure 1 The difference is that Figure 6 Also includes S610-S630, and the above S160 can be refined into Figure 6 The S640 in the.

[0160] S610: Input the first emotion image into a convolutional neural network to obtain candidate image features of the first emotion image output by a target network layer of the convolutional neural network.

[0161] The convolutional neural network here can be a VGG network. For example, candidate image features output by multiple target network layers of the VGG network can be extracted. That is, there are multiple candidate image features. Based on the multiple candidate image features, the emotional features of the first emotional image can be obtained. For ease of distinction, this embodiment records the obtained emotional features of the first emotional image as the first emotional features.

[0162] S620: Determine a first emotional feature corresponding to the first emotional image based on the candidate image features.

[0163] For example, for each candidate image feature output by the target network layer, a Gram matrix can be calculated, and the candidate image features can be concatenated to obtain the first emotion feature. In a similar manner, the second emotion feature corresponding to the reference emotion image can be obtained.

[0164] S630. Determine a third loss function value of the emotion loss function based on the first emotion feature and the second emotion feature corresponding to the reference emotion image.

[0165] Exemplarily, a distance function may be used as the sentiment loss function. Exemplarily, the third loss function value may be expressed by the following formula (9):

[0166]

[0167] in, is the third loss function value, V out Represents the first emotional feature, V acr Represents the second emotional feature. Of course, other loss functions can also be used, such as mean function or variance function. In this way, the emotional loss between the first emotional image and the reference emotional image can be obtained.

[0168] S640: Train a multimodal fusion module according to the first loss function value and the third loss function value to obtain a second image processing model.

[0169] Exemplarily, the first loss function value and the third loss function value may be weightedly summed to obtain a target loss function value, and the multimodal fusion module may be trained based on the target loss function value.

[0170] Based on the sentiment distribution loss, this embodiment further considers the sentiment loss of the images generated by the model, and trains the first image processing model by combining the sentiment distribution loss and the sentiment loss, so that the images generated by the first image processing model can more accurately express the emotions in the text.

[0171] In some embodiments, the multimodal fusion module can be trained based on the first loss function value, the noise loss function value, the second loss function value, and the third loss function value. For example, the first loss function value, the noise loss function value, the second loss function value, and the third loss function value can be weighted and summed to obtain a target loss function value, and the multimodal fusion module can be trained based on the target loss function value. In this way, the first image processing model can simultaneously take into account the emotional accuracy, image clarity, and content and structural consistency of the generated image, further improving the training effect of the first image processing model.

[0172] For example, the target loss function value can be expressed by the following formula:

[0173]

[0174] in, is the target loss function value, λ dm ,λ ed ,λtm and λ tm are weights respectively, exemplarily,

[0175] λ dm =1,λ ed =10,λ tm =0.001,λ as =10.

[0176] The following describes the training process of the image decoder. The training process of the image decoder can be performed before S120.

[0177] Taking the example that the sample text includes reference text, for example, one of the sample texts can be randomly selected as the reference text. The training process can include the following S710-S740.

[0178] S710 : Determine, from the sample texts, associated texts associated with the reference text according to the area of the emotion in the reference text in the preset emotion wheel.

[0179] The preset emotion wheel is used to display different types of emotions and the areas corresponding to the emotions in the preset emotion wheel. Figure 7 An exemplary diagram of an emotion wheel is provided. Taking the example of an emotion wheel containing eight emotion categories, each emotion category corresponds to a region, and different emotions are located in different regions. Thus, the emotion wheel can be divided into eight regions, each with a different number. Associated text related to a reference text can be, for example, text in a region adjacent to the region to which the reference text belongs, text in a region opposite to the region to which the reference text belongs, or text in the region to which the reference text belongs.

[0180] Taking the reference text belonging to region 1 as an example, the adjacent regions of region 1 may be region 2 and region 8, so texts with excitement and sadness as emotions may be selected from the sample texts as associated texts.

[0181] The opposite region of region 1 may be region 5, and thus texts with a happy emotion may be selected from the sample texts as associated texts.

[0182] For example, text expressing fear may be selected from the sample text as associated text.

[0183] In actual application, there can be one or more related texts. When there are multiple related texts, the emotions contained in each related text can be the same or different. The same related text can include one emotion or multiple emotions.

[0184] S720. Input the second emotional image corresponding to the reference text and the third emotional image corresponding to the associated text into the convolutional neural network respectively to obtain the second image features of the second emotional image output by the target network layer of the convolutional neural network and the third image features of the third emotional image output by the target network layer of the convolutional neural network.

[0185] Each sample text corresponds to a reference emotion image. To facilitate distinction, this embodiment records the reference emotion image corresponding to the reference text as the second emotion image, and records the reference emotion image corresponding to the associated text as the third emotion image.

[0186] The convolutional neural network here can be a VGG network. By inputting the second emotion image and the third emotion image into the VGG network respectively, the second image feature of the second emotion image output by the target network layer of the VGG network and the third image feature of the third emotion image output by the target network layer of the VGG network can be obtained.

[0187] S730. Determine a fourth loss function value of the emotion contrast loss function based on the second image feature and the third image feature.

[0188] Based on the emotion contrast loss, the image decoder can accurately understand the relationship between different emotions, so that the emotions in the text can be more accurately understood and accurately mapped to the image.

[0189] For example, the fourth loss function value of the emotion contrast loss function can be determined directly based on the second image feature and the third image feature. For example, the second image feature and the third image feature are input into the emotion contrast loss function to obtain the fourth loss function value.

[0190] Exemplarily, the emotional distance may be determined based on the second image feature and the third image feature, and then the fourth loss function value may be determined based on the emotional distance.

[0191] S740: Train the image decoder according to the fourth loss function value.

[0192] Exemplarily, the training ends when the number of iterations reaches a threshold, or the fourth loss function value reaches a minimum or tends to be stable.

[0193] When training the image decoder, this embodiment first selects a reference text from the sample text, then determines the associated text based on the reference text, and then determines the emotion contrast loss function value based on the second image feature of the second emotion image corresponding to the reference text and the third image feature of the third emotion image corresponding to the associated text. The image decoder is trained based on the emotion contrast loss function value, taking into account the relationship between different emotions and improving the image decoder's ability to understand emotions.

[0194] Taking the example that the associated text includes at least one of the following: positive example text, negative example text, and related text, in order to determine the associated text, illustratively, the above S710 may include the following steps:

[0195] In the case where the associated text includes the positive example text, at least one text is selected from the sample text corresponding to the reference area as the positive example text, and the reference area is the area to which the emotion in the reference text belongs in the preset emotion wheel;

[0196] In the case where the associated text includes a negative example text, selecting at least one text from the sample text corresponding to the opposite region of the reference region as the negative example text;

[0197] In the case where the associated text includes the relevant example text, at least one text is selected from the sample texts corresponding to the adjacent area of the reference area as the relevant text.

[0198] In this embodiment, the positive text and the reference text belong to the same region, that is, the sentiment in the positive text is the same as that in the reference text. The negative text is the text in the opposite region of the reference region. Figure 7 For example, assuming that the emotion of the reference text is excitement, the reference area is area 2, and the opposite area can be area 6. The negative text is also the text in area 6, and the corresponding emotion is anger.

[0199] The associated text is the text in the adjacent area of the reference area. Taking area 2 as the reference area, the adjacent areas are area 3 and area 1. Therefore, one or more sample texts can be selected from area 3 and area 2 respectively as the associated text.

[0200] In some embodiments, the associated text may include positive text, negative text, and related text at the same time, so that the image decoder can more accurately understand the relationship between different emotions, thereby improving the ability to understand emotions.

[0201] After the reference text, positive text, negative text and related text are determined, the reference emotional image corresponding to each text can be obtained, providing a basis for subsequent training of the image decoder.

[0202] This embodiment determines the associated text based on the position of the reference text in the emotion wheel. A variety of emotions can be obtained based on the reference text and the associated text. Therefore, when the image decoder is trained based on the reference emotion images corresponding to the reference text and the associated text, the image decoder can more accurately understand the relationship between different emotions, thereby improving the image decoder's ability to understand emotions, so that the output image can more accurately express the emotions in the text.

[0203] To determine the fifth loss function value of the emotion contrast loss function, illustratively, the above S730 may include the following steps:

[0204] determining a Gram matrix corresponding to the second image feature;

[0205] Determine the first sentiment vector corresponding to the reference text according to the Gram matrix;

[0206] Determining an emotional distance between the first emotional vector and the second emotional vector based on a vector distance between the first emotional vector and a second emotional vector corresponding to the associated text, and regions of the reference text and the associated text in a preset emotional wheel; wherein the second emotional vector is determined based on a third image feature;

[0207] According to the sentiment distance, the sentiment contrast loss function value of the sentiment contrast loss function is determined.

[0208] The specific calculation process of the Gram matrix can be found in related art and will not be repeated here. In this embodiment, a Gram matrix can be obtained for the second image features extracted by each layer of the network. For example, if the VGG network includes three layers, three Gram matrices can be obtained for the second emotion image. In some embodiments, the corresponding Gram matrices can be spliced in the order of the network from the first layer to the last layer. In this embodiment, the splicing result can be defined as an emotion vector, or it can also be called an emotion feature.

[0209] For example, Among them, V represents the sentiment vector, Concat represents concatenation, Represents the i-th upper triangular element of the j-th feature level in the Gram matrix, where the j-th feature level represents the features extracted by the j-th network layer.

[0210] In this way, the first sentiment vector corresponding to the reference text and the second sentiment vector corresponding to the associated text can be obtained, which provides a basis for the subsequent determination of the sentiment contrast loss function value.

[0211] Sentiment distance can represent the relationship between different emotions. For example, the closer the two emotions are, the smaller the sentiment distance is, and vice versa. Sentiment distance can be determined based on the vector distance between sentiment vectors and the area of the reference text and the associated text in the sentiment wheel.

[0212] For example, the sentiment distance between two sentiment vectors can be determined by the following formula (11):

[0213] F sw (V m ,V n )=|V m -V n | 2 / F dis (V m ,V n) (11)

[0214] Among them, F sw (V m ,V n ) represents the sentiment vector V m With the emotion vector V n The emotional distance between m -V n | 2 Represents the sentiment vector V m With the emotion vector V n The vector distance between them, F dis (V m ,V n ) represents the minimum number of steps between emotions m and n in the emotion wheel. For example, if emotion m is fear and emotion n is disgust, emotion m can reach the area where emotion n is located in two steps or in six steps. Therefore, the minimum number of steps between emotions m and n in the emotion wheel is 2.

[0215] Based on the above formula (11), the emotional distance between the first emotional vector and the second emotional vector can be obtained.

[0216] Taking the associated text including positive example text, negative example text and related text as an example, based on the above formula, we can obtain the emotional distance between the emotional vector corresponding to the positive example text and the first emotional vector corresponding to the reference text, as well as the emotional distance between the emotional vector corresponding to the negative example text and the first emotional vector corresponding to the reference text, and the emotional distance between the emotional vector corresponding to the related text and the first emotional vector corresponding to the reference text.

[0217] Based on the emotional distance, the emotional contrast loss function value can be obtained. For example, the emotional contrast loss function value can be determined by the following formula (12):

[0218]

[0219] in, is the value of the sentiment contrast loss function, V sed is the sentiment vector corresponding to the reference text, V pos is the sentiment vector corresponding to the positive text, V rel is the sentiment vector corresponding to the relevant text, V neg is the sentiment vector corresponding to the negative text, α and β are hyperparameters used to control the margin between sentiment vectors. In some embodiments, α = 0.02, β = 0.01.

[0220] This embodiment determines the emotion vector corresponding to the text based on the image features of the extracted emotion image, and determines the loss function value of the emotion contrast loss function based on the emotion distance between the emotion vectors corresponding to different texts, which provides a basis for subsequent training of the image decoder, so that the image decoder can more accurately understand the relationship between different emotions, and further improve the emotion understanding ability of the image decoder, so that the generated image can express the emotion in the text as accurately as possible.

[0221] In order to generate a suitable style, illustratively, before S740, the model training method may further include the following steps:

[0222] Input the first vector corresponding to the content image into the image decoder to obtain the first style feature and the first style image output by the image decoder, where the first style feature is the feature of the content image output by the i-th decoder layer;

[0223] determining a first statistical feature of a first style feature;

[0224] Determining a fifth loss function value of the style loss function based on the first statistical feature and the second statistical feature, where the second statistical feature is a statistical feature of the second style feature, and the second style feature is a feature output by the reference emotion image through the i-th decoder layer;

[0225] Accordingly, the above S740 may include the following steps:

[0226] The image decoder is trained according to the fourth loss function value and the fifth loss function value.

[0227] For example, the first statistical feature may include, but is not limited to, the mean and variance of the first style feature. Each decoder layer may correspond to a first statistical feature, thereby obtaining N-1 first statistical features, where N is the number of decoder layers. The first style image is the output of the Nth decoder layer. Similarly, the second statistical feature extracted from the reference emotion image by the i-th decoder layer can be obtained.

[0228] Based on the first statistical feature and the second statistical feature, the style loss of the first style image can be determined to obtain a fifth loss function value.

[0229] Taking the first statistical feature including mean and variance as an example, illustratively, the fifth loss function value can be determined by the following formula (13):

[0230]

[0231] in, is the fifth loss function value, μ and σ are the mean function and variance function respectively, and These are the features obtained by passing the i-th decoder layer for the first style image and the reference emotion image, respectively. This embodiment comprehensively considers both the global style loss and the local style loss of the first style image, taking into account more comprehensive factors. This allows the image decoder to more accurately learn the style differences between the first style image and the reference emotion image, and thus generate an appropriate style image.

[0232] On this basis, the image decoder can be trained based on the fourth loss function value and the fifth loss function value.

[0233] Exemplarily, the fourth loss function value and the fifth loss function value may be weightedly summed to obtain a sixth loss function value, and the image decoder may be trained based on the sixth loss function value.

[0234] Exemplarily, the sixth loss function value can be determined by the following formula (14):

[0235]

[0236] in, is the sixth loss function value, λ sm and λ s are weights respectively. In some embodiments, λ sm =1,λ s =0.01.

[0237] When training the image decoder, this embodiment comprehensively considers the emotional contrast loss and style loss, so that the image decoder can take into account both the artistic style of the image and the accuracy of the emotion, avoiding problems such as insufficient emotional understanding and excessive image stylization, thereby improving the training effect of the image decoder.

[0238] After the image decoder training is completed, its parameters will be fixed, retaining the learned artistic style and emotion prior knowledge, and applied to the training of the multimodal fusion module.

[0239] In order to further understand the deep-level emotions in the text, illustratively, the above-mentioned acquisition of sample text may include the following steps:

[0240] Get the original text;

[0241] Identify the emotions in the original text and obtain the descriptive information corresponding to the emotions;

[0242] The description information and the original text are fused to obtain the sample text.

[0243] The original text can be text that has not been processed for emotion. For example, a large language model can be used to identify the emotion in the original text. The large language model can also output corresponding descriptive information based on the identified emotion as context information for subsequent prompts for the filter effect of the image. Then, based on this descriptive information and the original text, a sample text is obtained. In other words, the sample text is text supplemented with filter effect information. This can provide more comprehensive emotional context prompts for the model's first image processing model, thereby improving the model's ability to understand emotion.

[0244] This embodiment introduces a large language model and chain thinking prompt technology, which can better understand the semantics in the context of the text description, so as to conduct in-depth emotional reasoning, which is conducive to improving the fidelity of the generated image to the emotions expressed in the text description. At the same time, it simulates the process of users viewing images from different perspectives, evaluates the distribution of emotions expressed in the generated image from four perspectives: color, texture, style, and blocks, accurately analyzes the emotions inspired by the image, and improves the fidelity of the generated image to the emotions expressed in the text description. At the same time, the texture mapping loss is designed so that the model can seek a balance between artistic style and Nie Rong consistency, which is conducive to improving the content consistency and emotional fidelity of the generated image. That is, through the above scheme, the image output by the model can accurately express the emotions in the text and produce a suitable filter effect, which effectively solves the problems of missing image content, insufficient understanding of emotions, and excessive stylization.

[0245] Based on the above embodiments, the present application further provides an image processing method, which may include the following steps:

[0246] Acquire a first image and a first text;

[0247] Encoding the first image to obtain an encoding vector;

[0248] Processing the first text according to the sentiment in the first text to obtain a second text;

[0249] The encoding vector and the text vector corresponding to the second text are input into a second image processing model to obtain a second image; wherein the second image processing model is trained according to the model training method in the above embodiment.

[0250] The first image here can be any RGB image, and the first text can be any text. For example, the first image can be encoded using an image encoder to obtain an encoding vector. The image encoder can be the image encoder used for model training, that is, the image encoder that obtains the first vector based on the content image.

[0251] The second text may be a text that supplements the emotional context prompt information. The specific processing process can refer to the original text in the above embodiment. For the sake of brevity, it will not be repeated here. The text vector corresponding to the second text can be obtained by encoding based on the CLIP model.

[0252] In actual application, the encoding vector corresponding to the first image and the text vector corresponding to the second text can be input into the second image processing model to obtain a second image that can accurately express the emotions in the first text. The content and structure of the second image are consistent with the first image, and the artistic style of the second image meets the user's expectations.

[0253] It should be noted that the model training method provided in the embodiments of the present application can be executed by a model training device or a processing module in the model training device for executing the model training method. In the embodiments of the present application, the model training device provided in the embodiments of the present application is described by taking the execution of the model training method by the model training device as an example.

[0254] Figure 8 A schematic diagram of the structure of a model training device provided in an embodiment of the present application.

[0255] like Figure 8 As shown, the model training device 800 may include:

[0256] An acquisition module 801 is configured to acquire a training sample set, where the training sample set includes at least two groups of training samples, each group of training samples includes a content image, a sample text, and a reference emotion image corresponding to the sample text, where the reference emotion image is used to reflect the emotion in the sample text;

[0257] A multimodal fusion module 802 is used to fuse the content image and the sample text and output a latent space vector. The multimodal fusion module belongs to the first image processing model.

[0258] An image decoder 803 is configured to decode the latent space vector and output a first emotional image. The image decoder is trained using the first vector corresponding to the content image as input and the reference emotional image as output. The image decoder belongs to the first image processing model.

[0259] Determining module 804, configured to determine an emotion distribution feature of the first emotion image based on the first image feature of the first emotion image; and determine a first loss function value of an emotion distribution loss function based on the emotion distribution feature of the first emotion image and the emotion distribution feature of the reference emotion image;

[0260] The training module 805 is used to train the multimodal fusion module according to the first loss function value to obtain a second image processing model.

[0261] Before training the first image processing model, this embodiment first trains the image decoder of the first image processing model based on the content image and the reference emotion image that can reflect the emotion, ensuring that the image decoder can accurately understand the relationship between emotions and improve the image decoder's ability to understand emotions. Then, based on the trained image decoder and the multimodal fusion module and training samples of the first image processing model, the loss function value of the emotion distribution of the first emotion image and the reference emotion image output by the first image processing model is determined, and the multimodal fusion module is trained based on the loss function value. That is, this embodiment combines the image decoder's understanding of emotion and the emotion distribution loss of the first emotion image and the reference emotion image to train the model from multiple levels, so that the model can more accurately understand the emotion in the text, and then more accurately map the emotion to the image to meet the user's emotional expression needs.

[0262] In some possible implementations of the embodiments of the present application, the first image feature includes at least two, and the emotion distribution feature includes an emotion value of a target emotion;

[0263] The determination module 804 is specifically configured to:

[0264] Inputting the first image feature into the corresponding emotion prediction model to obtain a candidate emotion value of the target emotion in the first emotion image output by the emotion prediction model; wherein different first image features correspond to different emotion prediction models, and the emotion prediction model is trained by using the reference image feature of the reference emotion image as input and the true emotion value of the target emotion in the reference emotion image as output;

[0265] The emotion value of the target emotion in the first emotion image is determined according to the candidate emotion values corresponding to each first image feature and the confidence of each emotion prediction model.

[0266] In some possible implementations of the embodiments of the present application, the target emotion includes at least two types of emotions;

[0267] The determination module 804 is specifically configured to:

[0268] For each type of emotion, determining an emotion difference between the first emotion image and the reference emotion image according to the emotion value of the emotion in the first emotion image and the emotion value of the emotion in the reference emotion image;

[0269] The first loss function value of the emotion distribution loss function is determined according to the emotion difference values corresponding to each emotion and the emotion values of each emotion in the reference emotion image.

[0270] In some possible implementations of the embodiments of the present application, the acquisition module 801 is further configured to acquire a texture mapping feature of the first emotional image, where the texture mapping feature is a feature obtained by passing the latent space vector through the i-th decoder layer of the image decoder, where 1≤i≤N-1, and N is the number of decoder layers included in the image decoder;

[0271] The determining module 804 is further configured to determine a second loss function value of the texture mapping loss function based on the texture mapping features of the first emotion image and the texture mapping features of the reference emotion image;

[0272] The training module 805 is specifically used to:

[0273] The multimodal fusion module is trained according to the first loss function value and the third loss function value to obtain a second image processing model.

[0274] In some possible implementations of the embodiments of the present application, the determination module 804 is further configured to input the first emotion image into the convolutional neural network before the training module 805 trains the multimodal fusion module according to the first loss function value to obtain the second image processing model, and obtain candidate image features of the first emotion image output by the target network layer of the convolutional neural network;

[0275] Determining a first emotional feature corresponding to the first emotional image based on the candidate image features;

[0276] Determining a third loss function value of the emotion loss function according to the first emotion feature and a second emotion feature corresponding to the reference emotion image;

[0277] The training module 805 is specifically used to:

[0278] The multimodal fusion module is trained according to the first loss function value and the third loss function value to obtain a second image processing model.

[0279] In some possible implementations of the embodiments of the present application, the sample text includes a reference text;

[0280] The determination module 804 is further configured to determine, from the sample text, associated text associated with the reference text based on the region of the emotion in the reference text in a preset emotion wheel, before the multimodal fusion module 802 fuses the content image and the sample text to obtain the latent space vector output by the multimodal fusion module 802, wherein the preset emotion wheel is configured to display different types of emotions and the regions corresponding to the emotions in the preset emotion wheel;

[0281] Inputting the second emotion image corresponding to the reference text and the third emotion image corresponding to the associated text into the convolutional neural network respectively, obtaining the second image feature of the second emotion image output by the target network layer of the convolutional neural network and the third image feature of the third emotion image output by the target network layer of the convolutional neural network;

[0282] Determining a fourth loss function value of the emotion contrast loss function based on the second image feature and the third image feature;

[0283] The training module 805 is further configured to train the image decoder according to the fourth loss function value.

[0284] In some possible implementations of the embodiments of the present application, the image decoder 803 is further configured to output a first style feature and a first style image based on a first vector corresponding to the input content image, where the first style feature is a feature of the content image output by the i-th decoder layer;

[0285] The determination module 804 is further configured to determine a first statistical feature of the first style feature;

[0286] Determining a fifth loss function value of the style loss function based on the first statistical feature and the second statistical feature, where the second statistical feature is a statistical feature of the second style feature, and the second style feature is a feature output by the reference emotion image through the i-th decoder layer;

[0287] The training module 805 is specifically used to:

[0288] The image decoder is trained according to the fourth loss function value and the fifth loss function value.

[0289] In some possible implementations of the embodiments of the present application, the acquisition module 801 is specifically configured to:

[0290] Get the original text;

[0291] The model training device 800 may further include:

[0292] The recognition module is used to identify the emotions in the original text and obtain the description information corresponding to the emotions;

[0293] The processing module is used to fuse the description information and the original text to obtain the sample text.

[0294] The model training device in the embodiment of the present application can be a device or a component in an electronic device, such as an integrated circuit or a chip. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a user computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0295] The electronic device in the embodiment of the present application may be a terminal having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0296] The model training device provided in the embodiment of the present application can achieve Figures 1 to 7 The various processes in the model training method embodiment can achieve the same technical effect. To avoid repetition, they will not be described here.

[0297] It should be noted that the image processing method provided in the embodiments of the present application can be executed by an image processing device or a processing module in the image processing device for executing the image processing method. In the embodiments of the present application, the image processing device provided in the embodiments of the present application is described by taking the image processing device executing the image processing method as an example.

[0298] Figure 9 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application.

[0299] like Figure 9 As shown, the image processing device 900 may include:

[0300] An acquisition module 901 is configured to acquire a first image and a first text;

[0301] An encoding module 902 is configured to encode the first image to obtain an encoding vector;

[0302] Processing module 903 is used to process the first text according to the emotion in the first text to obtain a second text; input the encoding vector and the second text into a second image processing model to obtain a second image; wherein the second image processing model is trained according to the model training method of the above embodiment.

[0303] like Figure 10 As shown, an embodiment of the present application also provides an electronic device 1000, including a processor 1001 and a memory 1002, wherein the memory 1002 stores programs or instructions that can be run on the processor 1001. When the program or instruction is executed by the processor 1001, the various steps of the above-mentioned model training method or image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be repeated here.

[0304] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile terminals and non-mobile terminals.

[0305] Figure 11 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0306] The electronic device 1100 includes but is not limited to components such as a radio frequency unit 1101 , a network module 1102 , an audio output unit 1103 , an input unit 1104 , a sensor 1105 , a display unit 1106 , a user input unit 1107 , an interface unit 1108 , a memory 1109 , and a processor 1110 .

[0307] Those skilled in the art will understand that the electronic device 1100 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 1110 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 11 The structure of the electronic device 1100 shown in the figure does not constitute a limitation on the electronic device 1100. The electronic device 1100 may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0308] The processor 1110 is configured to obtain a training sample set, the training sample set including at least two groups of training samples, each group of training samples including a content image, a sample text, and a reference emotion image corresponding to the sample text, the reference emotion image being used to reflect the emotion in the sample text;

[0309] Using the multimodal fusion module of the first image processing model to fuse the content image and the sample text, and obtain a latent space vector output by the multimodal fusion module;

[0310] Decoding the latent space vector using the image decoder of the first image processing model to obtain a first emotion image output by the image decoder; wherein the image decoder is trained using the first vector corresponding to the content image as input and the reference emotion image as output;

[0311] determining an emotion distribution feature of the first emotion image based on the first image feature of the first emotion image;

[0312] Determining a first loss function value of an emotion distribution loss function according to the emotion distribution characteristics of the first emotion image and the emotion distribution characteristics of the reference emotion image;

[0313] The multimodal fusion module is trained according to the first loss function value to obtain a second image processing model.

[0314] Before training the first image processing model, this embodiment first trains the image decoder of the first image processing model based on the content image and the reference emotion image that can reflect the emotion, ensuring that the image decoder can accurately understand the relationship between emotions and improve the image decoder's ability to understand emotions. Then, based on the trained image decoder and the multimodal fusion module and training samples of the first image processing model, the loss function value of the emotion distribution of the first emotion image and the reference emotion image output by the first image processing model is determined, and the multimodal fusion module is trained based on the loss function value. That is, this embodiment combines the image decoder's understanding of emotion and the emotion distribution loss of the first emotion image and the reference emotion image to train the model from multiple levels, so that the model can more accurately understand the emotion in the text, and then more accurately map the emotion to the image to meet the user's emotional expression needs.

[0315] In some possible implementations of the embodiments of the present application, the first image feature includes at least two, and the emotion distribution feature includes an emotion value of a target emotion;

[0316] The processor 1110 is specifically configured to:

[0317] Inputting the first image feature into the corresponding emotion prediction model to obtain a candidate emotion value of the target emotion in the first emotion image output by the emotion prediction model; wherein different first image features correspond to different emotion prediction models, and the emotion prediction model is trained by using the reference image feature of the reference emotion image as input and the true emotion value of the target emotion in the reference emotion image as output;

[0318] The emotion value of the target emotion in the first emotion image is determined according to the candidate emotion values corresponding to each first image feature and the confidence of each emotion prediction model.

[0319] In some possible implementations of the embodiments of the present application, the target emotion includes at least two types of emotions;

[0320] The processor 1110 is specifically configured to:

[0321] For each type of emotion, determining an emotion difference between the first emotion image and the reference emotion image according to the emotion value of the emotion in the first emotion image and the emotion value of the emotion in the reference emotion image;

[0322] The first loss function value of the emotion distribution loss function is determined according to the emotion difference values corresponding to each emotion and the emotion values of each emotion in the reference emotion image.

[0323] In some possible implementations of the embodiments of the present application, the processor 1110 is further configured to obtain a texture mapping feature of the first emotion image, where the texture mapping feature is a feature obtained by passing the latent space vector through the i-th decoder layer of the image decoder, where 1≤i≤N-1, and N is the number of decoder layers included in the image decoder;

[0324] Determining a second loss function value of a texture mapping loss function according to the texture mapping features of the first emotion image and the texture mapping features of the reference emotion image;

[0325] The processor 1110 is specifically configured to:

[0326] The multimodal fusion module is trained according to the first loss function value and the second loss function value to obtain a second image processing model.

[0327] In some possible implementations of the embodiments of the present application, the processor 1110 is further configured to, before training the multimodal fusion module according to the first loss function value to obtain the second image processing model, input the first emotion image into the convolutional neural network to obtain candidate image features of the first emotion image output by a target network layer of the convolutional neural network;

[0328] Determining a first emotional feature corresponding to the first emotional image based on the candidate image features;

[0329] Determining a third loss function value of the emotion loss function according to the first emotion feature and a second emotion feature corresponding to the reference emotion image;

[0330] The processor 1110 is specifically configured to:

[0331] The multimodal fusion module is trained according to the first loss function value and the third loss function value to obtain a second image processing model.

[0332] In some possible implementations of the embodiments of the present application, the sample text includes a reference text;

[0333] Processor 1110 is further configured to fuse the content image and the sample text using the multimodal fusion module of the first image processing model, and determine, from the sample text, associated text associated with the reference text based on the region of the emotion in the reference text in a preset emotion wheel before obtaining a latent space vector output by the multimodal fusion module, where the preset emotion wheel is configured to display different types of emotions and the regions corresponding to the emotions in the preset emotion wheel;

[0334] Inputting the second emotion image corresponding to the reference text and the third emotion image corresponding to the associated text into the convolutional neural network respectively, obtaining the second image feature of the second emotion image output by the target network layer of the convolutional neural network and the third image feature of the third emotion image output by the target network layer of the convolutional neural network;

[0335] Determining a fourth loss function value of the emotion contrast loss function based on the second image feature and the third image feature;

[0336] The image decoder is trained according to the fourth loss function value.

[0337] In some possible implementations of the embodiments of the present application, the processor 1110 is further configured to, before training the image decoder according to the fourth loss function value, input a first vector corresponding to the content image into the image decoder to obtain a first style feature and a first style image output by the image decoder, where the first style feature is a feature of the content image output by the i-th decoder layer;

[0338] determining a first statistical feature of a first style feature;

[0339] Determining a fifth loss function value of the style loss function based on the first statistical feature and the second statistical feature, where the second statistical feature is a statistical feature of the second style feature, and the second style feature is a feature output by the reference emotion image through the i-th decoder layer;

[0340] The processor 1110 is specifically configured to:

[0341] The image decoder is trained according to the fourth loss function value and the fifth loss function value.

[0342] It should be understood that in an embodiment of the present application, the input unit 1104 may include a graphics processing unit (GPU) 11041 and a microphone 11042, and the graphics processor 11041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1106 may include a display panel 11061, and the display panel 11061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1107 includes a touch panel 11071 and at least one of other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include two parts: a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0343] The memory 1109 can be used to store software programs and various data. The memory 1109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1109 may include a volatile memory or a non-volatile memory, or the memory 1109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 1109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0344] Processor 1110 may include one or more processing units. Optionally, processor 1110 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1110.

[0345] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned model training method or image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0346] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0347] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, which are coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-mentioned model training method or image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0348] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0349] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned model training method or image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0350] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0351] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0352] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A model training method, characterized in that: include: Acquire a training sample set, the training sample set including at least two groups of training samples, each group of training samples including a content image, a sample text, and a reference emotion image corresponding to the sample text, the reference emotion image being used to reflect the emotion in the sample text; Using a multimodal fusion module of a first image processing model to fuse the content image and the sample text, to obtain a latent space vector output by the multimodal fusion module; Decoding the latent space vector using an image decoder of the first image processing model to obtain a first emotion image output by the image decoder; wherein the image decoder is trained using the first vector corresponding to the content image as input and the reference emotion image as output; determining an emotion distribution feature of the first emotion image according to a first image feature of the first emotion image; Determining a first loss function value of an emotion distribution loss function according to the emotion distribution characteristics of the first emotion image and the emotion distribution characteristics of the reference emotion image; The multimodal fusion module is trained according to the first loss function value to obtain a second image processing model.

2. The method according to claim 1, characterized in that The first image features include at least two, and the emotion distribution feature includes the emotion value of the target emotion; The determining, based on the first image feature of the first emotional image, the emotional distribution feature of the first emotional image includes: Inputting the first image feature into a corresponding emotion prediction model to obtain a candidate emotion value of the target emotion in the first emotion image output by the emotion prediction model; wherein different first image features correspond to different emotion prediction models, and the emotion prediction model is trained using the reference image feature of the reference emotion image as input and the true emotion value of the target emotion in the reference emotion image as output; The emotion value of the target emotion in the first emotion image is determined according to the candidate emotion values corresponding to each of the first image features and the confidence of each of the emotion prediction models.

3. The method according to claim 2, characterized in that The target emotion includes at least two types of emotions; The determining, according to the emotion distribution characteristics of the first emotion image and the emotion distribution characteristics of the reference emotion image, a first loss function value of the emotion distribution loss function comprises: For each type of emotion, determining an emotion difference between the first emotion image and the reference emotion image according to the emotion value of the emotion in the first emotion image and the emotion value of the emotion in the reference emotion image; The first loss function value of the emotion distribution loss function is determined according to the emotion difference values corresponding to each emotion and the emotion values of each emotion in the reference emotion image.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Obtaining a texture mapping feature of the first emotional image, where the texture mapping feature is a feature obtained by passing the latent space vector through the i-th decoder layer of the image decoder, where 1≤i≤N-1, and N is the number of decoder layers included in the image decoder; Determining a second loss function value of a texture mapping loss function according to the texture mapping features of the first emotion image and the texture mapping features of the reference emotion image; The step of training the multimodal fusion module according to the first loss function value to obtain a second image processing model includes: The multimodal fusion module is trained according to the first loss function value and the second loss function value to obtain a second image processing model.

5. The method according to any one of claims 1 to 3, characterized in that Before training the multimodal fusion module according to the first loss function value to obtain the second image processing model, the method further includes: Inputting the first emotion image into a convolutional neural network to obtain candidate image features of the first emotion image output by a target network layer of the convolutional neural network; determining a first emotional feature corresponding to the first emotional image based on the candidate image features; Determining a third loss function value of an emotion loss function according to the first emotion feature and a second emotion feature corresponding to the reference emotion image; The step of training the multimodal fusion module according to the first loss function value to obtain a second image processing model includes: The multimodal fusion module is trained according to the first loss function value and the third loss function value to obtain a second image processing model.

6. The method according to any one of claims 1 to 3, characterized in that The sample text includes a reference text; Before fusing the content image and the sample text using the multimodal fusion module of the first image processing model to obtain the latent space vector output by the multimodal fusion module, the method further includes: Determining, from the sample text, associated text associated with the reference text based on the area of the emotion in the reference text in a preset emotion wheel, wherein the preset emotion wheel is used to display different types of emotions and the areas corresponding to the emotions in the preset emotion wheel; Inputting the second emotion image corresponding to the reference text and the third emotion image corresponding to the associated text into a convolutional neural network respectively, obtaining a second image feature of the second emotion image output by a target network layer of the convolutional neural network and a third image feature of the third emotion image output by the target network layer of the convolutional neural network; Determining a fourth loss function value of an emotion contrast loss function based on the second image feature and the third image feature; The image decoder is trained according to the fourth loss function value.

7. The method according to claim 6, characterized in that Before training the image decoder according to the fourth loss function value, the method further includes: Inputting a first vector corresponding to the content image into the image decoder to obtain a first style feature and a first style image output by the image decoder, where the first style feature is a feature of the content image output by the i-th decoder layer; determining a first statistical feature of the first style feature; determining a fifth loss function value of a style loss function based on the first statistical feature and the second statistical feature, where the second statistical feature is a statistical feature of a second style feature, and the second style feature is a feature output by the reference emotion image after passing through the i-th decoder layer; The step of training the image decoder according to the fourth loss function value comprises: The image decoder is trained according to the fourth loss function value and the fifth loss function value.

8. An image processing method, characterized in that: include: Acquire a first image and a first text; Encoding the first image to obtain an encoding vector; Processing the first text according to the sentiment in the first text to obtain a second text; The encoding vector and the text vector corresponding to the second text are input into a second image processing model to obtain a second image; wherein, the second image processing model is trained according to the model training method according to any one of claims 1-7.

9. A model training device, characterized in that: include: An acquisition module is configured to acquire a training sample set, wherein the training sample set includes at least two groups of training samples, each group of training samples includes a content image, a sample text, and a reference emotion image corresponding to the sample text, wherein the reference emotion image is used to reflect the emotion in the sample text; a multimodal fusion module, configured to fuse the content image and the sample text and output a latent space vector, wherein the multimodal fusion module belongs to the first image processing model; an image decoder, configured to decode the latent space vector and output a first emotional image; wherein the image decoder is trained using the first vector corresponding to the content image as input and the reference emotional image as output, and the image decoder belongs to the first image processing model; a determination module, configured to determine an emotion distribution feature of the first emotion image based on a first image feature of the first emotion image; and determine a first loss function value of an emotion distribution loss function based on the emotion distribution feature of the first emotion image and the emotion distribution feature of the reference emotion image; A training module is used to train the multimodal fusion module according to the first loss function value to obtain a second image processing model.

10. An image processing device, characterized in that: include: An acquisition module, configured to acquire a first image and a first text; an encoding module, configured to encode the first image to obtain an encoding vector; A processing module is used to process the first text according to the emotion in the first text to obtain a second text; input the encoding vector and the second text into a second image processing model to obtain a second image; wherein the second image processing model is trained according to the model training method according to any one of claims 1 to 7.

Images