Emotion recognition and emotion image generation method and device

Abstract

The invention discloses an emotion recognition and emotion image generation method and device, and is applied to the technical field of emotion artificial intelligence, and the method comprises the steps: obtaining a sample scene query identifier, a sample object query identifier and a sample image set; performing model training according to the sample scene query identifier, the sample object query identifier and the sample image set to obtain a target scene query identifier, a target object query identifier and a simultaneous processing model, and guiding the simultaneous processing model to identify scene features of the to-be-detected image by the target scene query identifier, the target object query identifier guides the simultaneous processing model to identify object features of the to-be-detected image; and according to the target scene query identifier, the target object query identifier and the simultaneous processing model, performing emotion recognition and emotion image generation on the to-be-detected image to obtain an emotion type of the to-be-detected image and a plurality of emotion images associated with the emotion type of the to-be-detected image. According to the invention, simultaneous training and synchronous processing of the emotion recognition task and the emotion image generation task can be realized.

Description

Technical Field

[0001] The present application relates to the field of emotional artificial intelligence technology, and in particular to methods and devices for emotion recognition and emotion image generation. Background Art

[0002] At present, emotion recognition tasks and emotion image generation tasks are usually regarded as two independent tasks, and the implementation of each task often relies on independent models. However, this approach not only ignores the inherent correlation between emotion recognition tasks and emotion image generation tasks, and fails to fully tap the potential of their synergy, but also limits the information sharing and collaborative optimization of models between emotion recognition tasks and emotion image generation tasks. In addition, some methods have attempted to explore the correlation between emotion recognition tasks and emotion image generation tasks, but these methods often only use the knowledge information of one to assist in identifying the other. Currently, there are few methods for joint training and synchronous processing of emotion recognition tasks and emotion image generation tasks. Summary of the invention

[0003] The embodiments of the present application provide a method and device for emotion recognition and emotion image generation, which are used to realize joint training and synchronous processing of emotion recognition tasks and emotion image generation tasks.

[0004] On the one hand, the embodiment of the present application provides a method for emotion recognition and emotion image generation, comprising the following steps:

[0005] Obtain a sample scene query identifier, a sample object query identifier, and a sample image set; wherein the sample scene query identifier is used to indicate the scene features of all the sample images, the sample object query identifier is used to indicate the object features of all the sample images, and the sample image set includes a plurality of sample images and an emotion label corresponding to each of the sample images;

[0006] Model training is performed according to the sample scene query identifier, the sample object query identifier and the sample image set to obtain a target scene query identifier, a target object query identifier and a joint processing model; wherein the target scene query identifier is used to guide the joint processing model to identify scene features of the image to be tested, and the target object query identifier is used to guide the joint processing model to identify object features of the image to be tested;

[0007] Emotion recognition and emotion image generation are performed on the image to be tested according to the target scene query identifier, the target object query identifier and the joint processing model to obtain the emotion type of the image to be tested and a plurality of emotion images associated with the emotion type of the image to be tested.

[0008] On the other hand, the embodiment of the present application provides an emotion recognition and emotion image generation device, including:

[0009] An acquisition module, used to acquire a sample scene query identifier, a sample object query identifier and a sample image set; wherein the sample scene query identifier is used to indicate the scene features of all the sample images, the sample object query identifier is used to indicate the object features of all the sample images, and the sample image set includes a plurality of sample images and an emotion label corresponding to each of the sample images;

[0010] A training module, used for performing model training according to the sample scene query identifier, the sample object query identifier and the sample image set to obtain a target scene query identifier, a target object query identifier and a joint processing model; wherein the target scene query identifier is used to guide the joint processing model to identify the scene features of the image to be tested, and the target object query identifier is used to guide the joint processing model to identify the object features of the image to be tested;

[0011] A dual-task processing module is used to perform emotion recognition and emotion image generation on the image to be tested according to the target scene query identifier, the target object query identifier and the joint processing model, so as to obtain the emotion type of the image to be tested and a plurality of emotion images associated with the emotion type of the image to be tested.

[0012] The beneficial effects of the present application are: providing a method and device for emotion recognition and emotion image generation, first, obtaining a sample scene query identifier, a sample object query identifier and a sample image set; wherein the sample scene query identifier is used to indicate the scene features of all sample images, the sample object query identifier is used to indicate the object features of all sample images, and the sample image set includes several sample images and emotion labels corresponding to each sample image; then, model training is performed according to the sample scene query identifier, the sample object query identifier and the sample image set to obtain a target scene query identifier, a target object query identifier and a joint processing model; wherein the target scene query identifier is used to guide the joint processing model to identify the scene features of the image to be tested, and the target object query identifier is used to guide the joint processing model to identify the object features of the image to be tested; finally, emotion recognition and emotion image generation are performed on the image to be tested according to the target scene query identifier, the target object query identifier and the joint processing model to obtain the emotion type of the image to be tested and several emotion images associated with the emotion type of the image to be tested. The present application can realize the joint training and synchronous processing of emotion recognition tasks and emotion image generation tasks, and improve the accuracy and efficiency of emotion recognition and the accuracy and efficiency of emotion image generation.

[0013] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a comparison diagram between the prior art provided by this application and this method;

[0015] Figure 2 is a flow chart of the emotion recognition and emotion image generation method provided by this application;

[0016] Figure 3 is an example diagram of scene query identification and object query identification provided by this application;

[0017] Figure 4 It is a schematic diagram of the emotion recognition and emotion image generation method provided by this application;

[0018] Figure 5 It is a schematic diagram of the sentiment correlation coefficient provided by this application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0020] The present application is further described below in conjunction with the accompanying drawings and specific embodiments. The described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0021] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0023] With the rapid development of emotional artificial intelligence, emotion recognition plays a vital role in many fields and has attracted widespread attention. In particular, in fields such as human-computer interaction and mental health, accurate emotion recognition can significantly improve user experience. Existing emotion recognition methods usually regard the emotion recognition task as a traditional classification task. Figure 1 Part (a) of Figure 1Part (a) of Figure 1 is an example of an existing emotion recognition method. These methods usually use a visual encoder to extract features of the image to be tested, and then map the extracted features to predefined emotion categories through a classification head. The core of these methods is to train a large amount of labeled data through a deep learning model to achieve accurate recognition and classification of emotions in images. However, traditional emotion recognition methods often face challenges such as category imbalance and difficulty in distinguishing fine-grained emotions when dealing with complex and diverse emotional expressions, which in turn limits their performance and promotion in practical applications.

[0024] In addition to emotion recognition tasks, emotion image generation tasks have also become an important research direction of emotion artificial intelligence. This generation task aims to generate image content that conforms to emotional expression based on specific emotional needs. It has a wide range of applications in fields such as digital media, advertising marketing, and virtual reality environments. For example, in a virtual reality environment, the emotion image generation task can dynamically generate corresponding scenes based on the user's emotional state, which helps to enhance immersion and user satisfaction. For another example, in advertising marketing, the emotion image generation task can generate advertising images that conform to consumer emotions, which helps to attract consumer attention and resonance. In current research on emotion image generation, a latent emotion space is usually constructed to closely cluster similar emotion representations together, while dissimilar emotion representations are separated. The construction of this latent space allows the differences between different emotions to be clearly reflected, thereby providing effective conditional input for generating images of specific emotions. Specifically, referring to Figure 1 part (b) of Figure 1 Part (b) is an example of an existing emotional image generation method. In order to generate an image that reflects a specific emotion, the corresponding emotional features are extracted from the latent space and used as conditional input to guide the diffusion model to complete the image generation process. This method can not only generate high-quality image content, but also ensure the consistency and accuracy of the generated image in emotional expression.

[0025] It can be seen that emotion recognition tasks and emotion image generation tasks, as two closely related but distinctive research directions, play an important role in promoting the emotional intelligence of intelligent systems. Emotion recognition tasks can provide the system with accurate perception of the user's emotional state, while emotion image generation tasks can enhance the system's emotional expression ability by generating image content that meets emotional needs.

[0026] At present, emotion recognition tasks and emotion image generation tasks are usually regarded as two independent tasks, and the implementation of each task often relies on independent models. For example, an independent model mainly focuses on extracting and classifying emotion features from the image to be tested to achieve the emotion recognition task, while another independent model focuses on generating corresponding image content according to specific emotion requirements to achieve the emotion image generation task. However, this approach not only ignores the inherent correlation between emotion recognition tasks and emotion image generation tasks, and fails to fully tap the potential of the synergy between the two, but also limits the information sharing and collaborative optimization of the models between emotion recognition tasks and emotion image generation tasks, and leads to problems such as resource reuse. With the development of deep learning and computer vision, some methods have tried to explore the correlation between emotion recognition tasks and emotion image generation tasks, but these methods often only use the knowledge information of one to assist in the recognition of the other, such as using the recognized emotion type to assist in the realization of emotion image generation. At present, there are few methods for joint training and synchronous processing of emotion recognition tasks and emotion image generation tasks.

[0027] In view of this, the present application provides a method and device for emotion recognition and emotion image generation, aiming to integrate emotion understanding and emotion image generation to achieve mutual reinforcement between the two. Through this mutual reinforcement, joint training of emotion recognition tasks and emotion image generation tasks can be achieved, and emotion image generation can be achieved while achieving emotion recognition.

[0028] First, the implementation steps of the emotion recognition and emotion image generation method provided by the present application will be described in detail below with reference to the accompanying drawings.

[0029] The emotion recognition and emotion image generation method provided in the present application can be applied to a terminal, a server, or software running in a terminal or a server. The terminal can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN) and big data and artificial intelligence platforms. In addition, the server can also be a node server in a blockchain network, but is not limited to this. Among them, blockchain is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm.

[0030] Reference Figure 2 , Figure 2It is a flowchart of the emotion recognition and emotion image generation method provided in the present application. The emotion recognition and emotion image generation method may include the following steps S101-S103.

[0031] S101, obtaining a sample scene query identifier, a sample object query identifier, and a sample image set.

[0032] It should be noted that the sample scene query identifier is used to indicate the scene features of all sample images, which is the visual representation of the scene level, while the sample object query identifier is used to indicate the object features of all sample images, which is the visual representation of the object level. In addition, all sample images share the sample scene query identifier and the sample object query identifier. Among them, the sample scene query identifier and the sample object query identifier are both learnable identifiers (Tokens), which can be learned on all sample images and optimized through the loss function, so that the joint processing model has the ability to identify and extract scene features and object features in the image.

[0033] It is understandable that the sample image set may include several sample images and emotion labels corresponding to each sample image. The sample image may contain scene content and object content. The scene content refers to the type of the background part of the sample image, and the object content refers to the type of object contained in the sample image. For example, in a certain image, the scene content is grass and the object content is a lion. The emotion label refers to the type of emotion expressed by the sample image. The corresponding emotion label can be obtained by labeling the sample image. Among them, the emotion type can be flexibly set according to the actual situation. For example, the emotion type can be entertainment, anger, disgust, excitement, fear, anger, etc., but it is not limited to this.

[0034] In this step, a sample scene query identifier and a sample object query identifier are obtained. The sample scene query identifier mainly indicates the scene-level features of all sample images, while the sample object query identifier indicates the object-level features of all sample images. At the same time, a number of sample images and the emotion labels corresponding to each sample image are obtained as a sample image set.

[0035] Optionally, both the sample scene query identifier and the sample object query identifier can be pre-set according to actual conditions, and this embodiment of the present application does not specifically limit this. Figure 3 As shown, Figure 3 The emotion type, scene query identifier and object query identifier corresponding to each visual input are displayed, and the scene query identifier and object query identifier are presented in the form of attention heat maps.

[0036] S102, performing model training according to the sample scene query identifier, the sample object query identifier and the sample image set to obtain the target scene query identifier, the target object query identifier and the joint processing model.

[0037] It should be noted that the target scene query identifier refers to the sample scene query identifier after training, which is used to guide the joint processing model to identify the scene features of the image to be tested. In other words, the joint processing model can identify and extract the scene-level visual representation in the image to be tested through the target scene query identifier; and the target object query identifier refers to the sample object query identifier after training, which is used to guide the joint processing model to identify the object features of the image to be tested. In other words, the joint processing model can identify and extract the object-level visual representation in the image to be tested through the target object query identifier. In addition, all images to be tested share the target scene query identifier and the target object query identifier.

[0038] It can be understood that the joint processing model refers to a model architecture that can simultaneously perform emotion recognition and emotion image generation.

[0039] In this step, the model training is carried out using sample scene query identifiers, sample object query identifiers and sample image sets, aiming to generate a joint processing model for simultaneous emotion recognition and emotion image generation, and obtain target scene query identifiers and target object query identifiers, thereby realizing the joint training of emotion recognition tasks and emotion image generation tasks and completing the mutual reinforcement between the two.

[0040] S103, performing emotion recognition and emotion image generation on the image to be tested according to the target scene query identifier, the target object query identifier and the joint processing model, to obtain the emotion type of the image to be tested and a plurality of emotion images associated with the emotion type of the image to be tested.

[0041] It should be noted that the emotional image associated with the emotional type of the image to be tested includes image content that matches the emotional type of the image to be tested. For example, if the emotional type of the image to be tested is anger, the emotional image associated with the emotional type of the image to be tested includes image content that matches anger.

[0042] It is understandable that the emotion type of the image to be tested refers to the emotion type expressed by the image to be tested. The emotion type can be flexibly set according to actual conditions, for example, the emotion type can be entertainment, anger, disgust, excitement, fear, anger, etc., but is not limited thereto.

[0043] In this step, first, the image to be tested is obtained, which contains scene content and object content; then, the target scene query identifier, the target object query identifier and the image to be tested are input into the joint processing model, and emotion recognition and emotion image generation are performed through the joint processing model to obtain the emotion type of the image to be tested and several emotion images associated with the emotion type of the image to be tested, thereby realizing the synchronous processing of the emotion recognition task and the emotion image generation task. In the process of emotion recognition and emotion image generation, the target scene query identifier can guide the joint processing model to identify and extract scene-level visual representations that are relevant to emotion and semantically rich, and the target object query identifier can guide the joint processing model to identify and extract object-level visual representations that are relevant to emotion and semantically rich. These visual representations can not only help the joint processing model to more accurately understand the emotional content in the image to be tested and improve the ability of the joint processing model in emotion understanding, but also help the joint processing model to capture the correlation between the emotion recognition task and the emotion image generation task, so that the joint processing model can more accurately capture and express subtle emotional differences, thereby effectively improving the accuracy of emotion recognition and the accuracy of emotion image generation. Since both the emotion recognition task and the emotion image generation task can be performed simultaneously, the efficiency of emotion recognition and the efficiency of emotion image generation can be effectively improved. For example, a lion image is input as the image to be tested, the scene attribute in the image is bushes, and the object attribute is lion. Through the above joint processing model, it can be identified that the emotion type of the image to be tested is anger, and several emotion images associated with anger are generated.

[0044] The application stage of the joint processing model in the embodiment of the present application will be further explained below.

[0045] In some implementations, the above-mentioned joint processing model may include:

[0046] A target emotion recognition model is used to perform emotion recognition on the image to be tested according to the target scene query identifier, the target object query identifier and the preset target category identifier, and obtain the target scene intermediate identifier, the target object intermediate identifier, the target category intermediate identifier and the emotion type of the image to be tested;

[0047] The target diffusion model is used to generate emotional images according to the target scene intermediate identifier, the target object intermediate identifier, the target category intermediate identifier and the image to be tested, and obtain a number of emotional images associated with the emotional type of the image to be tested.

[0048] It should be noted that the target scene query identifier refers to the scene query identifier obtained after processing the target scene query identifier by applying the target emotion recognition model, the target object intermediate identifier refers to the object query identifier obtained after processing the target object query identifier by applying the target emotion recognition model, and the target category intermediate identifier refers to the category identifier obtained after processing the target category identifier (Class Token) by applying the target emotion recognition model.

[0049] In this implementation, the joint processing model mainly includes two parts: a target emotion recognition model and a target diffusion model.

[0050] Among them, the target emotion recognition model is used for emotion recognition, which is specifically manifested in that emotion recognition is performed on the image to be tested according to the target scene query identifier, the target object query identifier and the preset target category identifier, and the target scene intermediate identifier, the target object intermediate identifier, the target category intermediate identifier and the emotion type of the image to be tested are obtained. In the process of emotion recognition, the target emotion recognition model can identify and capture the emotion-related and semantically rich scene-level visual representation in the image to be tested through the target scene query identifier, and can also identify and capture the emotion-related and semantically rich object-level visual representation in the image to be tested through the target object query identifier. Based on these multi-scale visual representations, the target emotion recognition model can more accurately understand the emotional content in the image to be tested, thereby effectively improving the accuracy of emotion recognition.

[0051] The target diffusion model is used to generate several emotional images, which is specifically manifested in generating emotional images according to the target scene intermediate identifier, the target object intermediate identifier, the target category intermediate identifier and the image to be tested, and obtaining several emotional images associated with the emotional type of the image to be tested. In the process of generating the emotional image, the target category intermediate identifier can encode a unique emotional representation, the target scene intermediate identifier provides the target diffusion model with scene-level semantic features, and the target object intermediate identifier provides the target diffusion model with object-level semantic features. These feature information can enable the target diffusion model to grasp more comprehensive emotional clues. These emotional clues can help the target diffusion model to more accurately capture and express subtle emotional differences, thereby promoting the generation of more expressive and delicate emotional images, and effectively improving the accuracy of emotional image generation.

[0052] Optionally, the target category identifier may be preset according to actual conditions, and this embodiment does not specifically limit this. For example, the target category identifier may be a preset vector, but is not limited thereto.

[0053] In some embodiments, reference Figure 4 , Figure 4 The principle diagram of the emotion recognition and emotion image generation method provided in this application, Figure 4 In the above example, “lion” refers to lion and “Bushland” refers to bush. The target emotion recognition model can include:

[0054] A linear mapping layer, used to obtain a first input sequence according to the image to be tested, the target scene query identifier and the target category identifier;

[0055] A first encoding structure is used to perform encoding processing on the first input sequence to obtain a first encoding sequence, where the first encoding sequence includes a first sub-scene query identifier, a first sub-category identifier, and a first sub-image block sequence;

[0056] A first interaction module is used to perform feature interaction on the first sub-scene query identifier and the first sub-image block sequence in the first coding sequence to obtain a first feature sequence, where the first feature sequence includes the target scene intermediate identifier, the first sub-category identifier and the first sub-image block sequence;

[0057] a second encoding structure, used to obtain a second input sequence according to the target object query identifier and the first feature sequence, and to encode the second input sequence to obtain a second encoding sequence, wherein the second encoding sequence includes the first sub-object query identifier, the second sub-scene query identifier, the second sub-category identifier, and the second sub-image block sequence;

[0058] A second interaction module is used to perform feature interaction on the first sub-object query identifier and the second sub-image block sequence in the second coding sequence to obtain a second feature sequence, where the second feature sequence includes the target object intermediate identifier, the second sub-scene query identifier, the second sub-category identifier and the second sub-image block sequence;

[0059] The third encoding structure is used to encode the second feature sequence to obtain an intermediate identifier of the target category;

[0060] The classification head is used to obtain the emotion type of the image to be tested based on the intermediate identifier of the target category.

[0061] In this embodiment, the abstract emotion understanding process is decomposed into a hierarchical visual chain, rather than just focusing on the existing feature extractor. The visual chain starts with scene-level features, and then identifies and extracts object-level features. These two levels of features are crucial for emotion recognition. These multi-scale prior features are then used to complete emotion recognition. In addition, this embodiment introduces two sets of learnable query identifiers and a multi-stage query mechanism, rather than being limited to directly inputting all query identifiers into the model for feature extraction. In this way, the target emotion recognition model can be prompted to effectively extract abstract emotion features and capture emotion-related and semantically rich visual representations, including object-level visual representations and scene-level visual representations, so that the target emotion recognition model has a more fine-grained emotion understanding capability, effectively improving the accuracy of emotion recognition.

[0062] Specifically, the present embodiment designs two sets of learnable query identifiers, namely the target scene query identifier and the target object query identifier, to decompose the emotion understanding process into a hierarchical visual chain structure. The function of the target scene query identifier is to extract scene-level features, and the function of the target object query identifier is to extract object-level features. In terms of structural design, the present embodiment decomposes the ViT model with an N-layer structure into three levels of encoding stages. The first stage is from the first encoder to the L1th encoder, the second stage is from the L1+1th encoder to the L2th encoder, and the third stage is from the L2+1th encoder to the Nth encoder. The encoders in each stage are connected in sequence, and all encoders are Transformer encoders. On this basis, the present embodiment constructs an emotion recognition model based on a vision transformer (ViT) as a target emotion recognition model, which includes a linear mapping layer, a first encoding structure, a first interaction module, a second encoding structure, a second interaction module, a third encoding structure, and a classification head connected in sequence. Among them, the first encoding structure corresponds to the first stage, and the first encoding structure is in Figure 4 In the figure, it is represented as "encoder 1 to L1", the second encoding structure corresponds to the second stage, and the second encoding structure is Figure 4 In the figure, it is represented as "encoder L1+1~L2", the third encoding structure corresponds to the third stage, and the third encoding structure is Figure 4 In the coder L2+1~N. Figure 4 The interaction module located between the first coding structure and the second coding structure is the first interaction module, and the interaction module located between the second coding structure and the third coding structure is the second interaction module.

[0063] More specifically, in the linear mapping layer, the input image is first divided into a number of non-overlapping image blocks (Patch), and then the several image blocks are linearly mapped to project them into the feature space to obtain a number of image block identifiers to be tested (Patch Tokens), and then the several image block identifiers to be tested are spliced ​​to obtain a sequence of image blocks to be tested, and finally, a target category identifier is added to the starting position of the image block sequence to be tested, and a target scene query identifier is added to the end position of the image block sequence to be tested, so as to obtain a first input sequence.

[0064] In the first coding structure, the first input sequence is coded by the first encoder to the L1th encoder to achieve feature extraction, thereby obtaining a first coded sequence. This coding process preliminarily establishes the correlation between the target scene query identifier and the image block identifier to be tested, which can be expressed as the following formula (1):

[0065]

[0066] In formula (1), represents the first coding sequence, is the first sub-scene query identifier, is the first subcategory identifier, is a first sub-image block sequence, and the first sub-image block sequence may be formed by splicing a plurality of first sub-image block identifiers; It is the encoding process from the first encoder to the L1th encoder; represents the first input sequence, is the image block sequence to be tested, is the target category identifier, is the target scene query identifier; [,,] represents the concatenation operation along the sequence dimension.

[0067] In order to enhance the interaction process between the scene query identifier and the image block identifier, the present embodiment further introduces a first interaction module for extracting more detailed scene features from the image block identifier. The interaction module is designed as a compact Transformer style network, which is sequentially provided with two layers of interaction structures and a layer of feature combination layer, the two layers of interaction structures are sequentially connected, and each layer of interaction structure includes a multi-head attention layer and a feedforward network. Specifically, the first encoding sequence is input into the first interaction module, and in each layer of interaction structure, the first sub-scene query identifier is first used as the query (Q), and the first sub-image block sequence is used as the key (K) and the value (V). The multi-head attention mechanism is used to process the first sub-scene query identifier and the first sub-image block sequence to obtain the first attention feature, and the first attention feature and the first sub-scene query identifier are added to obtain the first interaction identifier, and then the first interaction identifier is processed using a feedforward neural network (FNN) to obtain a second interaction identifier, and the second interaction identifier and the first interaction identifier are added to obtain a third interaction identifier. It can be understood that in the first interaction module, the third interaction identifier output by the first layer interaction structure is used as the first sub-scene query identifier of the second layer interaction structure, and the third interaction identifier output by the second layer interaction structure is input to the feature combination layer. For ease of understanding, the processing process of each interaction structure can be expressed as the following formula (2):

[0068]

[0069] In formula (2), represents a first interaction identifier; represents the first attention feature; A(·) represents the processing of the multi-head attention mechanism; represents the third interaction identifier; F(·) represents the processing of the feedforward network.

[0070] In the first interaction module, the third interaction identifier output by the second layer interaction structure is the optimized scene query identifier. On the basis of optimizing the scene query identifier, in the feature combination layer, the initial scene query identifier is combined with the optimized scene query identifier to further enhance the scene query capability of the model. Specifically, the product of the third interaction identifier output by the second layer interaction structure and a preset first modulation parameter is obtained as the first modulation identifier. The first modulation parameter is used to balance the contribution of the initial scene query identifier and the optimized scene query identifier to the scene features of the image, and the first modulation identifier and the first sub-scene query identifier are added to obtain the target scene intermediate identifier. The process can be expressed as the following formula (3):

[0071]

[0072] In formula (3), represents the target scene intermediate identifier; β1 represents the learnable first modulation parameter, which means that the first modulation parameter can be dynamically changed during the model training process. Among them, the initial value of the first modulation parameter before model training can be flexibly set according to the actual situation. For example, the initial value of the first modulation parameter can be 0.1, but it is not limited to this.

[0073] The first feature sequence can be obtained by the target scene intermediate identifier, the first subcategory identifier and the first sub-image block sequence, which can be expressed as Through the above steps, the target emotion recognition model can form a preliminary understanding of the global scene context of the entire image, enabling the target emotion recognition model to have coarse-grained emotion understanding capabilities.

[0074] On this basis, this embodiment introduces another query identifier specifically used to extract object-level features, namely the target object query identifier, to refine the emotion understanding ability of the target emotion recognition model. Specifically, the first feature sequence is input into the second encoding structure. In the second encoding structure, the target object query identifier is first added to the end of the first feature sequence to obtain a second input sequence, and then the second input sequence is encoded by the L1+1 encoder to the L2 encoder to achieve feature extraction, thereby obtaining a second encoding sequence. This encoding process preliminarily establishes the correlation between the target object query identifier and the first sub-image block identifier, and further enhances the correlation between the target scene intermediate identifier and the first sub-image block identifier, which is expressed as the following formula (4):

[0075]

[0076] In formula (4), represents the second coding sequence, is the second subcategory identifier, is a second sub-image block sequence, and the second sub-image block sequence may be formed by splicing a plurality of second sub-image block identifiers. Indicates the second sub-scene query identifier, Indicates the first child object query identifier; represents the second input sequence, Query the identifier for the target object; represents the encoding process from the L1+1th encoder to the L2th encoder; [,,,] represents the concatenation operation along the sequence dimension.

[0077] Similarly, in order to enhance the interaction process between the object query identifier and the sub-image block identifier, the present embodiment further introduces a second interaction module for extracting more detailed object features from the sub-image block identifier. The interaction module is also designed as a compact Transformer style network, which is sequentially provided with two layers of interaction structures and a layer of feature combination layer, the two layers of interaction structures are sequentially connected, and each layer of interaction structure includes a multi-head attention layer and a feedforward network. Specifically, the second encoding sequence is input into the second interaction module, and in each layer of interaction structure, the first sub-object query identifier is first used as the query (Q), and the second sub-image block sequence is used as the key (K) and the value (V). The multi-head attention mechanism is used to process the first sub-object query identifier and the second sub-image block sequence to obtain a second attention feature, and the second attention feature is added to the first sub-object query identifier to obtain a fourth interaction identifier, and then the fourth interaction identifier is processed using a feedforward network to obtain a fifth interaction identifier, and the fifth interaction identifier is added to the fourth interaction identifier to obtain a sixth interaction identifier. It can be understood that in the second interaction module, the sixth interaction identifier output by the first layer interaction structure will be used as the first sub-object query identifier of the second layer interaction structure, and the sixth interaction identifier output by the second layer interaction structure is input to the feature combination layer. For ease of understanding, the processing process of each interaction structure can be expressed as the following formula (5):

[0078]

[0079] In formula (5), represents a fourth interaction identifier; Indicates the second attention feature; Indicates the sixth interaction identifier.

[0080] In the second interaction module, the sixth interaction identifier output by the second layer interaction structure is the optimized object query identifier. On the basis of optimizing the object query identifier, in the feature combination layer, the initial object query identifier is combined with the optimized object query identifier to further enhance the object query capability of the model. Specifically, the product of the sixth interaction identifier output by the second layer interaction structure and a preset second modulation parameter is obtained as the second modulation identifier. The second modulation parameter is used to balance the contribution of the initial object query identifier and the optimized object query identifier to the object features of the image, and the second modulation identifier is added to the first sub-object query identifier to obtain the target object intermediate identifier. The process can be expressed as the following formula (6):

[0081]

[0082] In formula (6), is the intermediate identifier of the target object; β2 represents the learnable second modulation parameter, which means that the second modulation parameter can be dynamically changed during the model training process. The initial value of the second modulation parameter before model training can be flexibly set according to the actual situation. For example, the initial value of the first modulation parameter can be 0.1, but it is not limited to this.

[0083] The second feature sequence can be obtained by the target object intermediate identifier, the second sub-scene query identifier, the second sub-category identifier and the second sub-image block sequence, which is expressed as Through the above steps, the target emotion recognition model can form a preliminary understanding of the global object context of the entire image, and promote the emotion understanding ability of the target emotion recognition model to evolve from coarse-grained to fine-grained.

[0084] Subsequently, the second feature sequence is input into the third coding structure. In the third coding structure, the second feature sequence is coded by the L2+1th encoder to the Nth encoder to realize feature extraction, thereby obtaining a third coding sequence, which may include the target category intermediate identifier, the second sub-object query identifier, the third sub-scene query identifier and the third sub-image block sequence, and the target category intermediate identifier is extracted from the third coding sequence. This process further enhances the correlation between the second sub-scene query identifier and the first sub-object query identifier and the second sub-image block identifier, which is expressed as the following formula (7):

[0085]

[0086] In formula (7), represents the third coding sequence, represents the target category intermediate identifier, represents a third sub-image block sequence, and the third sub-image block sequence may be formed by splicing a plurality of third sub-image block identifiers; Indicates the third sub-scene query identifier; Indicates the second child object query identifier; It represents the encoding process from the L2+1th encoder to the Nth encoder.

[0087] Finally, the target classification intermediate identifier is input into the classification head, and the classification processing is performed by the classification head to obtain the emotion type of the image input into the target emotion recognition model to realize emotion recognition. Optionally, the classification head can be any classification head that can realize the classification processing, which will not be described in detail. It can be seen that the scene query identifier and the object query identifier work together in several encoders of the target emotion recognition model to form a hierarchical visual chain, through which a novel emotion understanding process can be constructed, which can effectively identify abstract emotions while extracting multi-scale emotion features to support the unity of dual tasks, thereby improving the accuracy of emotion recognition.

[0088] In some embodiments, reference Figure 4 , the above target diffusion model can include:

[0089] A fusion module, used for obtaining the diffusion condition to be tested according to the target scene intermediate identifier, the target object intermediate identifier and the target category intermediate identifier;

[0090] A fourth coding structure is used to perform coding processing on the diffusion condition to be tested and the image to be tested to obtain the characteristics of the condition to be tested and the characteristics of the image to be tested;

[0091] The image generation module is used to generate emotional images according to the characteristics of the conditions to be tested and the characteristics of the images to be tested, so as to obtain a plurality of emotional images associated with the emotional types of the images to be tested.

[0092] In this embodiment, it can be observed from the aforementioned visual chain of emotion understanding that the target category intermediate identification before the classification head can encode a unique emotion expression, while the multi-stage query identification (i.e., the target scene intermediate identification and the target object intermediate identification) captures a variety of emotional semantic features. By complementing each other with these two parts, a clear and diverse emotional space can be formed, thereby better guiding the diffusion model to generate emotionally clear and diverse images. Based on this, this embodiment constructs a conditional target diffusion model, which can include a fusion module, a fourth encoding structure, and an image generation module.

[0093] Specifically, the fusion module is used to generate diffusion conditions, which is specifically manifested in constructing the diffusion conditions to be tested by representing the target category intermediate identifiers representing the emotional expression, the target scene intermediate identifiers representing the emotional semantic features of the scene, and the target object intermediate identifiers representing the emotional semantic features of the object. The diffusion conditions to be tested represent the conditions of the current emotional expression, which will be used as the conditional input of the diffusion model to prompt the target diffusion model to grasp more comprehensive emotional clues. These emotional clues can help the target diffusion model to more accurately capture and express subtle emotional differences, thereby guiding the diffusion model to complete the image generation process. The fourth encoding structure is used to encode the diffusion conditions to be tested and the image to be tested, which is specifically manifested in encoding the image to be tested through a preset image encoder to obtain the image features to be tested, and encoding through a preset text encoder to obtain text embedding as the conditional features to be tested. The image generation module is a conditional diffusion model, which includes a forward denoising process and a backward denoising process. The type of the conditional diffusion model can be flexibly set according to the actual situation. For example, the conditional diffusion model can be a Classifier-free Guidance, but is not limited to this. In this way, it can promote the generation of more expressive and delicate emotional images, thereby effectively improving the accuracy of emotional image generation.

[0094] Optionally, the image encoder and the text encoder in the fourth coding structure can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the image encoder can be an image encoder based on a pre-training method or model of contrastive language-image pairs (CLIP), and the text encoder can be a text encoder of the CLIP model, but is not limited thereto.

[0095] In some embodiments, reference Figure 4 and Figure 5 , the diffusion condition to be tested is obtained according to the target scene intermediate identifier, the target object intermediate identifier and the target category intermediate identifier, and may include:

[0096] Linearly map the target scene intermediate identifier, the target object intermediate identifier and the target category intermediate identifier to obtain the target scene intermediate feature, the target object intermediate feature and the target category intermediate feature;

[0097] Obtaining the product of the target scene intermediate feature and the emotion correlation coefficient of the target scene intermediate identifier as the target scene feature; wherein the emotion correlation coefficient of the target scene intermediate identifier is used to indicate the correlation between the target scene intermediate identifier and the emotion type of the image to be tested;

[0098] Obtaining the product of the target object intermediate feature and the emotion correlation coefficient of the target object intermediate mark as the target object feature; wherein the emotion correlation coefficient of the target object intermediate mark is used to indicate the correlation between the target object intermediate mark and the emotion type of the image to be tested;

[0099] The target scene features, target object features and target category intermediate features are fused to obtain the diffusion conditions to be tested.

[0100] In this embodiment, before generating the diffusion conditions to be tested, independent multilayer perceptrons (MLP) are introduced for the target scene intermediate identifier, the target object intermediate identifier, and the target category intermediate identifier to map them all to the input space of the diffusion model. Specifically, the target scene intermediate features are obtained by linearly mapping the target scene intermediate identifier through the first multilayer perceptron, and the target object intermediate identifier is linearly mapped through the second multilayer perceptron to obtain the target object intermediate features, and the target category intermediate identifier is linearly mapped through the third multilayer perceptron to obtain the target category intermediate features. Figure 4 It is denoted as a “linear layer” in .

[0101] From the aforementioned visual chain of emotion understanding, it can be observed that multi-stage query identifiers can cover a series of levels of scene and object characteristics, and the emotional valence of these characteristics may vary significantly. For example, some elements show strong emotional resonance, such as the Ferris wheel associated with pleasure, flames associated with anger, etc., while other elements are emotionally neutral, such as trees and grass associated with multiple emotion types. In order to address these differences, this embodiment proposes an emotional correlation coefficient for quantitatively evaluating the relationship between each query identifier and the emotional representation, so as to measure and guide the importance of multi-stage query identifiers in the emotional image generation task, thereby further optimizing the performance of the target diffusion model. In other words, the emotional correlation coefficient will be used as the weight of the query identifier to evaluate the importance of these query identifiers in the fusion process.

[0102] Before calculating the emotion correlation coefficient of the target scene intermediate identifier and the emotion correlation coefficient of the target object intermediate identifier, this embodiment pre-integrates all images in the EmoSet data set, each image having corresponding scene attributes and object attributes, and then constructs a scene attribute database through the scene attributes of these images, and constructs an object attribute database through the object attributes of these images.

[0103] For the sentiment correlation coefficient of the target object intermediate identifier, first, the cosine similarity between each object attribute in the object attribute database and the target object intermediate identifier is calculated, and the object attribute with the highest cosine similarity is selected as the target object attribute, which is the attribute that is most semantically relevant to the target object intermediate identifier; then, all images with the target object attribute in the EmoSet dataset are input into the pre-trained sentiment classifier, and the sentiment classifier outputs the logarithmic probability that each image with the target object attribute is classified as the sentiment type of the image to be tested; finally, the sentiment correlation coefficient of the target object intermediate identifier can be calculated based on these logarithmic probabilities. Among them, the sentiment correlation coefficient of the target object intermediate identifier is used to indicate the correlation between the target object intermediate identifier and the sentiment type of the image to be tested. The larger the sentiment correlation coefficient of the target object intermediate identifier is, the more relevant the target object intermediate identifier is to the sentiment representation of the image to be tested, that is, the more relevant the target object intermediate identifier is to the sentiment type of the image to be tested, and vice versa. For ease of understanding, the sentiment correlation coefficient of the target object intermediate identifier satisfies the following formula (8):

[0104]

[0105] In formula (8), α Object Represents the sentiment correlation coefficient of the target object’s middle mark; M j represents the total number of images with target object attribute j; I K represents an image with target object attribute j; l(I K , e) represents image I K The logarithmic probability of being classified as the emotion type e of the image under test.

[0106] For the sentiment correlation coefficient of the target scene intermediate identifier, first, the cosine similarity between each scene attribute in the scene attribute database and the target scene intermediate identifier is calculated, and the scene attribute with the highest cosine similarity is selected as the target scene attribute, which is the attribute that is most semantically relevant to the target scene intermediate identifier; then, all images with the target scene attribute in the EmoSet dataset are input into the pre-trained sentiment classifier, and the sentiment classifier outputs the logarithmic probability that each image with the target scene attribute is classified as the sentiment type of the image to be tested; finally, the sentiment correlation coefficient of the target scene intermediate identifier can be calculated based on these logarithmic probabilities. Among them, the sentiment correlation coefficient of the target scene intermediate identifier is used to indicate the correlation between the target scene intermediate identifier and the sentiment type of the image to be tested. The larger the sentiment correlation coefficient of the target scene intermediate identifier is, the more relevant the target scene intermediate identifier is to the sentiment representation of the image to be tested, that is, the more relevant the target scene intermediate identifier is to the sentiment type of the image to be tested, and vice versa. For ease of understanding, the sentiment correlation coefficient of the target scene intermediate identifier satisfies the following formula (9):

[0107]

[0108] In formula (9), α Scene represents the sentiment correlation coefficient of the target scene middle mark; M j′ represents the total number of images with target scene attribute j′; I K′ represents an image with target scene attribute j′; l(I K′ ,e) represents image I K′ The logarithmic probability of being classified as the emotion type e of the image under test.

[0109] For example, refer to Figure 5 , a scene attribute database and an object attribute database are pre-built. The scene attribute database may include multiple scene attributes, such as amusement park, lake, landfill, and boxing ring; the object attribute database may include multiple object attributes, such as waves, carousels, caterpillars, surfboards, sculptures, spiders, lions, etc. Emotion types may include eight types: anger, disgust, awe, fear, entertainment, excitement, satisfaction, and frustration. The image to be tested is an image of a carousel. Through the target emotion recognition model, its emotion type can be obtained as entertainment. Through cosine similarity recognition, the attribute most relevant to the semantics of the target scene intermediate logo is amusement park, and through cosine similarity recognition, the attribute most relevant to the semantics of the target object intermediate logo is carousel. Then, according to the target object attribute and the emotion type of the image to be tested combined with the above formula (8), the emotion correlation coefficient of the target object intermediate logo can be calculated to be 0.81, and according to the target scene attribute and the emotion type of the image to be tested combined with the above formula (9), the emotion correlation coefficient of the target scene intermediate logo can be calculated to be 0.88.

[0110] After obtaining the emotional correlation coefficient of the target scene intermediate mark and the emotional correlation coefficient of the target object intermediate mark, first, the product of the target scene intermediate feature and the emotional correlation coefficient of the target scene intermediate mark is obtained as the target scene feature, and the product of the target object intermediate feature and the emotional correlation coefficient of the target object intermediate mark is obtained as the target object feature. Then, the target scene feature, the target object feature and the target category intermediate feature are fused. The fusion is realized in the form of element-by-element addition, so as to obtain the diffusion condition to be tested, as shown in the following formula (10):

[0111]

[0112] In formula (10), c represents the diffusion condition to be measured; Represents the feature fusion operation.

[0113] The training phase of the joint processing model in the embodiment of the present application will be further explained below.

[0114] In some embodiments, reference Figure 1 Part (c) and Figure 4 , Figure 1 Part (c) is a training principle diagram of the joint processing model of an embodiment of the present application. The model training is performed according to the sample scene query identifier, the sample object query identifier and the sample image set to obtain the target scene query identifier, the target object query identifier and the joint processing model, which may include:

[0115] Performing a first-stage training on a preset neural network model according to a sample scene query identifier, a sample object query identifier, and a sample image set to obtain an original emotion recognition model, a pre-trained scene query identifier, and a pre-trained object query identifier;

[0116] According to the original emotion recognition model and the preset original diffusion model, an original processing model is constructed;

[0117] The original processing model is trained in the second stage according to the pre-trained scene query identifier, the pre-trained object query identifier and the sample image set to obtain the target scene query identifier, the target object query identifier and the joint processing model.

[0118] It should be noted that the structure of the neural network model, the structure of the original emotion recognition model and the structure of the aforementioned target emotion recognition model are the same, and the difference between the three lies in the parameters of the model. Similarly, the structure of the original diffusion model and the structure of the aforementioned target diffusion model are the same, and the difference between the two also lies in the parameters of the model. Furthermore, the structure of the original processing model is the same as the structure of the target processing model, and the difference between the two also lies in the parameters of the model.

[0119] In this embodiment, a phased training strategy is designed for the training of the joint processing model. The first phase of training is mainly used to train the emotion recognition model, which is specifically manifested in that the preset neural network model is trained through the sample scene query identifier, the sample object query identifier and the sample image set to obtain the original emotion recognition model, the pre-trained scene query identifier and the pre-trained object query identifier to complete the pre-training of the emotion recognition model and ensure the accuracy of the emotion recognition model. After completing the first phase of training, the original processing model is constructed according to the original emotion recognition model and the preset original diffusion model. The second phase of training is mainly used to train the original processing model, which is specifically manifested in that the second phase of training is performed on the original processing model through the pre-trained scene query identifier obtained by the first phase of training, the pre-trained object query identifier obtained by the first phase of training and the original sample image set to obtain the target scene query identifier, the target object query identifier and the joint processing model, thereby realizing the training of the joint processing model. In this way, the joint training of the emotion recognition task and the emotion image generation task can be realized, and the mutual reinforcement between the two can be completed.

[0120] In some embodiments, reference Figure 1 Part (c) and Figure 4 The first stage of training the preset neural network model according to the sample scene query identifier, the sample object query identifier and the sample image set to obtain the original emotion recognition model, the pre-trained scene query identifier and the pre-trained object query identifier may include:

[0121] Divide the sample image set into M batches of sample image subsets;

[0122] Performing a k-th batch of forward propagation on the neural network model based on the first network parameters and the k-th batch of sample image subsets; wherein the first network parameters include a sample scene query identifier, a sample object query identifier, and a model parameter of the neural network model, k=1, 2, 3, ..., M;

[0123] When the forward propagation of the kth batch is completed, the total loss function value of the neural network model is obtained according to the scene contrast loss value, object contrast loss value and classification loss value of the neural network model;

[0124] The first network parameters are updated by using the total loss function value of the neural network model in combination with the gradient back propagation method to obtain updated first network parameters;

[0125] When the first stage of training does not meet the first preset condition, the updated first network parameters are determined as new first network parameters, k=k+1, and the process returns to the step of performing the kth batch of forward propagation on the neural network model based on the first network parameters and the kth batch of sample image subsets, until the first stage of training meets the first preset condition, and the original emotion recognition model, the pre-trained scene query identifier, and the pre-trained object query identifier are obtained according to the updated first network parameters;

[0126] The above steps start from k=1.

[0127] In this implementation, in the first stage of training, first, the sample image set is divided into M batches of sample image subsets, and each batch of sample image subsets includes multiple sample images and emotion labels corresponding to each sample image.

[0128] Then, the neural network model is forward propagated for the kth batch based on the first network parameters and the kth batch of sample image subsets. During the forward propagation process, each sample image in the kth batch of sample image subsets is subjected to a data flow similar to that of the target emotion recognition model to train the neural network model. Among them, the first network parameters include the sample scene query identifier, the sample object query identifier and the model parameters of the neural network model. The model parameters of the neural network model include the hyperparameters of the first interaction module, the second interaction module and the classification head. The specific types of the hyperparameters can be flexibly set according to the actual situation. Through the forward propagation process, the predicted emotion label, the sample scene intermediate identifier and the sample object intermediate identifier of each sample image in the kth batch of sample image subsets can be obtained. Among them, the predicted emotion label refers to the emotion label predicted by the neural network model during the forward propagation process; the sample scene intermediate identifier is the scene intermediate identifier output by the first interaction module, and its definition is the same as that of the aforementioned target scene intermediate identifier; and the sample object intermediate identifier is the object intermediate identifier output by the second interaction module, and its definition is the same as that of the aforementioned target object intermediate identifier. It should be noted that the image block identifier of each sample image performs feature interaction with the scene query identifier and the object query identifier in the first interaction module and the second interaction module. Since the scene attributes and object attributes of each sample image are different, the sample scene intermediate identifier and the sample object intermediate identifier of each sample image are also different.

[0129] When the forward propagation of the kth batch is completed, in order to enable the scene query identifier to have the function of guiding the emotion recognition model to extract scene features from the image, the present embodiment introduces a scene contrast loss function to achieve semantic supervision and alignment. The scene contrast loss function is a loss function based on a contrast learning mechanism. The contrast learning mechanism can effectively enhance the correlation between the scene query identifier and the emotional text description. For example, for a lion image, the scene attribute in the image is a bush. The text encoder of the CLIP model can be used to encode the corresponding scene attribute and use it as a supervision signal for the scene query identifier. More specifically, the text encoder of the CLIP model is used in advance to encode the scene attributes of each sample image in the sample image subset of the kth batch to obtain the scene encoding features of each sample image in the sample image subset of the kth batch. When the forward propagation is completed, the scene contrast loss value of the neural network model is obtained according to the scene encoding features of each sample image in the sample image subset of the kth batch and the sample scene intermediate identifier, combined with the following formula (11):

[0130]

[0131] In formula (11), Represents the scene contrast loss value; V represents the total number of sample images in the sample image subset of the kth batch; Represents the scene encoding features of the i-th sample image in the k-th batch of sample image subsets; represents the sample scene intermediate identifier of the i-th sample image in the sample image subset of the k-th batch; sim(·) represents the operation of dot product similarity; τ represents the preset temperature coefficient, which can be flexibly set according to actual conditions.

[0132] Similarly, in order to enable the object query identifier to have the function of guiding the emotion recognition model to extract object features from the image, the present embodiment introduces an object contrast loss function to achieve semantic supervision and alignment. The object contrast loss function is a loss function based on a contrast learning mechanism. The contrast learning mechanism can effectively enhance the correlation between the object query identifier and the emotional text description. For example, for a lion image, the object attribute in the image is a lion. The text encoder of the CLIP model can be used to encode the corresponding object attribute and use it as a supervisory signal for the object query identifier. More specifically, the text encoder of the CLIP model is used in advance to encode the object attributes of each sample image in the sample image subset of the kth batch to obtain the object encoding features of each sample image in the sample image subset of the kth batch. When the forward propagation is completed, the object contrast loss value of the neural network model is obtained according to the object encoding features of each sample image in the sample image subset of the kth batch and the intermediate identifier of the sample object, combined with the following formula (12):

[0133]

[0134] In formula (12), Represents the object contrast loss value of the neural network model; Represents the object encoding features of the i-th sample image in the k-th batch of sample image subsets; Represents the intermediate identifier of the sample object in the i-th sample image in the k-th batch of sample images.

[0135] In order to measure the loss of the neural network model in the sentiment classification task, the cross entropy loss value is obtained as the classification loss value of the neural network model according to the predicted sentiment label and sentiment label of each sample image in the kth batch of sample image subsets.

[0136] Finally, the scene contrast loss value, object contrast loss value, and classification loss value of the neural network model are weighted to obtain the total loss function value of the neural network model, as shown in the following formula (13):

[0137]

[0138] In formula (13), Represents the total loss function value of the neural network model, λ1 represents the weight of the scene contrast loss value, λ2 represents the weight of the object contrast loss value, and λ3 represents the weight of the classification loss value. The weight is used to balance the contribution of each loss value in the first stage of training.

[0139] Afterwards, the total loss function value of the neural network model is combined with the gradient back propagation method to update the first network parameters, that is, update the hyperparameters of the first interaction module, the hyperparameters of the second interaction module, the hyperparameters of the classification head, the sample object query identifier and the sample scene query identifier, so as to obtain the updated first network parameters.

[0140] Finally, determine whether the training of the first stage has reached the first preset condition. The first preset condition refers to the termination condition of the training of the first stage, which can be flexibly set according to the actual situation. For example, the first preset condition can be that the number of training times in the first stage reaches the preset total number of times, but it is not limited to this. If not, the updated first network parameters are determined as the new first network parameters, k=k+1, and return to the forward propagation step of the kth batch to achieve implicit feedback. If so, the original emotion recognition model is constructed using the updated first network parameters, and the sample scene query identifier in the updated first network parameters is determined as the pre-training scene query identifier, and the sample object query identifier in the updated first network parameters is determined as the pre-training object query identifier. In this way, pre-training of the emotion recognition model is achieved, which can effectively improve the performance of the emotion recognition model.

[0141] In some embodiments, reference Figure 1 Part (c) and Figure 4 The second stage of training the original processing model according to the pre-trained scene query identifier, the pre-trained object query identifier and the sample image set to obtain the target scene query identifier, the target object query identifier and the joint processing model may include:

[0142] Divide the sample image set into N′ batches of sample image subsets;

[0143] Performing a k'th batch forward propagation on the original processing model based on the second network parameters and the k'th batch of sample image subsets; wherein the second network parameters include a pre-trained scene query identifier, a pre-trained object query identifier, a model parameter of the original emotion recognition model, and a model parameter of the original diffusion model, k'=1, 2, 3, ..., N';

[0144] When the forward propagation of the k'th batch is completed, the total loss function value of the original processing model is obtained according to the total loss function value of the original emotion recognition model, the conditional loss function value of the original diffusion model, and the denoising loss function value of the original diffusion model;

[0145] The second network parameters are updated by using the total loss function value of the original processing model combined with the gradient back propagation method to obtain updated second network parameters;

[0146] When the second stage of training does not meet the second preset condition, the sample image subset of the k'+1th batch is augmented to obtain the amplified image set of the k'+1th batch as the new sample image subset of the k'+1th batch, the updated second network parameters are determined as the new second network parameters, k'=k'+1, and the process returns to the step of performing the k'th batch forward propagation on the original processing model based on the second network parameters and the sample image subset of the k'th batch, until the second stage of training meets the second preset condition, and the joint processing model, the target scene query identifier and the target object query identifier are obtained according to the updated second network parameters;

[0147] The above steps start from k'=1.

[0148] In this implementation, in the second stage of training, first, the sample image set is divided into N' batches of sample image subsets, and each batch of sample image subsets includes multiple sample images and emotion labels corresponding to each sample image.

[0149] Then, the original processing model is forward propagated for the k'th batch based on the second network parameters and the k'th batch of sample image subsets. During the forward propagation process, each sample image in the k'th batch of sample image subsets is subjected to a data flow similar to that of the joint processing model to train the original processing model. The second network parameters include pre-trained scene query identifiers, pre-trained object query identifiers, model parameters of the original emotion recognition model, and model parameters of the original diffusion model. The model parameters of the original emotion recognition model may include hyperparameters of the first interaction module, the second interaction module, and the classification head. The model parameters of the original diffusion model may include hyperparameters of the forward process and hyperparameters of the backward process. The specific types of hyperparameters may be flexibly set according to actual conditions. Through the forward propagation process, the predicted emotion labels, sample scene intermediate identifiers, sample object intermediate identifiers, and multiple sample emotion images of each sample image in the k'th batch of sample image subsets may be obtained. The sample emotion images refer to the emotion images generated by the original diffusion model through the forward propagation process.

[0150] When the k'th batch of forward propagation is completed, the total loss function value of the original emotion recognition model is obtained, and its specific implementation has been recorded in the above-mentioned implementation method, which will not be repeated here. At the same time, the conditional loss function value and denoising loss function value of the original diffusion model are obtained.

[0151] Specifically, based on the aforementioned correlation coefficient, the diffusion condition is explicitly supervised to ensure that it is aligned with the target emotional space. To this end, this embodiment proposes a conditional loss function. Among them, the target emotional space is a pre-constructed feature distribution set, which includes feature distribution vectors corresponding to each emotional type. More specifically, based on the principle of contrastive learning, each sample emotional image and its corresponding emotional label of the k'th batch are used as positive samples, and the selection of negative samples is guided by the original emotional recognition model, that is: for each sample emotional image of the k'th batch, the original emotional recognition model is used to predict the sample emotional image to obtain the predicted emotional label of the sample emotional image, and the emotional label of the sample emotional image is compared with the predicted emotional label. If the emotional label is equal to the predicted emotional label, it means that the emotional recognition is correct. At this time, the emotional type with the second highest score (i.e., prediction probability) is selected as the negative sample. If the emotional label is not equal to the predicted emotional label, it means that the emotional recognition is wrong. At this time, the predicted emotional label is selected as the negative sample. In addition, this embodiment adds an additional classification head to the diffusion condition to obtain the emotional distribution. The purpose of this step is to make the diffusion condition fall into the current target emotional space. On this basis, define e + is a positive sample, e - is a negative sample, e p is the sentiment distribution, for the triple (e + , e - , ep ), the conditional loss function value of the original diffusion model satisfies the following formula (14):

[0152]

[0153] In formula (14), represents the conditional loss function value of the original diffusion model, d(e p , e + ) represents the sentiment distribution e p and positive sample e + The cosine similarity of d(e p , e - ) represents the sentiment distribution e p and negative samples e - cosine similarity; ξ refers to the margin hyperparameter, which is used to control the minimum distance. Its value can be flexibly set according to actual conditions. For example, the margin hyperparameter can be 0.5, but is not limited to this.

[0154] As for the denoising loss function value of the original diffusion model, it can be expressed as the following formula (15):

[0155]

[0156] In formula (15), Represents the denoising loss function value of the original diffusion model; ∈ θ (·) represents the operation of the denoising network; x t represents the noise data at time step t; represents the time-dependent variance scheduling parameter, which is a predefined parameter of the diffusion model and is used to control the degree of noise added in the forward process, which is equivalent to adding noise to the original image at each time step; ∈ represents Gaussian noise; τ(c) represents the diffusion condition after encoding by the text encoder of the CLIP model, and its definition is the same as that of the aforementioned conditional feature to be tested; Represents the process of mean squared error loss.

[0157] Finally, the total loss function value of the original emotion recognition model, the conditional loss function value of the original diffusion model, and the denoising loss function value of the original diffusion model are weighted to obtain the total loss function value of the original processing model, as shown in the following formula (16):

[0158]

[0159] In formula (16), represents the total loss function value of the original processing model; γ1 represents the weight of the total loss function value of the original emotion recognition model; γ2 represents the weight of the conditional loss function value of the original diffusion model; γ3 represents the weight of the denoising loss function value of the original diffusion model. The weights are used to balance the contribution of each loss value in joint training.

[0160] Afterwards, the total loss function of the original processing model is combined with the gradient back propagation method to update the second network parameters, that is, update the hyperparameters of the first interaction module, the hyperparameters of the second interaction module, the hyperparameters of the classification head, the pre-trained object query identifier, the pre-trained scene query identifier, the hyperparameters of the fusion module and the hyperparameters of the original diffusion model, so as to obtain the updated second network parameters.

[0161] Finally, it is determined whether the training of the second stage reaches the second preset condition. The second preset condition refers to the termination condition of the training of the second stage, which can be flexibly set according to the actual situation. For example, the second preset condition can be that the number of training times of the second stage reaches the preset total number of times, but it is not limited to this. If not, the sample image subset of the k'+1 batch is amplified to obtain the amplified image set of the k'+1 batch as the new sample image subset of the k'+1 batch, so as to update the sample images of the next batch, and at the same time, the updated second network parameters are determined as the new second network parameters, and k'=k'+1 is set, and the forward propagation step of the k' batch is returned to realize implicit feedback. If so, the joint processing model is constructed using the updated second network parameters, and the pre-trained scene query identifier in the updated second network parameters is determined as the target scene query identifier, and the pre-trained object query identifier in the updated second network parameters is determined as the target object query identifier. In this way, the joint training of the emotion recognition task and the emotion image generation task is realized, and the mutual reinforcement between the two is completed.

[0162] In some embodiments, reference Figure 1 Part (c) of the above process of amplifying the sample image subset of the k'+1th batch to obtain the amplified image set of the k'+1th batch may include:

[0163] Obtain multiple sample emotion images of the k'th batch; wherein the sample emotion images are images generated by forward propagation of the k'th batch;

[0164] At least one amplified sample image is obtained by screening multiple sample emotion images in the k'th batch according to the emotion accuracy parameter and the semantic clarity parameter of each sample emotion image;

[0165] At least one amplified sample image is merged with the sample image subset of the k'+1th batch to obtain a set of amplified images of the k'+1th batch.

[0166] In this embodiment, just as emotion recognition can promote the effect of emotion image generation, emotion image generation can also promote the effect of emotion recognition. By generating diverse and expressive emotion images, the limited set of sample images can be effectively expanded, providing more samples for training and optimization of the emotion recognition model. This not only helps to alleviate the problem of data scarcity, but also enhances the ability of the emotion recognition model to recognize different emotional expressions. In addition, the generative feedback generated during the generation of emotion images can help the emotion recognition model optimize and refine its extraction and understanding of emotional features, thereby further improving the accuracy and robustness of the emotion recognition model. In order to make full use of this feature, this embodiment designs an efficient data filtering method for screening high-quality sample emotion images. The screened images will be used for further training of the model, thereby establishing an explicit feedback loop, which complements the implicit feedback of the above-mentioned loss function.

[0167] Specifically, the present embodiment uses the emotional accuracy parameter (Emo-A) and the semantic clarity parameter (Sem-C) as the screening criteria, and selects the sample emotional images with higher emotional accuracy parameters and higher semantic clarity parameters from the multiple sample emotional images of the k'th batch as the augmented sample images, and then merges at least one augmented sample image with the sample image subset of the k'+1th batch to obtain the augmented image set of the k'+1th batch. For example, the sample emotional images with the emotional accuracy parameter greater than the first preset threshold and the semantic clarity parameter greater than the second preset threshold are selected as the augmented sample images. For another example, all the sample emotional images are sorted from large to small according to the emotional accuracy parameter, and the top 50% of the sample emotional images are selected as candidate images, and then all the candidate images are sorted from large to small according to the semantic clarity parameter, and the top 50% of the candidate images are selected as the augmented sample images. By selecting images that are within a certain range on these two indicators, the present embodiment achieves a balance between emotional expressiveness and semantic clarity. This dual constraint ensures that only those high-quality images that can effectively arouse emotions and are semantically easy to understand are retained for further training.

[0168] More specifically, the emotion accuracy parameter of the sample emotion image verifies whether the sample emotion image conveys the expected emotion state. The higher the emotion accuracy parameter of the sample emotion image, the more relevant the sample emotion image is to its corresponding emotion label, and vice versa. By way of example, in some embodiments, emotion recognition is performed on the sample emotion image through a preset classifier to obtain a predicted emotion label of the sample emotion image, and the cosine similarity between the emotion label of the sample emotion image and the predicted emotion label is obtained as the emotion accuracy parameter of the sample emotion image. By way of example, in other embodiments, emotion recognition is performed on the sample emotion image through a preset classifier to obtain the classification accuracy of the sample emotion image (i.e., the probability corresponding to the predicted emotion type) as the emotion accuracy parameter of the sample emotion image.

[0169] The semantic clarity parameter of the sample emotional image verifies the semantic clarity of the image content and prevents the image from becoming too abstract or vague. The higher the semantic clarity parameter of the sample emotional image, the clearer the semantics of the sample emotional image, that is, the more it can reflect the expected object characteristics and expected scene characteristics. Conversely, the less clear its semantics is, the less it can reflect the expected object characteristics and expected scene characteristics. The semantic clarity parameter of the sample emotional image satisfies the following formula (17):

[0170] Sem-C=max(c object (b), c scene (b)) (17);

[0171] In formula (17), Sem-C represents the semantic clarity parameter of the sample emotional image; b represents the sample emotional image; c object (·) represents a pre-trained object classifier, which outputs the object semantic score of the sample emotion image (i.e., the probability corresponding to the predicted object attribute); c scene (·) represents a pre-trained scene classifier, which outputs the scene semantic score of the sample emotion image (i.e., the probability corresponding to the predicted scene attribute).

[0172] Finally, the embodiment of the present application also provides an emotion recognition and emotion image generation device, including:

[0173] An acquisition module is used to acquire a sample scene query identifier, a sample object query identifier, and a sample image set; wherein the sample scene query identifier is used to indicate the scene features of all sample images, the sample object query identifier is used to indicate the object features of all sample images, and the sample image set includes a number of sample images and an emotion label corresponding to each sample image;

[0174] A training module is used to perform model training according to a sample scene query identifier, a sample object query identifier and a sample image set to obtain a target scene query identifier, a target object query identifier and a joint processing model; wherein the target scene query identifier is used to guide the joint processing model to identify scene features of the image to be tested, and the target object query identifier is used to guide the joint processing model to identify object features of the image to be tested;

[0175] The dual-task processing module is used to perform emotion recognition and emotion image generation on the image to be tested according to the target scene query identifier, the target object query identifier and the joint processing model, and obtain the emotion type of the image to be tested and several emotion images associated with the emotion type of the image to be tested.

[0176] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0177] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are expected, wherein the order of various operations is changed and the sub-operation described as a part of a larger operation is performed independently.

[0178] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0179] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several programs to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0180] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable programs for implementing the logical functions, and may be embodied in any computer-readable medium for use by a program execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch and execute a program from a program execution system, device or apparatus), or in conjunction with such program execution systems, devices or apparatuses. For purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by a program execution system, device or apparatus, or in conjunction with such program execution systems, devices or apparatuses.

[0181] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0182] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable program execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0183] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0184] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0185] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the described embodiments. Technical personnel familiar with the field may make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A method for emotion recognition and emotion image generation, characterized in that: The following steps are involved: Obtain a sample scene query identifier, a sample object query identifier, and a sample image set; wherein the sample scene query identifier is used to indicate the scene features of all the sample images, the sample object query identifier is used to indicate the object features of all the sample images, and the sample image set includes a plurality of sample images and an emotion label corresponding to each of the sample images; Model training is performed according to the sample scene query identifier, the sample object query identifier and the sample image set to obtain a target scene query identifier, a target object query identifier and a joint processing model; wherein the target scene query identifier is used to guide the joint processing model to identify scene features of the image to be tested, and the target object query identifier is used to guide the joint processing model to identify object features of the image to be tested; Emotion recognition and emotion image generation are performed on the image to be tested according to the target scene query identifier, the target object query identifier and the joint processing model to obtain the emotion type of the image to be tested and a plurality of emotion images associated with the emotion type of the image to be tested.

2. The emotion recognition and emotion image generation method according to claim 1, characterized in that: The joint processing model includes: A target emotion recognition model is used to perform emotion recognition on the image to be tested according to the target scene query identifier, the target object query identifier and a preset target category identifier, and obtain a target scene intermediate identifier, a target object intermediate identifier, a target category intermediate identifier and the emotion type of the image to be tested; The target diffusion model is used to generate an emotional image based on the target scene intermediate identifier, the target object intermediate identifier, the target category intermediate identifier and the image to be tested, so as to obtain a plurality of emotional images associated with the emotional type of the image to be tested.

3. The emotion recognition and emotion image generation method according to claim 2, characterized in that: The target emotion recognition model includes: A linear mapping layer, used for obtaining a first input sequence according to the image to be tested, the target scene query identifier and the target category identifier; A first encoding structure is used to perform encoding processing on the first input sequence to obtain a first encoding sequence, wherein the first encoding sequence includes a first sub-scene query identifier, a first sub-category identifier, and a first sub-image block sequence; a first interaction module, configured to perform feature interaction on a first sub-scene query identifier and a first sub-image block sequence in the first coding sequence to obtain a first feature sequence, wherein the first feature sequence includes the target scene intermediate identifier, the first sub-category identifier, and the first sub-image block sequence; a second encoding structure, used to obtain a second input sequence according to the target object query identifier and the first feature sequence, and to perform encoding processing on the second input sequence to obtain a second encoding sequence, wherein the second encoding sequence includes a first sub-object query identifier, a second sub-scene query identifier, a second sub-category identifier, and a second sub-image block sequence; a second interaction module, configured to perform feature interaction on the first sub-object query identifier and the second sub-image block sequence in the second coding sequence to obtain a second feature sequence, wherein the second feature sequence includes the target object intermediate identifier, the second sub-scene query identifier, the second sub-category identifier and the second sub-image block sequence; A third encoding structure is used to encode the second feature sequence to obtain the target category intermediate identifier; The classification head is used to obtain the emotion type of the image to be tested according to the target category intermediate identifier.

4. The emotion recognition and emotion image generation method according to claim 2, characterized in that: The target diffusion model includes: A fusion module, used for obtaining the diffusion condition to be tested according to the target scene intermediate identifier, the target object intermediate identifier and the target category intermediate identifier; A fourth coding structure is used to encode the diffusion condition to be tested and the image to be tested to obtain a feature of the condition to be tested and a feature of the image to be tested; The image generation module is used to generate an emotional image according to the characteristics of the conditions to be tested and the characteristics of the images to be tested, so as to obtain a plurality of emotional images associated with the emotional types of the images to be tested.

5. The method for emotion recognition and emotion image generation according to claim 4, characterized in that: The obtaining of the diffusion condition to be tested according to the target scene intermediate identifier, the target object intermediate identifier and the target category intermediate identifier comprises: Linearly mapping the target scene intermediate identifier, the target object intermediate identifier and the target category intermediate identifier to obtain a target scene intermediate feature, a target object intermediate feature and a target category intermediate feature; Acquire the product of the target scene intermediate feature and the emotion correlation coefficient of the target scene intermediate identifier as the target scene feature; wherein the emotion correlation coefficient of the target scene intermediate identifier is used to indicate the correlation between the target scene intermediate identifier and the emotion type of the image to be tested; Acquire the product of the target object intermediate feature and the emotion correlation coefficient of the target object intermediate identifier as the target object feature; wherein the emotion correlation coefficient of the target object intermediate identifier is used to indicate the correlation between the target object intermediate identifier and the emotion type of the image to be tested; The target scene feature, the target object feature and the target category intermediate feature are fused to obtain the diffusion condition to be measured.

6. The emotion recognition and emotion image generation method according to claim 1, characterized in that: The model training is performed according to the sample scene query identifier, the sample object query identifier and the sample image set to obtain the target scene query identifier, the target object query identifier and the joint processing model, including: Performing a first-stage training on a preset neural network model according to the sample scene query identifier, the sample object query identifier, and the sample image set to obtain an original emotion recognition model, a pre-trained scene query identifier, and a pre-trained object query identifier; Constructing an original processing model according to the original emotion recognition model and the preset original diffusion model; The original processing model is trained in the second stage according to the pre-trained scene query identifier, the pre-trained object query identifier and the sample image set to obtain the target scene query identifier, the target object query identifier and the joint processing model.

7. The method for emotion recognition and emotion image generation according to claim 6, characterized in that: The first stage of training a preset neural network model according to the sample scene query identifier, the sample object query identifier and the sample image set to obtain an original emotion recognition model, a pre-trained scene query identifier and a pre-trained object query identifier includes: Dividing the sample image set into M batches of sample image subsets; Performing a k-th batch of forward propagation on the neural network model based on the first network parameters and the k-th batch of sample image subsets; wherein the first network parameters include the sample scene query identifier, the sample object query identifier and the model parameters of the neural network model, k=1, 2, 3, ..., M; When the forward propagation of the kth batch is completed, the total loss function value of the neural network model is obtained according to the scene contrast loss value, the object contrast loss value and the classification loss value of the neural network model; Using the total loss function value of the neural network model combined with the gradient back propagation method to update the first network parameters, to obtain updated first network parameters; When the first stage of training does not meet the first preset condition, the updated first network parameters are determined as new first network parameters, k=k+1, and the process returns to the step of performing the kth batch of forward propagation on the neural network model based on the first network parameters and the kth batch of sample image subsets, until the first stage of training meets the first preset condition, and the original emotion recognition model, the pre-trained scene query identifier, and the pre-trained object query identifier are obtained according to the updated first network parameters; The above steps start from k=1.

8. The method for emotion recognition and emotion image generation according to claim 6, characterized in that: The second stage of training the original processing model according to the pre-trained scene query identifier, the pre-trained object query identifier and the sample image set to obtain the target scene query identifier, the target object query identifier and the joint processing model includes: Dividing the sample image set into N′ batches of sample image subsets; Performing a k′th batch of forward propagation on the original processing model based on the second network parameters and the k′th batch of sample image subsets; wherein the second network parameters include the pre-trained scene query identifier, the pre-trained object query identifier, the model parameters of the original emotion recognition model and the model parameters of the original diffusion model, k′=1, 2, 3, ..., N′; When the forward propagation of the k′th batch is completed, the total loss function value of the original processing model is obtained according to the total loss function value of the original emotion recognition model, the conditional loss function value of the original diffusion model and the denoising loss function value of the original diffusion model; The second network parameters are updated by using the total loss function value of the original processing model in combination with a gradient back propagation method to obtain updated second network parameters; When the second stage of training does not meet the second preset condition, the sample image subset of the k′+1th batch is augmented to obtain the augmented image set of the k′+1th batch as the new sample image subset of the k′+1th batch, the updated second network parameters are determined as the new second network parameters, k′=k′+1 is set, and the process returns to the step of performing the k′th batch forward propagation on the original processing model based on the second network parameters and the sample image subset of the k′th batch, until the second stage of training meets the second preset condition, and the joint processing model, the target scene query identifier and the target object query identifier are obtained according to the updated second network parameters; The above steps start from k'=1.

9. The method for emotion recognition and emotion image generation according to claim 8, characterized in that: The step of performing amplification processing on the sample image subset of the k'+1th batch to obtain the amplified image set of the k'+1th batch includes: Acquire a plurality of sample emotion images of the k'th batch; wherein the sample emotion images are images generated by forward propagation of the k'th batch; Screening a plurality of the sample emotion images in the k'th batch to obtain at least one augmented sample image according to the emotion accuracy parameter and the semantic clarity parameter of each of the sample emotion images; At least one of the amplified sample images is combined with the sample image subset of the k'+1th batch to obtain the amplified image set of the k'+1th batch.

10. An emotion recognition and emotion image generation device, characterized in that: include: An acquisition module, used to acquire a sample scene query identifier, a sample object query identifier and a sample image set; wherein the sample scene query identifier is used to indicate the scene features of all the sample images, the sample object query identifier is used to indicate the object features of all the sample images, and the sample image set includes a plurality of sample images and an emotion label corresponding to each of the sample images; A training module, used for performing model training according to the sample scene query identifier, the sample object query identifier and the sample image set to obtain a target scene query identifier, a target object query identifier and a joint processing model; wherein the target scene query identifier is used to guide the joint processing model to identify the scene features of the image to be tested, and the target object query identifier is used to guide the joint processing model to identify the object features of the image to be tested; A dual-task processing module is used to perform emotion recognition and emotion image generation on the image to be tested according to the target scene query identifier, the target object query identifier and the joint processing model, so as to obtain the emotion type of the image to be tested and a plurality of emotion images associated with the emotion type of the image to be tested.

Images