Emotion recognition methods, devices, equipment, media and products

Abstract

This invention discloses an emotion recognition method, apparatus, device, medium, and product, belonging to the field of computer vision technology. The method includes: acquiring a first query sample, which includes a first query image; generating a first emotion description text for the first query image using a multimodal large language model; calculating the similarity between the first query image, the first emotion description text, and an image of the i-th sample in a query database to obtain a hybrid retrieval score for the first query image on the i-th sample; selecting an initial candidate set based on the hybrid retrieval scores of each sample in the query database; performing diversity optimization on the initial candidate set to obtain a context example set; and generating an emotion category prediction result for the first query sample using a large-scale vision-language model based on the context example set. This method can effectively improve the accuracy of emotion recognition.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an emotion recognition method, apparatus, device, medium, and product. Background Technology

[0002] Visual Emotion Recognition (VER) aims to infer human emotional states by analyzing visual cues in images, such as facial expressions, body movements, and scene atmosphere.

[0003] Currently, most mainstream visual emotion recognition methods are based on deep supervised learning. These methods typically rely on large-scale, manually annotated, high-quality datasets (such as EmoSet and ArtPhoto) to train convolutional neural networks (CNNs) or visual Transformers. However, constructing emotion datasets faces significant challenges: on the one hand, emotion annotation is highly subjective and ambiguous, with disagreements often existing between annotators, leading to extremely high annotation costs and noise; on the other hand, models trained on specific datasets often struggle to generalize to new domains or scenarios. Therefore, data scarcity and poor generalization ability have become bottlenecks restricting the development of traditional emotion recognition technologies.

[0004] With the development of artificial intelligence, large-scale vision-language models (VLMs) have demonstrated powerful multimodal understanding capabilities. In-Context Learning (ICL), as an emerging paradigm, allows models to adapt to new tasks using only a small number of examples (i.e., "context") contained in input prompts without updating parameters. This few-shot or even zero-shot learning approach is well-suited to the data-scarce reality of emotion recognition.

[0005] However, existing contextual example retrieval methods are mostly based on single-modal similarity, making it difficult to achieve accurate sentiment alignment. For example, retrieval based solely on visual features (such as CLIP visual encoding) can easily lead to "visual-semantic mismatch," where the retrieved image may be similar to the query image in composition or color, but express completely different emotions (e.g., a tranquil landscape image might be incorrectly matched with a landscape image conveying loneliness). This visual similarity cannot effectively represent sentiment relevance, thus misleading model inference. Therefore, contextual example retrieval methods in related technologies suffer from insufficient accuracy when applied to the field of sentiment recognition. Summary of the Invention

[0006] This invention provides an emotion recognition method, apparatus, device, medium, and product, which can effectively improve the accuracy of emotion recognition. The technical solution includes at least the following: In a first aspect, an emotion recognition method is provided, comprising: acquiring a first query sample, the first query sample including a first query image; generating a first emotion description text for the first query image using a multimodal large language model; calculating the similarity between the first query image, the first emotion description text, and an image of an i-th sample in a query database to obtain a hybrid retrieval score of the first query image on the i-th sample, the query database including multiple samples, each sample including an image; selecting an initial candidate set based on the hybrid retrieval scores of each sample in the query database; performing diversity optimization on the initial candidate set to obtain a context example set; and generating an emotion category prediction result for the first query sample using a large visual-language model based on the context example set.

[0007] Optionally, each sample in the query database further includes a label indicating the sentiment category to which the sample belongs. The diversity optimization of the initial candidate set to obtain the context example set includes: counting the occurrence frequency of each sentiment category in the initial candidate set; removing sentiment categories with an occurrence frequency less than a noise threshold from the initial candidate set; retaining only the sentiment category with an occurrence frequency greater than the dominant threshold in the initial candidate set if there exists a sentiment category with an occurrence frequency greater than the dominant threshold; sampling proportionally from each sentiment category in the initial candidate set to construct an intermediate candidate pool; and filtering from the intermediate candidate pool using the maximum boundary correlation algorithm to obtain the context example set.

[0008] Optionally, the objective function of the maximum boundary correlation algorithm is expressed by the following formula:

[0009] in, For the i-th sample in the intermediate candidate pool The objective function value of the maximum boundary correlation algorithm. This is the intermediate candidate pool. For the first query image, The set of samples that have been selected and belong to the context example set. for The j-th sample in the dataset, This is the preset first balancing weight.

[0010] Optionally, calculating the similarity between the first query image, the first sentiment description text, and the image of the i-th sample in the query database to obtain a mixed retrieval score for the first query image on the i-th sample includes: calculating a first similarity between the first query image and the image of the i-th sample; calculating a second similarity between the first sentiment description text and the image of the i-th sample; and obtaining a mixed retrieval score for the first query image on the i-th sample based on the first similarity and the second similarity.

[0011] Optionally, generating the sentiment category prediction result of the first query sample using a large visual-language model based on the context example set includes: concatenating the first query sample, the sentiment category prediction instruction, the candidate sentiment category set, and the context example set and inputting them into the large visual-language model to obtain the sentiment category prediction result of the first query image generated by the large visual-language model. The sentiment category prediction instruction is used to instruct the large visual-language model to generate the sentiment category prediction result of the first query sample based on the input data.

[0012] Secondly, an emotion recognition device is also provided, comprising: an acquisition module for acquiring a first query sample, the first query sample including a first query image; an emotion description generation module for generating a first emotion description text for the first query image using a multimodal large language model; a hybrid retrieval score calculation module for calculating the similarity between the first query image, the first emotion description text, and an image of the i-th sample in a query database, to obtain a hybrid retrieval score for the first query image on the i-th sample, the query database including multiple samples, each sample including an image; a filtering module for filtering an initial candidate set based on the hybrid retrieval scores of each sample in the query database; an optimization module for performing diversity optimization on the initial candidate set to obtain a context example set; and an emotion category prediction module for generating an emotion category prediction result for the first query sample using a large visual-language model based on the context example set.

[0013] Optionally, each sample in the query database also includes a label, which indicates the sentiment category to which the sample belongs. The optimization module is further used to count the occurrence frequency of each sentiment category in the initial candidate set, remove sentiment categories with an occurrence frequency less than a noise threshold from the initial candidate set, and retain only the sentiment category with an occurrence frequency greater than the dominant threshold in the initial candidate set if there is a sentiment category with an occurrence frequency greater than the dominant threshold. The module also samples each sentiment category in the initial candidate set proportionally to construct an intermediate candidate pool, and uses the maximum boundary correlation algorithm to filter from the intermediate candidate pool to obtain the context example set.

[0014] Optionally, in the optimization module, the objective function of the maximum boundary correlation algorithm is expressed by the following formula:

[0015] in, For the i-th sample in the intermediate candidate pool The objective function value of the maximum boundary correlation algorithm. This is the intermediate candidate pool. For the first query image, The set of samples that have been selected and belong to the context example set. for The j-th sample in the dataset, This is the preset first balancing weight.

[0016] Optionally, the hybrid retrieval scoring module is further configured to calculate a first similarity between the first query image and the image of the i-th sample; calculate a second similarity between the first sentiment description text and the image of the i-th sample; and obtain a hybrid retrieval score for the first query image on the i-th sample based on the first similarity and the second similarity.

[0017] Optionally, the sentiment category prediction module is further configured to concatenate the first query sample, the sentiment category prediction instruction, the candidate sentiment category set, and the context example set and input them into the large-scale visual-language model to obtain the sentiment category prediction result of the first query image generated by the large-scale visual-language model. The sentiment category prediction instruction is used to instruct the large-scale visual-language model to generate the sentiment category prediction result of the first query sample based on the input data.

[0018] Thirdly, a computer device is also provided, comprising: a memory and a processor, wherein the memory stores at least one computer program, the at least one computer program being loaded and executed by the processor to perform the emotion recognition method described in the above embodiments.

[0019] Fourthly, a computer-readable storage medium is also provided, wherein at least one computer program is stored in the computer-readable storage medium, the at least one computer program being loaded and executed by a processor to perform the emotion recognition method described in the above embodiments.

[0020] Fifthly, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the method described in the first aspect.

[0021] The beneficial effects of the technical solution provided by this invention include at least the following: In this embodiment, by employing a context learning paradigm, high-performance emotion recognition can be achieved without the need for expensive full-parameter fine-tuning or training of large-scale vision-language models. This significantly reduces computational resource consumption and dependence on large-scale labeled data. By using a multimodal large language model to generate a first emotion description text for the first query image, this first emotion description text can serve as a proxy for the emotional semantics of the first query sample during retrieval. This overcomes the limitation of relying solely on visual features in capturing abstract emotions (such as the visual similarity between loneliness and tranquility), significantly improving the emotional relevance of the retrieved samples, resolving the "visual-semantic misalignment" problem, and enhancing the accuracy of emotion recognition. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this embodiment, the accompanying drawings used in the description of the embodiment will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart of an exemplary embodiment of the emotion recognition method provided by the present invention is shown; Figure 2 A schematic diagram of the structure of an emotion recognition device provided in an exemplary embodiment of the present invention is shown; Figure 3 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of the present invention. Detailed Implementation

[0024] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains. The terms “first,” “second,” “third,” and similar terms used in this patent application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an” or “a” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “comprising” or “including” and similar terms mean that the elements or objects preceding “comprising” or “including” encompass the elements or objects listed following “comprising” or “including” and their equivalents, but do not exclude other elements or objects.

[0025] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0026] Example 1.

[0027] Figure 1 A flowchart illustrating an exemplary embodiment of the emotion recognition method provided by the present invention is shown, the method being executable by a computer device. See also Figure 1 The method includes: In step 101, the first query sample is obtained.

[0028] The first query sample includes the first query image.

[0029] In this embodiment, the first query sample is a sample for which sentiment recognition (sentiment category prediction) needs to be performed. For example, the first query sample can be a query sample input by the user.

[0030] Typically, when users input an image to search for information, they also input some text to describe the question they want to ask. In this case, the first query sample includes both the first query image and the first query text, which is used to describe the question to be asked in the first query image in textual form. In sentiment recognition scenarios, the first query text is usually used to indicate the sentiment category of the first query image.

[0031] For example, the first query sample can be represented as ,in The first image to be queried. This is the first query text.

[0032] In step 102, a first sentiment description text for the first query image is generated using a multimodal large language model.

[0033] In this embodiment, the multimodal large language model is a pre-trained multimodal large model with image and text understanding capabilities. The input to this multimodal large language model is a first prompt word and a first query image. The output is the first sentiment description text. Here, the sentiment description text is used to transform the visual information of the first query image into an explicit text description, and the first cue word is used to guide the multimodal large language model to output the first sentiment description text of the first query image.

[0034] For example, the first prompt is: Analyze this image. Describe the core emotional content and atmosphere, focusing on facial expressions, body language, and environmental context. Output a concise description of the emotional state.

[0035] In step 103, the similarity between the first query image, the first sentiment description text, and the image of the i-th sample in the query database is calculated to obtain the hybrid retrieval score of the first query image on the i-th sample.

[0036] In this embodiment, the query database is a pre-built database. The query database includes multiple samples, each sample being an "image-question-answer" triple, meaning that any sample in the query database contains an image, a question, and an answer. Here, within the same sample, the question is a sentiment query text about the image (e.g., "What emotion does this image convey?"), and the answer is the image's label, indicating the sentiment category to which the image in that sample belongs.

[0037] For example, querying a database can be represented as ,in It queries the i-th sample in the database. It queries the image of the i-th sample in the database. It is aimed at Emotional inquiry text, yes The tag. To query the total number of samples in the database.

[0038] In the process of calculating similarity, only the images of each sample in the database need to be queried.

[0039] Before implementing step 103, a visual encoder (such as the Vision Encoder of the CLIP model) is needed to convert the first query image and the images of each sample in the query database into high-dimensional visual feature vectors, and a text encoder (such as the Text Encoder of the CLIP model) is needed to convert the first sentiment description text into a high-dimensional semantic feature vector. Furthermore, to eliminate the influence of vector magnitude on similarity calculation, all the above feature vectors need to be normalized (e.g., L2 normalization), so that the subsequent dot product operation can be directly equivalent to cosine similarity calculation.

[0040] In this case, step 103 may optionally include steps 1031 to 1033 as follows.

[0041] Step 1031: Calculate the first similarity between the first query image and the image of the i-th sample.

[0042] Calculating the first similarity essentially involves calculating the similarity (e.g., cosine similarity) between the visual feature vector of the first query image and the visual feature vector of the i-th sample image. This process can be represented by formula (1).

[0043] (1) In formula (1), For the first query image Image of the i-th sample The first similarity between them, which belongs to a visual similarity. This is the visual feature vector of the first query image. Let be the visual feature vector of the image of the i-th sample.

[0044] The first similarity reflects the degree of similarity between the first query image and the image of the i-th sample in terms of physical appearance, such as composition and color.

[0045] Step 1032: Calculate the second similarity between the first sentiment description text and the image of the i-th sample.

[0046] Calculating the second similarity essentially involves calculating the similarity (e.g., cosine similarity) between the semantic feature vector of the first sentiment description text and the visual feature vector of the image of the i-th sample. This process can be represented by formula (2).

[0047] (2) In formula (2), For the first emotional description text Image of the i-th sample The second similarity between them belongs to a cross-modal semantic similarity. This is the semantic feature vector of the first sentiment description text. Let be the visual feature vector of the image of the i-th sample.

[0048] The second similarity reflects whether the visual representation of the sample image matches the underlying sentiment description of the query image.

[0049] Step 1033: Based on the first similarity and the second similarity, obtain the mixed retrieval score of the first query image on the i-th sample.

[0050] After obtaining the first similarity and the second similarity, formula (3) is used to obtain the mixed retrieval score of the first query image on the i-th sample.

[0051] (3) In formula (3), The mixed retrieval score for the first query image on the i-th sample. The preset second balancing weight, The value range is [0.3, 0.9], for example, 0.3, 0.7 or 0.9. The meanings of the other parameters in formula (3) are the same as those in formula (1) and formula (2), and are omitted here.

[0052] In step 104, an initial candidate set is selected based on the mixed search scores of each sample in the query database.

[0053] By performing step 103 on each sample in the query database, a mixed retrieval score for each sample in the query database can be obtained. The mixed retrieval scores of each sample are sorted from largest to smallest, and the top K samples in the sorting are selected as the initial sample set. K is a positive integer. For example, K is 100.

[0054] In step 105, diversity optimization is performed on the initial candidate set to obtain the context example set.

[0055] Optionally, step 105 includes steps 1051 to 1053 as follows.

[0056] In step 1051, the frequency of occurrence of each emotion category in the initial candidate set is counted, and emotion categories with a frequency less than the noise threshold are removed from the initial candidate set. If there is an emotion category with a frequency greater than the dominant threshold, only that emotion category with a frequency greater than the dominant threshold is retained in the initial candidate set.

[0057] Here, the initial candidate set contains multiple sentiment categories, and each sentiment category contains multiple samples. Suppose there are K sentiment categories in the initial candidate set, and the a-th sentiment category has b samples, then the frequency of the a-th sentiment category is b / K.

[0058] Both the noise threshold and the dominance threshold are preset thresholds. If the frequency of occurrence of a certain emotion category is less than the noise threshold, it means that multiple samples of that emotion category are noise samples and can be removed from the initial candidate set. If the frequency of occurrence of a certain emotion category is greater than the dominance threshold, it means that the samples of that emotion category have an overwhelming advantage (dominance) in the initial candidate set. In this case, only the samples of that emotion category can be retained, and samples of other categories can be removed.

[0059] Step 1051 achieves the correction of the category distribution of the initial candidate set.

[0060] In step 1052, an intermediate candidate pool is constructed by sampling proportionally from each sentiment category in the initial candidate set.

[0061] For example, if the size of the intermediate candidate pool is m, then the number of samples required to sample the c-th sentiment category in the initial candidate set is . ,in, This represents the number of samples required to sample the c-th sentiment category in the initial candidate set. The size of the intermediate candidate pool. Let be the number of samples in the c-th sentiment category of the initial candidate set. It is a sentiment category in the initial candidate set. For the first candidate in the initial candidate set Number of samples for each emotion category.

[0062] After performing step 1051, the total number of samples in the initial candidate set after category distribution correction may be less than K (some sentiment categories may be removed). When performing step 1052, it is essentially a proportional sampling of the initial candidate set after category distribution correction, so only the sentiment categories present in the initial candidate set after category distribution correction need to be considered.

[0063] If the required number of samples for each sentiment category in the initial candidate set after category-specific correction is known, sampling can be performed from the initial candidate set based on this number. In this embodiment, sampling is performed according to the magnitude of the mixed retrieval score. For example, for multiple samples of the c-th sentiment category, sampling is required from... In the case of n samples being sent to the intermediate candidate pool, the top samples of the c-th sentiment category are retrieved by pooling multiple samples and ranking them from largest to smallest. Each sample is a sample taken from the c-th sentiment category.

[0064] This is equivalent to performing a priori setting of the class distribution for the intermediate candidate pool, making the class distribution of samples in the intermediate candidate pool more balanced.

[0065] In step 1053, the maximum boundary correlation algorithm is used to filter from the intermediate candidate pool to obtain the context example set.

[0066] Here, the Maximum Marginal Relevance (MMR) algorithm is a greedy iterative algorithm that selects the sample with the largest objective function value as a sample in the context example set during each iteration. After multiple iterations using the MMR algorithm, the final context example set can be obtained.

[0067] Alternatively, the objective function of the maximum boundary correlation algorithm can be expressed by the following formula (4): (4) In formula (4), For the i-th sample in the intermediate candidate pool The objective function value of the maximum boundary correlation algorithm. As an intermediate candidate pool, The first image to be queried. The set of samples that have been selected and belong to the context example set. for The j-th sample in the dataset, This is a preset first balancing weight. For example, The value range is [0.2, 0.8], for example, 0.2, 0.6 or 0.8.

[0068] In this objective function, To improve relevance gain, ensure that the selected samples are highly similar to the first query image. This is a redundancy penalty term that ensures visual differences between the selected samples and the previously screened samples. This objective function balances relevance and redundancy to guarantee the reliability of the final selected set of context examples.

[0069] Existing retrieval strategies often lack optimization for the overall quality of the contextual example set. Simple Top-K retrieval often results in an overabundance of one sentiment type in the example set (class imbalance) or highly repetitive visual representations of multiple retrieved samples (lack of diversity). This redundant and biased contextual information not only wastes the limited input window of the VLM but also limits the model's ability to capture rich sentiment features through analogical learning.

[0070] In this embodiment, steps 104 to 105 effectively eliminate noisy samples with conflicting labels and force the context examples to exhibit visual diversity. This avoids overfitting due to too many examples or incorrect inference due to class confusion, thus achieving superior recognition accuracy and stability compared to existing methods in a low-sample setting.

[0071] In step 106, based on the context example set, a large vision-language model is used to generate a sentiment category prediction result for the first query image.

[0072] Optionally, step 106 includes: concatenating the first query sample, the sentiment category prediction instruction, the candidate sentiment category set, and the context example set and inputting them into a large-scale vision-language model to obtain the sentiment category prediction result of the first query image generated by the large-scale vision-language model.

[0073] The sentiment category prediction instruction is used to instruct a large vision-language model to generate a sentiment category prediction result for the first query image based on the input data.

[0074] Here, the sentiment category prediction instruction is a type of cue word used to guide a large vision-language model to generate a sentiment category prediction result for the first query image based on the input data.

[0075] For example, the emotion category prediction instruction is: Please identify the emotion category of the current image based on the following example.

[0076] The candidate sentiment category set is a collection of all available sentiment categories. This large vision-language model selects one sentiment category from the candidate sentiment category set as the sentiment category prediction result for the first query sample.

[0077] After receiving the input data, the large-scale visual-language model (VLM) performs analogical reasoning based on the patterns presented in the context example set to generate the sentiment category of the first query sample, which is the sentiment category prediction result of the first query sample. This sentiment category prediction result is the sentiment recognition conclusion for the first query sample.

[0078] Experimental results show that the emotion recognition method in this embodiment outperforms existing emotion recognition methods in terms of accuracy on multiple benchmark datasets such as ArtPhoto, EmotionROI, and EmoSet. It is also applicable to various visual-language models with different architectures (such as LLaVA, Qwen-VL, Otter, etc.), demonstrating strong cross-domain adaptability.

[0079] In this embodiment, by employing a context learning paradigm, high-performance emotion recognition can be achieved without the need for expensive full-parameter fine-tuning or training of large-scale vision-language models. This significantly reduces computational resource consumption and dependence on large-scale labeled data. By using a multimodal large language model to generate a first emotion description text for the first query image, this first emotion description text can serve as a proxy for the emotional semantics of the first query sample during retrieval. This overcomes the limitation of relying solely on visual features in capturing abstract emotions (such as the visual similarity between loneliness and tranquility), significantly improving the emotional relevance of the retrieved samples, resolving the "visual-semantic misalignment" problem, and enhancing the accuracy of emotion recognition.

[0080] The following are device embodiments of this application. For details not described in detail in the device embodiments, please refer to the above method embodiments.

[0081] Example 2.

[0082] Figure 2 A schematic diagram of the structure of an emotion recognition device provided in an exemplary embodiment of the present invention is shown. The emotion recognition device 200 includes: an acquisition module 201, an emotion description generation module 202, a hybrid retrieval scoring calculation module 203, a filtering module 204, an optimization module 205, and an emotion category prediction module 206.

[0083] The acquisition module 201 is used to acquire a first query sample, which includes a first query image; The sentiment description generation module 202 is used to generate the first sentiment description text of the first query image using a multimodal large language model; The hybrid retrieval score calculation module 203 is used to calculate the similarity between the first query image, the first sentiment description text and the image of the i-th sample in the query database, and to obtain the hybrid retrieval score of the first query image on the i-th sample. The query database includes multiple samples, and each sample includes one image. The filtering module 204 is used to filter out an initial candidate set based on the mixed search scores of each sample in the query database; Optimization module 205 is used to perform diversity optimization on the initial candidate set to obtain a context example set; The sentiment category prediction module 206 is used to generate a sentiment category prediction result for the first query sample based on a large visual-language model using a set of contextual examples.

[0084] Optionally, each sample in the query database also includes a label, which indicates the sentiment category to which the sample belongs. The optimization module 205 is also used to count the occurrence frequency of each sentiment category in the initial candidate set, remove sentiment categories whose occurrence frequency is less than the noise threshold from the initial candidate set, and retain only the sentiment category whose occurrence frequency is greater than the dominant threshold in the initial candidate set if there is a sentiment category whose occurrence frequency is greater than the dominant threshold. The module also samples each sentiment category in the initial candidate set proportionally to construct an intermediate candidate pool, and uses the maximum boundary correlation algorithm to filter from the intermediate candidate pool to obtain the context example set.

[0085] Optionally, in optimization module 205, the objective function of the maximum boundary correlation algorithm is expressed by the following formula:

[0086] in, For the i-th sample in the intermediate candidate pool The objective function value of the maximum boundary correlation algorithm. As an intermediate candidate pool, The first image to be queried. The set of samples that have been selected and belong to the context example set. for The j-th sample in the dataset, This is the preset first balancing weight.

[0087] Optionally, the hybrid retrieval score calculation module 203 is further configured to calculate a first similarity between the first query image and the image of the i-th sample; calculate a second similarity between the first sentiment description text and the image of the i-th sample; and obtain a hybrid retrieval score for the first query image on the i-th sample based on the first similarity and the second similarity.

[0088] Optionally, the sentiment category prediction module 206 is also used to concatenate the first query sample, the sentiment category prediction instruction, the candidate sentiment category set, and the context example set and input them into the large-scale vision-language model to obtain the sentiment category prediction result of the first query image generated by the large-scale vision-language model. The sentiment category prediction instruction is used to instruct the large-scale vision-language model to generate the sentiment category prediction result of the first query sample based on the input data.

[0089] It should be noted that the emotion recognition device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the emotion recognition device and the emotion recognition method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0090] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods are possible. Furthermore, the functional modules in each embodiment of the invention can be integrated into a single processor, exist as separate physical entities, or consist of two or more modules integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0091] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal device (which may be a personal computer, mobile phone, or communication device, etc.) or processor to execute all or part of the steps of the method of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0092] Figure 3 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of the present invention. For example... Figure 3 As shown, the computer device 300 includes a processor 301 and a memory 302.

[0093] Processor 301 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 301 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 301 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 301 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 301 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0094] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 302 are used to store at least one instruction, which is executed by the processor 301 to implement the emotion recognition method provided in the embodiments of the present invention.

[0095] Those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the computer device 300, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0096] This invention also provides a non-transitory computer-readable storage medium, wherein when the instructions in the storage medium are executed by the processor of a computer device, the computer device is able to execute the emotion recognition method provided in this invention.

[0097] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the emotion recognition method provided in this invention.

[0098] The above description is merely an optional embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An emotion recognition method, characterized in that, The method includes: Obtain a first query sample, which includes a first query image; A multimodal large language model is used to generate the first sentiment description text for the first query image; Calculate the similarity between the first query image, the first sentiment description text, and the image of the i-th sample in the query database to obtain the mixed retrieval score of the first query image on the i-th sample. The query database includes multiple samples, and each sample includes one image. An initial candidate set is selected based on the mixed search scores of each sample in the query database; The initial candidate set is then optimized for diversity to obtain a context example set; Based on the aforementioned contextual example set, a large-scale visual-language model is used to generate a sentiment category prediction result for the first query sample.

2. The method according to claim 1, characterized in that, Each sample in the query database also includes a label indicating the sentiment category to which the sample belongs. The initial candidate set is then subjected to diversity optimization to obtain a contextual example set, which includes: The frequency of occurrence of each emotion category in the initial candidate set is counted. Emotion categories with a frequency lower than the noise threshold are removed from the initial candidate set. If the frequency of occurrence of an emotion category is higher than the dominant threshold, only that emotion category with a frequency higher than the dominant threshold is retained in the initial candidate set. An intermediate candidate pool is constructed by sampling proportionally from each sentiment category in the initial candidate set; The maximum boundary correlation algorithm is used to filter from the intermediate candidate pool to obtain the context example set.

3. The method according to claim 2, characterized in that, The objective function of the maximum boundary correlation algorithm is expressed by the following formula: in, For the i-th sample in the intermediate candidate pool The objective function value of the maximum boundary correlation algorithm. This is the intermediate candidate pool. For the first query image, The set of samples that have been selected and belong to the context example set. for The j-th sample in the dataset, This is the preset first balancing weight.

4. The method according to any one of claims 1 to 3, characterized in that, The calculation of the similarity between the first query image, the first sentiment description text, and the image of the i-th sample in the query database to obtain the mixed retrieval score of the first query image on the i-th sample includes: Calculate the first similarity between the first query image and the image of the i-th sample; Calculate the second similarity between the first sentiment description text and the image of the i-th sample; Based on the first similarity and the second similarity, obtain the mixed retrieval score of the first query image on the i-th sample.

5. The method according to any one of claims 1 to 3, characterized in that, The step of generating the sentiment category prediction result of the first query sample using a large visual-language model based on the context example set includes: The first query sample, the sentiment category prediction instruction, the candidate sentiment category set, and the context example set are concatenated and input into the large-scale visual-language model to obtain the sentiment category prediction result of the first query image generated by the large-scale visual-language model. The sentiment category prediction instruction is used to instruct the large-scale visual-language model to generate the sentiment category prediction result of the first query sample based on the input data.

6. An emotion recognition device, characterized in that, The device includes: The acquisition module is used to acquire a first query sample, wherein the first query sample includes a first query image; The sentiment description generation module is used to generate the first sentiment description text of the first query image using a multimodal large language model; The hybrid retrieval score calculation module is used to calculate the similarity between the first query image, the first sentiment description text, and the image of the i-th sample in the query database, and to obtain the hybrid retrieval score of the first query image on the i-th sample. The query database includes multiple samples, and each sample includes an image. The filtering module is used to filter out an initial candidate set based on the mixed retrieval scores of each sample in the query database; The optimization module is used to perform diversity optimization on the initial candidate set to obtain a context example set; The sentiment category prediction module is used to generate a sentiment category prediction result for the first query sample based on the context example set and a large vision-language model.

7. The apparatus according to claim 6, characterized in that, Each sample in the query database also includes a label, which is used to indicate the sentiment category to which the sample belongs. The optimization module is also used to count the occurrence frequency of each sentiment category in the initial candidate set, remove sentiment categories whose occurrence frequency is less than the noise threshold from the initial candidate set, and retain only the sentiment category whose occurrence frequency is greater than the dominant threshold in the initial candidate set if there is a sentiment category whose occurrence frequency is greater than the dominant threshold. An intermediate candidate pool is constructed by sampling proportionally from each sentiment category in the initial candidate set; The maximum boundary correlation algorithm is used to filter from the intermediate candidate pool to obtain the context example set.

8. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores at least one computer program, which is loaded and executed by the processor to implement the method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the method of any one of claims 1 to 5.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1 to 5.

Images