Large model dialogue method and device based on multi-modal emotion, equipment and medium

Abstract

The invention relates to a large model dialogue method and device based on multi-modal emotion, equipment and a medium, and particularly relates to the technical field of natural language processing. Comprising the steps of receiving multi-modal input of a user, and performing multi-modal emotion analysis on the multi-modal input to obtain multi-modal emotion representation; inputting the multi-modal emotion representation and the current dialogue context into an emotion generation network to obtain a current emotion tendency output by the emotion generation network; updating the historical emotional tendency based on the current emotional tendency to obtain a target emotional tendency; and inputting the target emotional tendency and the current dialogue context into a decoder to obtain an emotional reply content output by the decoder. According to the embodiment of the invention, the accuracy of emotion understanding is improved, the method is suitable for user emotion fluctuation, and coherent and consistent emotion reply content can be generated.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a large-model dialogue method, device, equipment and medium based on multimodal emotions. Background Art

[0002] Emotion understanding and generation is one of the core links of intelligent dialogue systems. Its goal is to accurately identify the user's emotional state by analyzing the user input and generate a natural language response that conforms to the emotional context.

[0003] In terms of emotion understanding, in order to improve the accuracy of emotion recognition, some studies have introduced multimodal information such as speech and images for emotion analysis. For example, the emotion recognition of text is assisted by the tone, speed, and volume changes in speech, or the facial expressions and body movements of users in images. This multimodal fusion method significantly enhances the ability of emotion recognition and provides richer input information for the emotion generation of dialogue systems.

[0004] In terms of emotion generation, existing technologies mainly rely on large language models to generate text, and regulate the emotional tone and content of the generated text by incorporating emotional features or emotional labels into the input. For example, conditional generative networks or emotion-enhanced pre-training methods are used to fine-tune large language models so that they can generate natural language responses that meet specific emotional features.

[0005] However, the above-mentioned multimodal fusion emotion understanding method and the emotion generation method based on large language models all adopt static generation strategies and cannot adapt in real time according to the changes in user emotions. In addition, the existing dialogue system has limited ability to maintain emotional memory and plan emotional paths, and cannot meet the needs of emotional consistency and guidance in multi-round dialogue scenarios. Summary of the invention

[0006] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a large model dialogue method, device, equipment and medium based on multimodal emotions. The technical solutions provided by the embodiments of the present application are as follows:

[0007] In a first aspect, the present application provides a large-model dialogue method based on multimodal emotion, the method comprising: receiving a user's multimodal input, and performing multimodal emotion analysis on the multimodal input to obtain a multimodal emotion representation; inputting the multimodal emotion representation and the current dialogue context into an emotion generation network to obtain a current emotion tendency output by the emotion generation network; updating the historical emotion tendency based on the current emotion tendency to obtain a target emotion tendency; inputting the target emotion tendency and the current dialogue context into a decoder to obtain an emotion response content output by the decoder.

[0008] As an optional implementation in an embodiment of the present application, multimodal input includes: user text, user voice and user image; multimodal sentiment analysis is performed on the multimodal input to obtain a multimodal sentiment representation, including: inputting the user text into a pre-trained large language model to obtain text sentiment information output by the pre-trained large language model; inputting the user voice into a pre-trained speech recognition model to obtain speech sentiment information output by the pre-trained speech recognition model; inputting the user image into a pre-trained image recognition model to obtain image sentiment information output by the pre-trained image recognition model; and fusing the text sentiment information, speech sentiment information and image sentiment information to obtain a multimodal sentiment representation.

[0009] As an optional implementation in the embodiment of the present application, the historical emotional tendency is updated based on the current emotional tendency to obtain the target emotional tendency, including: obtaining the historical emotional tendency from the emotional memory bank; wherein the emotional memory bank is used to store the user's emotional tendency in each round of conversation; based on the current emotional tendency and its importance weight coefficient, updating the historical emotional tendency to obtain the target emotional tendency.

[0010] As an optional implementation in the embodiment of the present application, after obtaining the historical emotional tendency from the emotional memory library, the method also includes: based on the current emotional tendency and its time decay factor, updating the historical emotional tendency to obtain the target emotional tendency.

[0011] As an optional implementation in the embodiment of the present application, updating the historical emotional tendency to obtain the target emotional tendency includes: calculating the emotional deviation value between the updated historical emotional tendency and the preset emotional tendency; when the emotional deviation value is greater than the preset threshold, adjusting the updated historical emotional tendency until the emotional deviation value is less than or equal to the preset threshold, thereby obtaining the target emotional tendency.

[0012] In a second aspect, the present application provides a large model dialogue device based on multimodal emotions, the device comprising:

[0013] The sentiment analysis module is used to receive multimodal input from the user and perform multimodal sentiment analysis on the multimodal input to obtain multimodal sentiment representation;

[0014] The emotion generation module is used to input the multimodal emotion representation and the current conversation context into the emotion generation network to obtain the current emotion tendency output by the emotion generation network;

[0015] The emotion updating module is used to update the historical emotion tendency based on the current emotion tendency to obtain the target emotion tendency;

[0016] The reply generation module is used to input the target emotional tendency and the current conversation context into the decoder to obtain the emotional reply content output by the decoder.

[0017] As an optional implementation in the embodiment of the present application, multimodal input includes: user text, user voice and user image; the sentiment analysis module includes: a text sentiment analysis submodule, which is used to input user text into a pre-trained large language model to obtain text sentiment information output by the pre-trained large language model; a speech sentiment analysis submodule, which is used to input user voice into a pre-trained speech recognition model to obtain speech sentiment information output by the pre-trained speech recognition model; an image sentiment analysis submodule, which is used to input user images into a pre-trained image recognition model to obtain image sentiment information output by the pre-trained image recognition model; and a sentiment fusion submodule, which is used to fuse text sentiment information, speech sentiment information and image sentiment information to obtain multimodal sentiment representation.

[0018] As an optional implementation in the embodiment of the present application, the emotion update module includes: an acquisition submodule, used to acquire historical emotion tendencies from an emotion memory bank; wherein the emotion memory bank is used to store the user's emotion tendencies in each round of conversation; a first update submodule, used to update the historical emotion tendencies based on the current emotion tendencies and their importance weight coefficients to obtain target emotion tendencies.

[0019] As an optional implementation in the embodiment of the present application, the emotion updating module further includes: a second updating submodule, which is used to update the historical emotion tendency to obtain the target emotion tendency based on the current emotion tendency and its time decay factor.

[0020] As an optional implementation in the embodiment of the present application, the first update submodule or the second update submodule is specifically used to: calculate the emotional deviation value between the updated historical emotional tendency and the preset emotional tendency; when the emotional deviation value is greater than the preset threshold, adjust the updated historical emotional tendency until the emotional deviation value is less than or equal to the preset threshold, and obtain the target emotional tendency.

[0021] In a third aspect, the present application provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the large model dialogue method based on multimodal emotions is implemented as described in the first aspect or any one of its optional embodiments.

[0022] In a fourth aspect, the present application provides a computer-readable storage medium, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the large model dialogue method based on multimodal emotions as described in the first aspect or any one of its optional embodiments.

[0023] In a fifth aspect, the present application provides a computer program product, comprising: the computer program product comprises a computer program, and when the computer program runs on a computer, the computer implements the large model dialogue method based on multimodal emotions as described in the first aspect or any one of its optional embodiments.

[0024] Compared with the prior art, the technical solution provided by the embodiments of the present application has the following advantages:

[0025] The disclosed embodiments provide a large-model dialogue method, apparatus, device and medium based on multimodal emotion, wherein the method first receives the multimodal input of the user, and performs multimodal emotion analysis on the multimodal input to obtain a multimodal emotion representation; then, the multimodal emotion representation and the current context are input into the emotion generation network to obtain the current emotion tendency output by the emotion generation network; then, based on the current emotion tendency, the historical emotion tendency is updated to obtain the target emotion tendency, and the target emotion tendency and the current dialogue context are further input into the decoder to obtain the emotion response content output by the decoder. In this way, the present application deeply understands the emotion corresponding to the user input through multimodal fusion to obtain the current emotion tendency, thereby improving the accuracy of emotion understanding, and then adapts to the user's emotional fluctuations, updates the historical emotions in multiple rounds of dialogue scenarios, so as to generate coherent and consistent emotion response content, thereby improving the naturalness and fluency of human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0028] Figure 1 A flowchart of a large model dialogue method based on multimodal emotions according to an embodiment of the present application;

[0029] Figure 2 A schematic diagram of the structure of a large model dialogue device based on multimodal emotions according to an embodiment of the present application;

[0030] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] Although the existing emotion understanding and generation technology based on large language models has improved the accuracy of emotion recognition and the quality of generated text to a certain extent, it still faces the following problems in practical applications:

[0032] (1) Single-modality limitation of emotion understanding: Existing emotion analysis methods mainly rely on text input for emotion recognition. Although they can recognize certain emotional features, they cannot fully capture the user's emotional signals. The importance of non-verbal signals such as intonation, speech speed, and facial expressions in speech in emotion understanding has not been fully utilized. Single-modality emotion recognition limits the system's ability to accurately grasp the user's complex emotions, resulting in insufficient accuracy and adaptability of emotional feedback.

[0033] (2) Static and single emotion generation: Current emotion generation is mostly based on fixed emotion labels or strategies, lacking the ability to dynamically adjust to user emotion changes. This static generation strategy makes it difficult for the system to respond in real time to the user's emotional fluctuations during the conversation, resulting in the generated content being unable to keep pace with the user's emotional state, affecting the naturalness and fluency of the conversation.

[0034] (3) Lack of emotional coherence and memory: In multi-round conversations, emotional coherence is crucial to improving user experience. However, existing technologies have weak memory and management capabilities for emotional information, and the system cannot effectively retain and utilize emotional information in the conversation history. This makes it difficult to track emotional changes in multi-round conversations, resulting in defects in the system's emotional response in terms of coherence and context adaptation.

[0035] (4) Insufficient emotional guidance and intervention: Existing emotion generation methods mainly focus on generating responses based on the user's current emotional state, and lack the ability to guide and intervene in emotions. In some application scenarios, such as mental health counseling or emotion regulation, the system cannot actively guide the user's emotional transformation, limiting its application potential in emotional support.

[0036] In order to solve all or part of the above technical problems, an embodiment of the present application provides a large model dialogue method based on multimodal emotions, which first receives the multimodal input of the user, and performs multimodal emotion analysis on the multimodal input to obtain a multimodal emotion representation; then, the multimodal emotion representation and the current context are input into the emotion generation network to obtain the current emotion tendency output by the emotion generation network; then, based on the current emotion tendency, the historical emotion tendency is updated to obtain the target emotion tendency, and the target emotion tendency and the current dialogue context are further input into the decoder to obtain the emotion response content output by the decoder. In this way, the present application deeply understands the emotion corresponding to the user input through multimodal fusion to obtain the current emotion tendency, thereby improving the accuracy of emotion understanding, and then adapts to the user's emotional fluctuations, updates the historical emotions in multiple rounds of dialogue scenarios, so as to generate coherent and consistent emotion response content, thereby improving the naturalness and fluency of human-computer interaction.

[0037] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0038] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0039] A large model dialogue method based on multimodal emotions provided in the embodiments of the present application can be implemented by electronic devices, including but not limited to personal computers, laptops, tablet computers, smart phones, etc. The operating system of the electronic device may include Android, the mobile operating system (iOS) developed by Apple, the operating system (Windows) developed by Microsoft Corporation of the United States, etc., and the embodiments of the present application are not limited to this. The electronic device can be operated alone to implement the present application, or it can be connected to the network and implement the present application through interactive operations with other computer devices in the network. Among them, the network in which the electronic device is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN) network, etc.

[0040] It should be noted that the protection scope of the large-model dialogue method based on multimodal emotions described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0041] like Figure 1 As shown, Figure 1 This is a flow chart of a multimodal emotional dialogue method according to an embodiment of the present application. The method can be performed based on a multimodal emotional large model dialogue device or electronic device. The method mainly includes the following steps S101 to S104:

[0042] S101, receiving a multimodal input from a user, and performing a multimodal sentiment analysis on the multimodal input to obtain a multimodal sentiment representation.

[0043] Among them, multimodal input includes: user input in other modal forms such as user text, user voice and user image.

[0044] In some embodiments, receiving multimodal input from a user and performing independent sentiment analysis on inputs of different modalities include the following steps S1011 to S1014:

[0045] S1011. Input the user text into the pre-trained large language model to obtain the text sentiment information output by the pre-trained large language model.

[0046] The large language model may be (Bidirectional Encoder Representations from Transformers, BERT), which is used to perform sentiment classification on user text to identify the user's emotional state. Text sentiment information V txt It is used to indicate the emotional state expressed by the user's text. The pre-trained large prediction model is a convergent model trained based on sample text and its corresponding emotional labels.

[0047] S1012. Input the user's voice into the pre-trained voice recognition model to obtain the voice emotion information output by the pre-trained voice recognition model.

[0048] The pre-trained speech recognition model can first extract the speech features of the user's speech, including but not limited to intonation features, speech speed features, and volume features. Then, the speech emotion information is analyzed based on the speech features through a deep learning network (Convolutional Neural Networks, CNN). audio It is used to indicate the emotional state of the user's voice expression. The pre-trained speech recognition model is a convergent model trained based on sample speech and its corresponding emotion labels.

[0049] S1013. Input the user image into the image recognition model to obtain the image emotion information output by the pre-trained image recognition model.

[0050] The pre-trained image recognition model can first extract facial expression information (or body movement information) from the user image, and then analyze the image emotion information V corresponding to the facial expression information (or body movement information) through the image recognition network. image . Image emotional information V image It is used to indicate the emotional state expressed by the user's image. The pre-trained image recognition model is a convergent model trained based on sample images and their corresponding emotional labels.

[0051] S1014, integrating text emotion information, speech emotion information and image emotion information to obtain multimodal emotion representation.

[0052] According to the following formula (1), the text sentiment information V txt , speech emotion information V audio and image sentiment information V image Perform weighted fusion to obtain the multimodal sentiment representation V emotion :

[0053] V emotion =α1·V txt +α2·V audio +α3·V image (1)

[0054] In formula (1), α1, α2, and α3 are fusion weights, which reflect the contribution of each modality’s emotional information to emotional understanding.

[0055] A single modality may produce uncertainty due to factors such as noise and occlusion. The above embodiment obtains a multimodal emotional representation by independently extracting the emotional information of each modality and fusing them so that the emotional information of different modalities complement each other, which is very helpful for accurately judging emotions. By utilizing the powerful text semantic understanding ability of the large language model, in-depth analysis of text emotions is performed, and multimodal emotional representation is generated by fusing the intonation features in the speech and the facial expression features in the image, etc., which effectively improves the depth of emotional understanding and provides high-quality input for emotion generation. Through the fusion of multimodal emotional features, complex and diverse emotional states can be better captured. Multimodal emotional representation can also help track the dynamic evolution of emotions.

[0056] S102: Input the multimodal emotion representation and the current conversation context into the emotion generation network to obtain the current emotion tendency output by the emotion generation network.

[0057] Optionally, according to formula (2), the multimodal sentiment representation V emotion And the current conversation context is input into the emotion generation network f(·), and the current emotion tendency output by the emotion generation network is obtained.

[0058] Z emotion=f(X,V emotion ) (2)

[0059] In formula (2), Z emotion represents the current emotional tendency, X represents the current dialogue context, which is the content information that has been communicated with the user during the current dialogue process; f(·) is a pre-designed emotion generation network, which is used to adjust the emotional tendency of the emotional reply content according to the current dialogue context and multimodal emotion representation.

[0060] The above-mentioned embodiments can accurately grasp the current emotional tendency, so that the machine knows the user's current emotional state, which is conducive to improving the friendliness and smoothness of the conversation.

[0061] In some embodiments, in order to ensure the consistency and coherence of emotions in multiple rounds of dialogue, this application introduces an emotional memory management mechanism. emotion Mark the timestamp t to get Set the current emotional tendency Z emotion Corresponding to the timestamp t, it is stored in the emotional memory library M. The emotional memory library M is used to store the user's emotional tendency in each round of dialogue, which can be expressed as the following formula (3).

[0062]

[0063] The above-mentioned embodiment integrates the user's emotional tendency in multiple rounds of conversations with the conversation context information, and maintains the emotional memory through the emotional memory library.

[0064] S103. Update the historical sentiment tendency based on the current sentiment tendency to obtain the target sentiment tendency.

[0065] In some embodiments, when executing step S103, the historical emotional tendency is first obtained from the emotional memory library; then, based on the current emotional tendency and its importance weight coefficient, the historical emotional tendency is updated to obtain the target emotional tendency. Specifically, according to the following formula (4):

[0066]

[0067] In formula (4), β is the current sentiment tendency Z emotion The importance weight coefficient of . Indicates historical emotional tendencies.

[0068] In some other embodiments, when executing step S103, the historical emotional tendency is first obtained from the emotional memory library; then, based on the current emotional tendency and its time decay factor, the historical emotional tendency is updated to obtain the target emotional tendency. Specifically, according to the following formula (5):

[0069]

[0070] In formula (5), γ t is the time decay factor, which is used to control the decay rate of historical sentiment tendency, γ t =e -λt Sentiment tendencies change over time, and updating historical sentiment tendencies based on the time decay factor can adapt to such changes. The time decay factor can prevent the model from over-reliance on old data, allowing the model to learn more general rules and improve the model's generalization ability.

[0071] The above-mentioned embodiments use the updated historical emotional tendency as the target emotional tendency. In order to achieve positive emotional guidance, the present application introduces an emotional guidance mechanism. On the basis of any of the above-mentioned embodiments, after adjusting the updated historical emotional tendency, specifically, it includes: first calculating the emotional deviation value between the updated historical emotional tendency and the preset emotional tendency; then, when the emotional deviation value is greater than the preset threshold, adjusting the updated historical emotional tendency until the emotional deviation value is less than or equal to the preset threshold, and obtaining the final target emotional tendency. Among them, the preset emotional tendency is set based on the preset application scenario. For example, in the psychological counseling scenario, the preset emotional tendency is a positive tendency, and in the customer service scenario, the preset emotional tendency is a neutral tendency. The preset threshold is a pre-set emotional deviation threshold.

[0072] According to formula (6), the updated historical sentiment tendency is calculated With preset emotional tendency Z target The sentiment deviation value τ between them is:

[0073]

[0074] In formula (6), ||·|| represents the L2 norm. target It is a preset emotional tendency.

[0075] When the emotion deviation value τ is greater than the preset threshold, it means that the updated historical emotion tendency is not suitable for the preset application scenario of the current dialogue. Then, the updated historical emotion tendency is adjusted according to the following formula (7): Until the sentiment deviation value τ is less than or equal to the preset threshold:

[0076]

[0077] In formula (7), Z generated represents the target emotional tendency; α is the adjustment coefficient, which is used to control the intensity of emotional adjustment.

[0078] The above-mentioned embodiment fine-tunes the updated historical emotional tendency based on the dialogue scenario to make it close to the preset emotional tendency, and fits the actual dialogue scenario on the basis of maintaining the emotional coherence of the dialogue.

[0079] S104: Input the target emotional tendency and the current conversation context into the decoder to obtain the emotional response content output by the decoder.

[0080] Among them, the decoder is the decoder of the large language model.

[0081] In some embodiments, the target sentiment tendency is the updated historical sentiment tendency According to the following formula (8), the target sentiment tendency and the current conversation context X t+1 Input encoder (Deocder) to generate emotional response content y t+1 :

[0082]

[0083] The above-mentioned embodiment ensures the consistency and coherence of emotions in multiple rounds of dialogue by memorizing and dynamically managing emotional tendencies.

[0084] In some embodiments, the target sentiment tendency is the updated and adjusted historical sentiment tendency Z generated According to formula (9), the target sentiment tendency Z generated and the current conversation context X t+1 Input the decoder and get the emotional response content y output by the decoder t+1 .

[0085] y t+1 =Deocder(X t+1 , Z generated ) (9)

[0086] The above-mentioned embodiments not only ensure the naturalness and contextual consistency of the generated emotional response content, but also enhance the emotion guidance and scene adaptation capabilities, providing users with a more empathetic intelligent dialogue experience.

[0087] In summary, the embodiment of the present application provides a large model dialogue method based on multimodal emotion. The method first receives the multimodal input of the user, and performs multimodal emotion analysis on the multimodal input to obtain a multimodal emotion representation; then, the multimodal emotion representation and the current context are input into the emotion generation network to obtain the current emotion tendency output by the emotion generation network; then, the historical emotion tendency is updated based on the current emotion tendency to obtain the target emotion tendency, and the target emotion tendency and the current dialogue context are further input into the decoder to obtain the emotion reply content output by the decoder. In this way, the present application introduces a multimodal dynamic emotion feedback mechanism, integrates multimodal emotion features such as text, voice and image, significantly enhances the system's ability to recognize user emotions, and provides a basis for generating high-quality emotion reply content. At the same time, the present application designs a dynamic adjustment strategy based on real-time changes in user emotions, so that the dialogue system can generate a natural language response that is more in line with the user's emotional state in multiple rounds of interaction. In addition, the present application also constructs an emotional memory management and path planning module, which records the user's emotional change trend and combines the emotional goal guidance mechanism to achieve the functions of emotional consistency and user emotion regulation. Specifically, through precise multimodal emotion recognition, dynamic feedback regulation and emotion goal planning, this application can not only provide natural and coherent emotional dialogue responses, but also help users to actively guide their emotions in specific scenarios, thereby meeting the higher requirements for emotion understanding and generation in practical applications.

[0088] like Figure 2 As shown, Figure 2 A schematic diagram of the structure of a large model dialogue device based on multimodal emotions provided in an embodiment of the present application, the device comprising:

[0089] The sentiment analysis module 201 is used to receive multimodal input from a user and perform multimodal sentiment analysis on the multimodal input to obtain a multimodal sentiment representation;

[0090] The emotion generation module 202 is used to input the multimodal emotion representation and the current conversation context into the emotion generation network to obtain the current emotion tendency output by the emotion generation network;

[0091] The emotion updating module 203 is used to update the historical emotion tendency based on the current emotion tendency to obtain the target emotion tendency;

[0092] The reply generation module 204 is used to input the target emotional tendency and the current conversation context into the decoder to obtain the emotional reply content output by the decoder.

[0093] As an optional implementation in the embodiment of the present application, multimodal input includes: user text, user voice and user image; the sentiment analysis module 201 includes: a text sentiment analysis submodule, which is used to input user text into a pre-trained large language model to obtain text sentiment information output by the pre-trained large language model; a speech sentiment analysis submodule, which is used to input user voice into a pre-trained speech recognition model to obtain speech sentiment information output by the pre-trained speech recognition model; an image sentiment analysis submodule, which is used to input user images into a pre-trained image recognition model to obtain image sentiment information output by the pre-trained image recognition model; and a sentiment fusion submodule, which is used to fuse text sentiment information, speech sentiment information and image sentiment information to obtain multimodal sentiment representation.

[0094] As an optional implementation in the embodiment of the present application, the emotion update module 203 includes: an acquisition submodule, used to acquire historical emotion tendencies from an emotion memory bank; wherein the emotion memory bank is used to store the user's emotion tendencies in each round of conversation; and a first update submodule, used to update the historical emotion tendencies based on the current emotion tendencies and their importance weight coefficients to obtain target emotion tendencies.

[0095] As an optional implementation in the embodiment of the present application, the emotion updating module 203 further includes: a second updating submodule, which is used to update the historical emotion tendency to obtain the target emotion tendency based on the current emotion tendency and its time decay factor.

[0096] As an optional implementation in the embodiment of the present application, the first update submodule or the second update submodule is specifically used to: calculate the emotional deviation value between the updated historical emotional tendency and the preset emotional tendency; when the emotional deviation value is greater than the preset threshold, adjust the updated historical emotional tendency until the emotional deviation value is less than or equal to the preset threshold, and obtain the target emotional tendency.

[0097] For the specific limitations of electronic devices, please refer to the limitations of the large model dialogue method based on multimodal emotions in the above text, which will not be repeated here. Each module in the above electronic device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0098] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0099] In one embodiment, the multimodal emotion-based large model dialogue method provided by the present application can be implemented in the form of a computer program, which can be used in Figure 3 The computer program composed of the program modules enables the processor to execute the steps of the multimodal emotion-based large model dialogue method of each embodiment of the present application described in this specification.

[0100] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the large model dialogue method based on multimodal emotions in each embodiment of the present application described in this specification are implemented.

[0101] In one embodiment, a computer program product is provided, including: the computer program product includes a computer program, and when the computer program is run on a computer, the computer implements the steps of the multimodal emotion-based large model dialogue method of each embodiment of the present application described in this specification.

[0102] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.

[0103] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0104] In the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0105] In this application, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0106] In this application, computer-readable media includes permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0107] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0108] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large model dialogue method based on multimodal emotions, characterized in that: include: Receiving a multimodal input from a user, and performing a multimodal sentiment analysis on the multimodal input to obtain a multimodal sentiment representation; Inputting the multimodal emotion representation and the current conversation context into an emotion generation network to obtain a current emotion tendency output by the emotion generation network; Based on the current sentiment tendency, the historical sentiment tendency is updated to obtain the target sentiment tendency; The target emotional tendency and the current dialogue context are input into a decoder to obtain emotional response content output by the decoder.

2. The method according to claim 1, characterized in that The multimodal input includes: user text, user voice and user image; The performing multimodal sentiment analysis on the multimodal input to obtain a multimodal sentiment representation includes: Inputting the user text into a pre-trained large language model to obtain text sentiment information output by the pre-trained large language model; Inputting the user's speech into a pre-trained speech recognition model to obtain speech emotion information output by the pre-trained speech recognition model; Inputting the user image into a pre-trained image recognition model to obtain image emotion information output by the pre-trained image recognition model; The text emotion information, the speech emotion information and the image emotion information are integrated to obtain a multimodal emotion representation.

3. The method according to claim 1, characterized in that The updating of the historical sentiment tendency based on the current sentiment tendency to obtain the target sentiment tendency includes: Acquiring the historical emotional tendency from an emotional memory library; wherein the emotional memory library is used to store the emotional tendency of the user in each round of conversation; Based on the current emotional tendency and its importance weight coefficient, the historical emotional tendency is updated to obtain the target emotional tendency.

4. The method according to claim 3, characterized in that After acquiring the historical emotional tendency from the emotional memory library, the method further includes: Based on the current emotional tendency and its time decay factor, the historical emotional tendency is updated to obtain the target emotional tendency.

5. The method according to any one of claims 3 or 4, characterized in that: The updating of the historical sentiment tendency to obtain the target sentiment tendency includes: Calculating the emotional deviation value between the updated historical emotional tendency and the preset emotional tendency; When the emotion deviation value is greater than a preset threshold, the updated historical emotion tendency is adjusted until the emotion deviation value is less than or equal to the preset threshold, thereby obtaining the target emotion tendency.

6. A large model dialogue device based on multimodal emotions, characterized in that: include: The sentiment analysis module is used to receive the multimodal input of the user and perform multimodal sentiment analysis on the multimodal input to obtain a multimodal sentiment representation; An emotion generation module, used for inputting the multimodal emotion representation and the current conversation context into an emotion generation network to obtain a current emotion tendency output by the emotion generation network; An emotion updating module, used for updating the historical emotion tendency based on the current emotion tendency to obtain a target emotion tendency; The reply generation module is used to input the target emotional tendency and the current dialogue context into a decoder to obtain the emotional reply content output by the decoder.

7. The device according to claim 6, characterized in that The multimodal input includes: user text, user voice and user image; The sentiment analysis module comprises: A text sentiment analysis submodule, used for inputting the user text into a pre-trained large language model to obtain text sentiment information output by the pre-trained large language model; A speech emotion analysis submodule, used for inputting the user speech into a pre-trained speech recognition model to obtain speech emotion information output by the pre-trained speech recognition model; An image sentiment analysis submodule, used to input the user image into a pre-trained image recognition model to obtain image sentiment information output by the pre-trained image recognition model; The emotion fusion submodule is used to fuse the text emotion information, the speech emotion information and the image emotion information to obtain a multimodal emotion representation.

8. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the large model dialogue method based on multimodal emotions as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the large model dialogue method based on multimodal emotions as described in any one of claims 1 to 5 is implemented.

10. A computer program product, characterized in that include: The computer program product comprises a computer program, and when the computer program is run on a computer, the computer is enabled to implement the multimodal emotion-based large model dialogue method according to any one of claims 1 to 5.

Images