Multi-modal sentiment analysis system and method, equipment, medium and program product

Abstract

The invention provides a multi-modal sentiment analysis system and method, equipment, a medium and a program product. The method comprises the following steps: acquiring multi-modal data of a user; performing preprocessing on the multi-mode data; carrying out fusion analysis on the preprocessed multi-modal data; and scoring the user emotion according to the multi-modal data fusion analysis result to obtain a scoring result. According to the invention, through fusing multi-modal data such as videos, sounds and texts, the emotion state of the user can be captured more comprehensively, and the accuracy and robustness of emotion recognition are improved; the emotion change of the user can be perceived in real time, and the dialogue strategy is adjusted accordingly, so that the emotion interaction of the AI large model is more natural and smooth, and the user experience is improved; the method is not only suitable for emotion interaction dialogue application of the AI large model, but also can be widely applied to multiple fields of education, psychological consultation, customer service and the like, and has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal sentiment analysis system and method, device, medium, and program product. Background Art

[0002] With the rapid development of artificial intelligence technology, emotional computing has become an important bridge connecting people and machines. Traditional emotional computing often relies on single-modal data (such as text or images), which makes it difficult to fully capture the complex and changeable emotional state of users. The fusion of multimodal data such as video, sound, and text provides a richer and more accurate source of information for emotional analysis. Therefore, developing an emotional analysis system that can efficiently process multimodal inputs and accurately output user emotional scores is of great significance to improving the intelligence level of AI systems. Summary of the invention

[0003] In order to achieve the above-mentioned purpose and other advantages of the present invention, the first object of the present invention is to provide a multimodal sentiment analysis method, comprising the following steps: Obtain user multimodal data; Preprocessing the multimodal data; Perform fusion analysis on the preprocessed multimodal data; The user's emotions are scored according to the multimodal data fusion analysis results to obtain the scoring results.

[0004] Furthermore, the multimodal data includes video data, sound data and text data.

[0005] Furthermore, the video data includes the user's facial expressions and body language, the sound data includes the user's voice information, and the text data includes the user's text input or the conversion result of the voice recognition system.

[0006] Furthermore, the step of preprocessing the multimodal data includes: Denoising and feature extraction are performed on the multimodal data.

[0007] Furthermore, the step of extracting features from the multimodal data includes: Extracting facial features from the video data by face detection and expression recognition technology; Extracting speech features from the sound data by speech recognition and emotional timbre analysis technology; The text data is processed by word segmentation and part-of-speech tagging through natural language processing technology.

[0008] Furthermore, the step of performing fusion analysis on the preprocessed multimodal data includes: Based on the deep learning algorithm and combined with the attention mechanism, the pre-processed multimodal data is fused and analyzed, and the facial expressions in the video data, the emotional intonation in the sound data, and the emotional vocabulary information in the text data are comprehensively considered to achieve a comprehensive understanding of the user's emotions.

[0009] Furthermore, the method further comprises the steps of: Build a high-precision sentiment analysis model based on deep learning; The sentiment analysis model is trained with labeled multimodal datasets to accurately identify the user's emotional state and give corresponding sentiment scores.

[0010] Furthermore, regularization and Dropout techniques are used to prevent overfitting during the training of the sentiment analysis model.

[0011] Furthermore, the step of scoring user emotions according to the multimodal data fusion analysis results includes: The sentiment analysis model is used to score user sentiment based on the multimodal data fusion analysis results.

[0012] Furthermore, the user emotion score is in the form of a continuous value or a discrete classification.

[0013] Furthermore, the method further comprises the steps of: The scoring results are used as an important reference for emotional interactive dialogues of AI large models to adjust dialogue strategies, express care, or provide personalized suggestions.

[0014] The second object of the present invention is to provide a multimodal sentiment analysis system, which applies the above method and includes a multimodal data acquisition module, a data preprocessing module, a multimodal deep fusion module, and a sentiment analysis module; wherein, The multimodal data acquisition module is used to obtain multimodal data of the user; The data preprocessing module is used to preprocess the multimodal data; The multimodal deep fusion module is used to perform fusion analysis on the preprocessed multimodal data; The sentiment analysis module is used to score the user's sentiment according to the multimodal data fusion analysis result to obtain a scoring result.

[0015] A third object of the present invention is to provide a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0016] A fourth object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0017] A fifth object of the present invention is to provide a computer program product, comprising a computer program, which implements the steps of the above method when executed by a processor.

[0018] Compared with the prior art, the present invention has the following beneficial effects: By integrating multimodal data such as video, sound, and text, the present invention can capture the user's emotional state more comprehensively and improve the accuracy and robustness of emotion recognition. It can perceive the user's emotional changes in real time and adjust the dialogue strategy accordingly to make the emotional interaction of the AI ​​large model more natural and smooth, thereby improving the user experience. The present invention is not only suitable for the emotional interaction dialogue application of the AI ​​large model, but can also be widely used in education, psychological counseling, customer service and other fields, and has broad application prospects.

[0019] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. The specific implementation of the present invention is given in detail by the following embodiments and their accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 The multimodal sentiment analysis method process of Example 1 is as follows Figure 1 ; Figure 2 The multimodal sentiment analysis method process of Example 1 is as follows Figure 2 ; Figure 3 This is a flow chart of multimodal data feature extraction in Example 1; Figure 4 Construct a flow chart for the sentiment analysis model of Example 1; Figure 5 This is the business process diagram of the multimodal sentiment analysis system; Figure 6 This is a schematic diagram of sentiment scoring results; Figure 7 Schematic diagram of the multimodal sentiment analysis system of Example 2 Figure 1 ; Figure 8 Schematic diagram of the multimodal sentiment analysis system of Example 2 Figure 2 ; Fig. 9 This is a schematic diagram of a computer device in Example 3; Fig.10 This is a schematic diagram of a computer-readable storage medium of Example 4. DETAILED DESCRIPTION

[0021] The present invention is further described below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. It should be noted that, under the premise of no conflict, the embodiments or technical features described below can be arbitrarily combined to form a new embodiment.

[0022] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

[0023] The figure numbers in this application are only used to distinguish the various steps in the scheme, and are not used to limit the execution order of the various steps. The specific execution order is subject to the description in the specification.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0025] Example 1 A multimodal sentiment analysis method, such as Figure 1 As shown, the following steps are included: S1. Obtain user multimodal data; In some embodiments, Figure 2 , Figure 5 As shown, the multimodal data includes video data, sound data and text data. For example, data in multiple modes such as video, sound, and text can be received from the user end. Video data can capture the user's facial expressions and body language through a camera, sound data can collect the user's voice information through a microphone, and text data can come from the user's text input or the conversion result of a speech recognition system.

[0026] S2. preprocessing the multimodal data; In some embodiments, the step of preprocessing the multimodal data includes: The multimodal data is subjected to denoising and feature extraction.

[0027] Specifically, Figure 3 As shown, the step of extracting features from the multimodal data includes: S21, extracting facial features from the video data by using face detection, expression recognition and other technologies; S22, extracting speech features from the sound data by using speech recognition, emotional timbre analysis and other technologies; S23, performing word segmentation, part-of-speech tagging, etc. on the text data through natural language processing technology.

[0028] S3, performing fusion analysis on the preprocessed multimodal data; In some embodiments, the step of performing fusion analysis on the pre-processed multimodal data includes: Based on deep learning algorithms, combined with attention mechanisms, etc., the pre-processed multimodal data is fused and analyzed, which can comprehensively consider information such as facial expressions in the video data, emotional intonation in the sound data, and emotional vocabulary in the text data, so as to achieve a comprehensive understanding of user emotions.

[0029] This embodiment adopts multimodal deep fusion technology, which can simultaneously process multiple data modalities such as video, sound and text, and through the attention mechanism or fusion layer in the deep learning model, it can realize effective interaction and fusion between different modal information, which can more comprehensively capture the user's emotional expression and improve the accuracy and robustness of emotion recognition.

[0030] S4. Score the user's emotions according to the multimodal data fusion analysis results to obtain a scoring result.

[0031] Optionally, the user emotion score is in the form of a continuous value or a discrete classification, depending on the requirements of the application scenario.

[0032] In some embodiments, Figure 4 As shown, the steps include: S300, build a high-precision sentiment analysis model based on deep learning; S310: The sentiment analysis model is trained using the annotated multimodal dataset to accurately identify the user's emotional state and give a corresponding emotional score. For example, the sentiment analysis model is trained using a large number of annotated multimodal datasets in the own dataset to accurately identify the user's emotional state and give a corresponding emotional score.

[0033] Optionally, regularization, Dropout and other techniques are used during the training of the sentiment analysis model to prevent overfitting, thereby ensuring the stability and generalization ability of the model in complex environments.

[0034] In combination with the above embodiment, the step of scoring the user's emotions according to the multimodal data fusion analysis results includes: The sentiment analysis model is used to score user sentiment based on the multimodal data fusion analysis results.

[0035] In some embodiments, the steps include: The scoring results are used as an important reference for the emotional interaction dialogue of the AI ​​large model to adjust the dialogue strategy, express care or provide personalized suggestions, etc. Figure 6 shown.

[0036] This embodiment implements a real-time emotional feedback mechanism, which can output emotional scores and corresponding dialogue feedback while the user is inputting video, sound or text, and promptly feed back such results to the AI ​​big model. This real-time performance enables the AI ​​big model to perceive the user's emotional changes more promptly, and adjust the dialogue strategy accordingly, improve the naturalness and fluency of human-computer interaction, obtain the user's emotional state every time he speaks, and better capture the emotional changes of users when they mention certain things or certain people, so that AI psychological counseling can more keenly explore the causes of users' psychological problems.

[0037] It should be noted that the multimodal sentiment analysis method provided in this embodiment is not only applicable to emotional interactive dialogue applications in AI large models, but can also be widely used in education, psychological counseling, customer service and other fields. Its powerful emotion recognition ability and real-time feedback mechanism enable the system to provide users with a more personalized and intelligent service experience.

[0038] This embodiment provides a multimodal sentiment analysis method, which includes the following steps: obtaining user multimodal data; preprocessing the multimodal data; performing fusion analysis on the preprocessed multimodal data; and scoring the user's emotions based on the multimodal data fusion analysis results to obtain a scoring result. This embodiment can comprehensively process multiple modal input data such as video, sound, and text, perform accurate sentiment analysis, and output user sentiment scores. The system is particularly suitable for emotional interactive dialogue applications in large AI models, and can significantly improve the intelligence level of human-computer interaction and user experience.

[0039] Example 2 A multimodal sentiment analysis system is provided, and the above method is applied. For a detailed description of the method, reference can be made to the corresponding description in the above method embodiment, and no further description is given here. Figure 7 , Figure 8 As shown, the system includes a multimodal data acquisition module 51, a data preprocessing module 52, a multimodal deep fusion module 53, and a sentiment analysis module 54; wherein, The multimodal data acquisition module is used to obtain multimodal data of the user; The data preprocessing module is used to preprocess the multimodal data; The multimodal deep fusion module is used to perform fusion analysis on the preprocessed multimodal data; The sentiment analysis module is used to score the user's sentiment according to the multimodal data fusion analysis result to obtain a scoring result.

[0040] Based on the technical solution of the above embodiment, optionally, the multimodal data includes video data, sound data and text data.

[0041] Based on the technical solution of the above embodiment, optionally, the video data includes the user's facial expressions and body language, the sound data includes the user's voice information, and the text data includes the user's text input or the conversion result of the voice recognition system.

[0042] Based on the technical solution of the above embodiment, optionally, the step of preprocessing the multimodal data includes: Denoising and feature extraction are performed on the multimodal data.

[0043] Based on the technical solution of the above embodiment, optionally, the step of extracting features from the multimodal data includes: Extracting facial features from the video data by face detection and expression recognition technology; Extracting speech features from the sound data by speech recognition and emotional timbre analysis technology; The text data is processed by word segmentation and part-of-speech tagging through natural language processing technology.

[0044] Based on the technical solution of the above embodiment, optionally, the step of performing fusion analysis on the pre-processed multimodal data includes: Based on the deep learning algorithm and combined with the attention mechanism, the pre-processed multimodal data is fused and analyzed, and the facial expressions in the video data, the emotional intonation in the sound data, and the emotional vocabulary information in the text data are comprehensively considered to achieve a comprehensive understanding of the user's emotions.

[0045] Based on the technical solution of the above embodiment, optionally, the method further includes the following steps: Build a high-precision sentiment analysis model based on deep learning; The sentiment analysis model is trained with labeled multimodal datasets to accurately identify the user's emotional state and give corresponding sentiment scores.

[0046] Based on the technical solution of the above embodiment, optionally, regularization and Dropout technology are used to prevent overfitting during the training of the sentiment analysis model.

[0047] Based on the technical solution of the above embodiment, optionally, the step of scoring the user's emotions according to the multimodal data fusion analysis result includes: The sentiment analysis model is used to score user sentiment based on the multimodal data fusion analysis results.

[0048] Based on the technical solution of the above embodiment, optionally, the user emotion score is in the form of a continuous value or a discrete classification.

[0049] Based on the technical solution of the above embodiment, optionally, the method further includes the following steps: The scoring results are used as an important reference for emotional interactive dialogues of AI large models to adjust dialogue strategies, express care, or provide personalized suggestions.

[0050] The multimodal sentiment analysis system provided in this embodiment adopts a modular design, and each module is relatively independent and easy to expand. This design enables the system to easily integrate new data processing algorithms or sentiment analysis models to meet the needs of different application scenarios. At the same time, the flexibility of the system also facilitates subsequent maintenance and upgrades, and the entire sentiment analysis result output can still be completed even if a modality is missing.

[0051] This embodiment provides a multimodal sentiment analysis system, which includes a multimodal data acquisition module, a data preprocessing module, a multimodal deep fusion module, and a sentiment analysis module. This embodiment can comprehensively process multimodal input data such as video, sound, and text, perform accurate sentiment analysis, and output user sentiment scores. The system is particularly suitable for emotional interactive dialogue applications in AI large models, and can significantly improve the intelligence level of human-computer interaction and user experience.

[0052] Example 3 A computer device 600, such as Fig. 9 As shown, it includes a memory 610, a processor 620, and a computer program 630 stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a multimodal sentiment analysis method are implemented. For a detailed description of the method, reference may be made to the corresponding description in the above method embodiment, which will not be repeated here.

[0053] Example 4 A computer readable storage medium such as Fig.10 As shown, a computer program is stored thereon, and when the computer program is executed by the processor, the steps of a multimodal sentiment analysis method are implemented. For a detailed description of the method, reference may be made to the corresponding description in the above method embodiment, and no further description is given here.

[0054] Example 5 A computer program product, the computer program product comprising a computer program, wherein when the computer program is executed by a processor, the steps of a multimodal sentiment analysis method are implemented. For a detailed description of the method, reference may be made to the corresponding description in the above method embodiment, which will not be repeated here.

[0055] The number of devices and processing scales described here are used to simplify the description of the present invention. Applications, modifications and variations of the present invention will be obvious to those skilled in the art.

[0056] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and implementation modes. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

[0057] The apparatus, computer device, non-volatile computer storage medium and method provided in the embodiments of this specification correspond to each other, and therefore, the apparatus, computer device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, computer device and non-volatile computer storage medium will not be repeated here.

[0058] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software units for implementing the method and structures within the hardware component.

[0059] The systems, devices or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions in various units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or more software and / or hardware.

[0060] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0061] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0062] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0063] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0064] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0065] The specification may be described in the general context of computer-executable instructions executed by a computer, such as program units. Generally, program units include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program units may be located in local and remote computer storage media, including storage devices.

[0066] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0067] The above description is only an embodiment of this specification and is not intended to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of one or more embodiments of this specification.

Claims

1. A multimodal sentiment analysis method, characterized in that: The following steps are involved: Obtain user multimodal data; Preprocessing the multimodal data; Perform fusion analysis on the preprocessed multimodal data; The user's emotions are scored according to the multimodal data fusion analysis results to obtain the scoring results.

2. A multimodal sentiment analysis method as claimed in claim 1, characterized in that: The multimodal data includes video data, sound data and text data.

3. A multimodal sentiment analysis method as claimed in claim 2, characterized in that: The video data includes the user's facial expressions and body language, the sound data includes the user's voice information, and the text data includes the user's text input or the conversion result of the voice recognition system.

4. A multimodal sentiment analysis method as claimed in claim 2, characterized in that: The step of preprocessing the multimodal data comprises: Denoising and feature extraction are performed on the multimodal data.

5. A multimodal sentiment analysis method as claimed in claim 4, characterized in that: The step of extracting features from the multimodal data includes: Extracting facial features from the video data by face detection and expression recognition technology; Extracting speech features from the sound data by speech recognition and emotional timbre analysis technology; The text data is processed by word segmentation and part-of-speech tagging through natural language processing technology.

6. A multimodal sentiment analysis method as claimed in claim 5, characterized in that: The step of performing fusion analysis on the pre-processed multimodal data includes: Based on the deep learning algorithm and combined with the attention mechanism, the pre-processed multimodal data is fused and analyzed, and the facial expressions in the video data, the emotional intonation in the sound data, and the emotional vocabulary information in the text data are comprehensively considered to achieve a comprehensive understanding of the user's emotions.

7. A multimodal sentiment analysis method as claimed in claim 1, characterized in that: Also includes the steps: Build a high-precision sentiment analysis model based on deep learning; The sentiment analysis model is trained with labeled multimodal datasets to accurately identify the user's emotional state and give corresponding sentiment scores.

8. A multimodal sentiment analysis method as claimed in claim 7, characterized in that: Regularization and Dropout techniques are used to prevent overfitting during the training of the sentiment analysis model.

9. A multimodal sentiment analysis method as claimed in claim 7, characterized in that: The step of scoring user emotions according to the multimodal data fusion analysis results includes: The sentiment analysis model is used to score user sentiment based on the multimodal data fusion analysis results.

10. A multimodal sentiment analysis method according to claim 1, characterized in that: The user emotion score is in the form of a continuous value or a discrete classification.

11. A multimodal sentiment analysis method according to claim 1, characterized in that: Also includes the steps: The scoring results are used as an important reference for emotional interactive dialogues of AI large models to adjust dialogue strategies, express care, or provide personalized suggestions.

12. A multimodal sentiment analysis system, using the method according to any one of claims 1 to 11, characterized in that: It includes multimodal data acquisition module, data preprocessing module, multimodal deep fusion module and sentiment analysis module; among them, The multimodal data acquisition module is used to obtain multimodal data of the user; The data preprocessing module is used to preprocess the multimodal data; The multimodal deep fusion module is used to perform fusion analysis on the preprocessed multimodal data; The sentiment analysis module is used to score the user's sentiment according to the multimodal data fusion analysis result to obtain a scoring result.

13. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images