Multi-modal data fusion sentiment analysis system and method

Through the emotion analysis method of multimodal data fusion, the image video and audio characteristics are analyzed, and the problem of unsystematic emotion analysis in the existing technology is solved, the accuracy of emotion analysis is achieved, and the communication tension caused by sensitive information is avoided.

CN118916675AInactive Publication Date: 2024-11-08INNER MONGOLIA ACADEMY OF SCIENCE & TECHNOLOGY +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410994816.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing emotion analysis system fails to systematically analyze image videos and audio, causing sensitive images or sensitive words in communication between people to cause intense changes in emotions, causing communication tension and depression, and affecting relationships.

Method used

The emotion analysis method of multimodal data fusion is adopted to obtain image video and audio feature information, analyze expression changes and audio decibel changes, correct emotional change information, extract key feature information, generate emotional analysis text, and improve the accuracy of emotion analysis.

Benefits of technology

Effectively identify and process sensitive images and sensitive vocabulary to avoid mood swings, improve the atmosphere of communication between people, and prevent relationships from deteriorating.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916675A_ABST
    Figure CN118916675A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-modal data fusion emotion analysis system and method, and relates to the technical field of emotion recognition, and the method comprises the steps: carrying out the correction processing of first emotion change information, and obtaining first emotion change correction information; and analyzing and processing the image feature related information according to the audio decibel change time to obtain second emotion change information. The target image video and the target audio are analyzed through the image video feature extraction module and the audio feature extraction module, the image feature related information and the audio feature related information are determined, and the image feature related information and the audio feature related information are analyzed and corrected. The sentiment analysis text is generated according to the sensitive images and the sensitive vocabularies, the testee is tested according to the sentiment analysis text, the accuracy of the sentiment analysis text is determined, it is avoided that the sensitive images and the sensitive vocabularies appear during follow-up communication between people, communication is tense and depressive, and meanwhile the situation that the relation between the sensitive images and the sensitive vocabularies becomes bad is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition technology, and in particular to a multi-modal data fusion emotion analysis system and method. Background Art

[0002] Human beings' perception and interaction with the things around them are rich and varied, based on multiple modes such as vision, hearing, smell, taste and touch. Similarly, the learning and research of single-modal features by computers no longer meets the pursuit of modern science. With the development of science and technology, research in the field of artificial intelligence has achieved excellent results and has been successfully applied in many fields such as natural language processing, image recognition, recommendation systems, and target detection.

[0003] The existing sentiment analysis system does not conduct systematic analysis of images, videos and audios. When sensitive images or sensitive words appear in communication between people, emotions change drastically, making communication between people tense and depressing, and causing the relationship between the two to deteriorate. Summary of the invention

[0004] In order to solve the above technical problems, a multimodal data fusion sentiment analysis system and method are provided. This technical solution solves the problem that the existing sentiment analysis system proposed in the above background technology does not perform systematic analysis of images, videos and audios. When sensitive images or sensitive words appear in communication between people, emotions change drastically, making communication between people tense and depressed, and causing the relationship between the two to become bad.

[0005] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0006] A sentiment analysis method for multimodal data fusion, comprising:

[0007] Obtain target image video and target audio;

[0008] Perform feature extraction processing on the target image video to obtain image feature related information, wherein the image feature related information includes expression change features and expression change time corresponding to the expression change features;

[0009] Perform feature extraction processing on the target audio to obtain audio feature related information, wherein the audio feature related information includes an audio decibel change feature and an audio decibel change time corresponding to the audio decibel change feature;

[0010] Analyze and process the audio feature related information according to the expression change time to obtain the first emotion change information;

[0011] Performing correction processing on the first emotion change information to obtain first emotion change correction information;

[0012] Analyze and process the image feature related information according to the audio decibel change time to obtain second emotion change information;

[0013] Correcting the second emotion change information to obtain second emotion change correction information;

[0014] Perform feature extraction processing according to the first emotion change correction information and the second emotion change correction information to obtain feature key information;

[0015] Generate sentiment analysis text based on key feature information, conduct test sentiment analysis on the tester, and obtain test image video and test target audio;

[0016] Analyze and process the test image video and test target audio to determine the accuracy of the sentiment analysis text.

[0017] Preferably, the step of performing feature extraction processing on the target image video to obtain information related to image features specifically includes the following steps:

[0018] Perform image preprocessing on the target image video through a face recognition algorithm to obtain image video frames;

[0019] Perform feature extraction on image and video frames to obtain expression change features;

[0020] The target image video is searched and processed according to the expression change characteristics to obtain the expression change time.

[0021] Preferably, the step of performing feature extraction processing on the target audio to obtain audio feature related information specifically comprises the following steps:

[0022] Convert the target audio to obtain an audio digital signal corresponding to the target audio;

[0023] Performing audio preprocessing on the audio digital signal;

[0024] The audio digital information is converted from the time domain to the frequency domain through the fast Fourier transform algorithm, the audio digital information is filtered and noise-reduced in the frequency domain, and the processed audio digital signal is converted from the frequency domain to the time domain through the inverse Fourier transform algorithm to obtain pure target audio;

[0025] Make comparative judgments based on pure target audio;

[0026] If the decibel of the pure target audio is greater than or less than the set threshold range, it is determined that the emotion has changed, and the audio decibel change feature is obtained;

[0027] If the decibel level of the pure target audio is within the set threshold range, it is determined that the emotion has not changed, and the decibel feature of the audio is not recorded;

[0028] The target audio is searched and processed according to the audio decibel change characteristics to obtain the audio decibel change time.

[0029] Preferably, the step of analyzing and processing the audio feature-related information according to the expression change time to obtain the first emotion change information specifically comprises the following steps:

[0030] Traverse the expression change time to obtain the discontinuous points of expression change time;

[0031] Performing data grouping processing on the expression change time according to the discontinuous points of the expression change time to obtain the expression change time group;

[0032] A time series search is performed on the audio feature related information according to the expression change time group, and the audio decibel change characteristics of the audio feature related information are intercepted by taking the start time of the expression change time group as the first interception point, and the end time of the expression change time group as the first cutoff point to stop intercepting the audio decibel change characteristics and obtain the first emotion change information.

[0033] Preferably, the step of performing correction processing on the first emotion change information to obtain the first emotion change correction information specifically comprises the following steps:

[0034] Determine and process the audio decibel change feature in the first emotion change information;

[0035] If the decibel in the audio decibel change feature at the first interception point is greater than or less than the set threshold interval, the audio decibel change feature is expanded and intercepted, and the audio decibel change feature of the set time before the first interception point is obtained; if the decibel in the audio decibel change feature of the set time before the first interception point is greater than or less than the set threshold interval, it is determined that the expression change is caused by the audio decibel change, and the first emotion change correction information is output;

[0036] If the decibel in the audio decibel change feature of the first interception point is within the set threshold interval, the audio decibel change feature is traversed with the first interception point as the starting point, and the audio decibel change feature of the set time after the first interception point is obtained. If the decibel in the audio decibel change feature of the set time after the first interception point is greater than or less than the set threshold interval, it is determined that the audio decibel change is caused by the expression change, and the first emotion change correction information is output;

[0037] If the decibel in the audio decibel change feature within a set time after the first interception point is within a set threshold range, it is determined that the expression change cannot cause the audio decibel change, and the first emotion change correction information is output.

[0038] Preferably, the step of analyzing and processing the image feature related information according to the audio decibel change time to obtain the second emotion change information specifically comprises the following steps:

[0039] Traverse the audio decibel change time to obtain the discontinuous points of the audio decibel change time;

[0040] Performing data grouping processing on the audio decibel change time according to the discontinuous points of the audio decibel change time to obtain an audio decibel change time group;

[0041] A time series search is performed on the image feature related information according to the audio decibel change time group, and the expression change characteristics of the image feature related information are intercepted by taking the start time of the audio decibel change time group as the second interception point. The end time of the audio decibel change time group is taken as the second cutoff point to stop intercepting the expression change characteristics and obtain the second emotion change information.

[0042] Preferably, the step of performing correction processing on the second emotion change information to obtain the second emotion change correction information specifically comprises the following steps:

[0043] Calculate and process the first emotion change correction information and the second emotion change information to obtain the corrected second emotion change information;

[0044] Among them, the specific calculation model for obtaining and correcting the second emotion change information is:

[0045] EC = ES - (ES ∩ EF);

[0046] Wherein, EC is the correction information of the second emotion change; EF is the correction information of the first emotion change; ES is the second emotion change information;

[0047] Determine and process the expression change feature in the corrected second emotion change information;

[0048] If the difference between the expression change feature at the second interception point and the set expression feature is greater than the set first threshold, the expression change feature is expanded and intercepted to obtain the expression change feature at the set time before the second interception point. If the difference between the expression change feature at the set time before the second interception point and the set expression feature is greater than the set first threshold, it is determined that the audio decibel change is caused by the expression change, and the second emotion change correction information is output;

[0049] If the difference between the expression change feature at the second interception point and the set expression feature is less than the set first threshold, the expression change feature is traversed with the second interception point as the starting point to obtain the expression change feature at the set time after the second interception point. If the difference between the expression change feature at the set time after the second interception point and the set expression feature is greater than the set first threshold, it is determined that the expression change is caused by the audio decibel change, and the second emotion change correction information is output;

[0050] If the difference between the expression change feature at the set time after the second interception point and the set expression feature is less than the set first threshold, it is determined that the audio decibel change cannot cause the expression change, and the second emotion change correction information is output.

[0051] Preferably, performing feature extraction processing according to the first emotion change correction information and the second emotion change correction information to obtain feature key information specifically comprises the following steps:

[0052] Performing audio analysis on the first emotion change correction information and the second emotion change correction information to determine emotion change keywords;

[0053] Analyze and process the emotion change keywords to determine feature key information, where the feature key information includes image information and keyword information;

[0054] The first emotion change correction information and the second emotion change correction information are traversed and processed according to the characteristic key information to obtain the emotion fluctuation index.

[0055] Preferably, the analyzing and processing of the test image video and the test target audio to determine the accuracy of the sentiment analysis text specifically comprises the following steps:

[0056] Perform feature extraction processing on the test image video and the test target audio to obtain the test expression change features and the test audio decibel change features;

[0057] Analyze and process the test expression change characteristics and the test audio decibel change characteristics to obtain the test emotion fluctuation index;

[0058] Calculate and process the emotional fluctuation index and the test emotional fluctuation index to obtain the test emotional offset;

[0059] Conduct judgment and processing on the test emotion offset;

[0060] If the test sentiment offset is greater than the set second threshold, the output sentiment analysis text cannot accurately analyze the sentiment fluctuations;

[0061] If the test sentiment offset is less than or equal to the set second threshold, the output sentiment analysis text can accurately analyze the sentiment fluctuations.

[0062] Furthermore, a multimodal data fusion sentiment analysis system is proposed, which is used to implement the multimodal data fusion sentiment analysis method as described above, including:

[0063] A data storage module, the data storage module is used to store target image video and target audio;

[0064] An image and video feature extraction module, which is used to perform image preprocessing and feature extraction on the target image and video, and obtain image and video frames, expression change features, and expression change time;

[0065] An audio feature extraction module, which is used to perform conversion processing, audio preprocessing, and feature extraction processing on the target audio to obtain an audio digital signal, a pure target audio, and an audio decibel change time;

[0066] A first emotion change information generating module, which analyzes and processes audio feature related information according to expression change time to obtain first emotion change information;

[0067] A second emotion change information generating module, wherein the second emotion change information generating module analyzes and processes image feature related information through audio decibel change time to obtain a second emotion change information generating module;

[0068] A correction module, wherein the correction module is used to correct the first emotion change information and the second emotion change information to obtain the first emotion change correction information and the second emotion change correction information;

[0069] A feature key information determination module, wherein the feature key information determination module is used to perform feature extraction processing on the first emotion change correction information and the second emotion change correction information to obtain feature key information;

[0070] A text generation module, wherein the text generation module generates sentiment analysis text according to feature key information;

[0071] A test module, wherein the test module performs an emotional test analysis on the tester according to the emotional analysis text, and obtains a test image video and a test target audio;

[0072] The text accuracy determination module analyzes and judges the test image video and the test target audio to determine the accuracy of the sentiment analysis text.

[0073] Compared with the prior art, the present invention provides a multimodal data fusion sentiment analysis system and method, which has the following beneficial effects:

[0074] The present invention analyzes the target image video and the target audio through an image and video feature extraction module and an audio feature extraction module, determines image feature related information and audio feature related information, analyzes and corrects the image feature related information and audio feature related information, obtains sensitive images and sensitive words, generates a sentiment analysis text according to the sensitive images and sensitive words, tests the tester according to the sentiment analysis text, determines the accuracy of the sentiment analysis text, avoids the appearance of sensitive images and sensitive words in subsequent communication between people, makes the communication tense and depressing, and also avoids the relationship between the two from becoming bad. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 This is a flow chart of steps S100-S1000 in a sentiment analysis method for multimodal data fusion proposed by the present invention;

[0076] Figure 2 This is a flow chart of steps S201-S203 in a sentiment analysis method for multimodal data fusion proposed by the present invention;

[0077] Figure 3 This is a flow chart of steps S301-S307 in a sentiment analysis method for multimodal data fusion proposed by the present invention;

[0078] Figure 4 This is a flow chart of steps S401-S403 in a sentiment analysis method for multimodal data fusion proposed by the present invention;

[0079] Figure 5 This is a flow chart of steps S501-S504 in a sentiment analysis method for multimodal data fusion proposed by the present invention;

[0080] Figure 6 This is a flow chart of steps S601-S603 in a sentiment analysis method for multimodal data fusion proposed by the present invention;

[0081] Figure 7 This is a flow chart of steps S701-S705 in a sentiment analysis method for multimodal data fusion proposed by the present invention;

[0082] Figure 8 This is a flow chart of steps S801-S803 in a sentiment analysis method for multimodal data fusion proposed by the present invention;

[0083] Fig. 9 A flowchart of steps S1001-S1006 in a multimodal data fusion sentiment analysis method proposed in the present invention

[0084] Fig.10This is a structural block diagram of a multimodal data fusion sentiment analysis system proposed in the present invention. DETAILED DESCRIPTION

[0085] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art may think of other obvious variations.

[0086] Reference Figure 1 As shown, a sentiment analysis method for multimodal data fusion includes:

[0087] S100, obtaining target image video and target audio;

[0088] S200, performing feature extraction processing on the target image video to obtain image feature related information, wherein the image feature related information includes expression change features and expression change time corresponding to the expression change features;

[0089] S300, performing feature extraction processing on the target audio to obtain audio feature related information, where the audio feature related information includes an audio decibel change feature and an audio decibel change time corresponding to the audio decibel change feature;

[0090] S400, analyzing and processing audio feature related information according to the expression change time to obtain first emotion change information;

[0091] S500, performing correction processing on the first emotion change information to obtain first emotion change correction information;

[0092] S600, analyzing and processing the image feature related information according to the audio decibel change time to obtain second emotion change information;

[0093] S700, performing correction processing on the second emotion change information to obtain second emotion change correction information;

[0094] S800, performing feature extraction processing according to the first emotion change correction information and the second emotion change correction information to obtain feature key information;

[0095] S900, generating sentiment analysis text according to key feature information, performing test sentiment analysis on the tester, and obtaining test image video and test target audio;

[0096] S1000, analyzing and processing the test image video and the test target audio to determine the accuracy of the sentiment analysis text;

[0097] It is understandable to those skilled in the art that during the communication process, emotional fluctuations may occur due to sensitive images or sensitive words, causing the atmosphere to become tense and depressing during the communication process, which may in turn cause the breakdown of the friendship between the two parties. Therefore, the target audio and target image video are analyzed to determine the sensitive information, so as to avoid the appearance of sensitive information during the communication process and make the communication easier and happier.

[0098] Reference Figure 2 As shown in the figure, the feature extraction process of the target image video is performed to obtain the image feature related information, which specifically includes the following steps:

[0099] S201, performing image preprocessing on the target image video by using a face recognition algorithm to obtain an image video frame;

[0100] S202, performing feature extraction processing on the image video frame to obtain expression change features;

[0101] S203, searching and processing the target image video according to the expression change characteristics to obtain the expression change time;

[0102] In this embodiment, the target image video is preprocessed to remove irrelevant factors that interfere with the subsequent analysis process. In addition, the changes in facial expressions are captured through a convolutional neural network algorithm, and the time of facial expression changes is recorded. This makes the subsequent comprehensive analysis in combination with the target audio more convenient and quick, and can also improve the accuracy of the sentiment analysis text.

[0103] Reference Figure 3 As shown, the feature extraction process of the target audio and the acquisition of audio feature related information specifically include the following steps:

[0104] S301, converting the target audio to obtain an audio digital signal corresponding to the target audio;

[0105] S302, performing audio preprocessing on the audio digital signal;

[0106] S303, converting the audio digital information from the time domain to the frequency domain by using a fast Fourier transform algorithm, filtering and reducing noise on the audio digital information in the frequency domain, and converting the processed audio digital signal from the frequency domain to the time domain by using an inverse Fourier transform algorithm to obtain pure target audio;

[0107] S304, performing comparison and judgment based on the pure target audio;

[0108] S305: If the decibel of the pure target audio is greater than or less than the set threshold range, it is determined that the emotion has changed, and the audio decibel change feature is obtained;

[0109] S306: If the decibel level of the pure target audio is within the set threshold range, it is determined that the emotion has not changed, and the decibel feature of the audio is not recorded;

[0110] S307, searching and processing the target audio according to the audio decibel change characteristics to obtain the audio decibel change time;

[0111] Those skilled in the art will appreciate that the target audio contains noise and sounds that are not part of the communication between the two. Therefore, the target audio is converted into a digital signal, and the digital signal is converted from the time domain to the frequency domain for processing through Fourier transform. The digital signal is filtered and spectrally analyzed in the frequency domain to remove the DC component in the digital signal, and the noise is removed through low-pass filtering so that the processed digital signal contains only audio information of communication, which makes subsequent audio feature extraction operations more convenient and quick. In addition, by setting a threshold interval, when the decibel of the audio is greater than or less than the threshold interval, it indicates that the emotion is in a fluctuating state, and the audio decibel change time is recorded to provide data support for subsequent comprehensive analysis.

[0112] Reference Figure 4 As shown, analyzing and processing the audio feature related information according to the expression change time to obtain the first emotion change information specifically includes the following steps:

[0113] S401, traverse the expression change time to obtain discontinuous points of the expression change time;

[0114] S402, performing data grouping processing on the expression change time according to the discontinuous points of the expression change time, and obtaining an expression change time group;

[0115] S403, performing a time series search for audio feature related information according to the expression change time group, using the start time of the expression change time group as the first interception point, intercepting the audio decibel change feature of the audio feature related information, using the end time of the expression change time group as the first cutoff point, stopping intercepting the audio decibel change feature, and obtaining the first emotion change information;

[0116] In this embodiment, the expression changes are discontinuous within a period of time, so the expression change time is also discontinuous. The expression change time is a data set that stores the time of each expression change. Therefore, the discontinuous points of the expression change time are used as audio interception points to determine whether sensitive words cause emotional fluctuations or sensitive images cause emotional fluctuations.

[0117] Reference Figure 5 As shown, performing correction processing on the first emotion change information and obtaining the first emotion change correction information specifically includes the following steps:

[0118] S501, determining and processing the audio decibel change feature in the first emotion change information;

[0119] S502, if the decibel in the audio decibel change feature at the first interception point is greater than or less than the set threshold interval, expand and intercept the audio decibel change feature, obtain the audio decibel change feature for a set time before the first interception point, if the decibel in the audio decibel change feature for a set time before the first interception point is greater than or less than the set threshold interval, determine that the expression change is caused by the audio decibel change, and output first emotion change correction information;

[0120] S503, if the decibel in the audio decibel change feature of the first interception point is within the set threshold interval, traverse the audio decibel change feature with the first interception point as the starting point, obtain the audio decibel change feature of the set time after the first interception point, if the decibel in the audio decibel change feature of the set time after the first interception point is greater than or less than the set threshold interval, determine that the audio decibel change is caused by the expression change, and output the first emotion change correction information;

[0121] S504: if the decibel in the audio decibel change feature at the set time after the first interception point is within the set threshold range, it is determined that the expression change cannot cause the audio decibel change, and the first emotion change correction information is output;

[0122] Those skilled in the art can understand that by selecting the audio decibel change characteristics of a set time before the first interception point for analysis and judgment, whether the decibel of the audio is already higher than the set threshold interval before the expression changes. If so, it means that the sensitive words cause the emotions to fluctuate, and then cause the expression to change. If not, it means that the expression fluctuates first, which means that seeing the sensitive image causes the expression to change, and then causes the expression to change.

[0123] Reference Figure 6 As shown, analyzing and processing the image feature related information according to the audio decibel change time to obtain the second emotion change information specifically includes the following steps:

[0124] S601, traverse the audio decibel change time to obtain the audio decibel change time discontinuity points;

[0125] S602, performing data grouping processing on the audio decibel change time according to the discontinuous points of the audio decibel change time, and obtaining an audio decibel change time group;

[0126] S603. Perform a time series search for information related to image features according to the audio decibel change time group, use the start time of the audio decibel change time group as the second interception point, intercept the expression change features of the image feature related information, use the end time of the audio decibel change time group as the second cutoff point, stop intercepting the expression change features, and obtain the second emotion change information.

[0127] Reference Figure 7 As shown, performing correction processing on the second emotion change information and obtaining the second emotion change correction information specifically includes the following steps:

[0128] S701, calculating and processing the first emotion change correction information and the second emotion change information to obtain corrected second emotion change information;

[0129] Among them, the specific calculation model for obtaining and correcting the second emotion change information is:

[0130] EC = ES - (ES ∩ EF);

[0131] Wherein, EC is the correction information of the second emotion change; EF is the correction information of the first emotion change; ES is the second emotion change information;

[0132] S702, judging and processing the expression change feature in the corrected second emotion change information;

[0133] S703, if the difference between the expression change feature at the second interception point and the set expression feature is greater than the set first threshold, the expression change feature is expanded and intercepted, and the expression change feature at the set time before the second interception point is obtained; if the difference between the expression change feature at the set time before the second interception point and the set expression feature is greater than the set first threshold, it is determined that the audio decibel change is caused by the expression change, and the second emotion change correction information is output;

[0134] S704, if the difference between the expression change feature at the second interception point and the set expression feature is less than the set first threshold, traverse the expression change feature with the second interception point as the starting point, obtain the expression change feature at the set time after the second interception point, if the difference between the expression change feature at the set time after the second interception point and the set expression feature is greater than the set first threshold, determine that the expression change is caused by the audio decibel change, and output second emotion change correction information;

[0135] S705: if the difference between the expression change feature at the set time after the second interception point and the set expression feature is less than the set first threshold, it is determined that the audio decibel change cannot cause the expression change, and the second emotion change correction information is output;

[0136] It can be understood by those skilled in the art that there is identical data information between the first emotion change correction information and the second emotion change information. Therefore, the second emotion change information is corrected according to the first emotion change correction information so that there is no identical data between the first emotion change correction information and the second emotion change correction information, so that the first emotion change correction information and the second emotion change correction information are two unrelated individuals, so that the subsequently generated emotion analysis text does not contain repeated factors, and at the same time makes the storage space of the emotion analysis text smaller. In addition, when emotions fluctuate, expressions will change and are inconsistent with normal expressions. Therefore, the difference value is introduced for comparison with the first threshold to determine whether to expand the intercepted expression change features.

[0137] Reference Figure 8 As shown, performing feature extraction processing according to the first emotion change correction information and the second emotion change correction information to obtain feature key information specifically includes the following steps:

[0138] S801, performing audio analysis on the first emotion change correction information and the second emotion change correction information to determine emotion change keywords;

[0139] S802, analyzing and processing the emotion change keywords to determine feature key information, where the feature key information includes image information and keyword information;

[0140] S803, traversing and processing the first emotion change correction information and the second emotion change correction information according to the characteristic key information to obtain an emotion fluctuation index;

[0141] In this embodiment, the first emotion change correction information and the second emotion change correction information are traversed according to the characteristic key information to determine the amplitude of expression change and the decibel level of the sound, and the emotion fluctuation index is set according to the amplitude of expression change and the decibel level of the sound to determine whether the characteristic key information needs to be paid attention to in subsequent communication. If the emotion fluctuation index is too large, the characteristic key information should be avoided as much as possible during the communication process to avoid excessive emotional fluctuations.

[0142] Reference Fig. 9 As shown, the test image video and the test target audio are analyzed and processed to determine the accuracy of the sentiment analysis text, which specifically includes the following steps:

[0143] S1001 performs feature extraction processing on the test image video and the test target audio to obtain the test expression change features and the test audio decibel change features;

[0144] S1002 analyzes and processes the test expression change characteristics and the test audio decibel change characteristics to obtain the test emotion fluctuation index;

[0145] S1003 calculates and processes the emotion fluctuation index and the test emotion fluctuation index to obtain the test emotion offset;

[0146] S1004 determines and processes the test emotion offset;

[0147] S1005: if the test emotion offset is greater than the set second threshold, the output emotion analysis text cannot accurately analyze the emotion fluctuation;

[0148] S1006: if the test emotion offset is less than or equal to the set second threshold, outputting the emotion analysis text can accurately analyze the emotion fluctuation;

[0149] It will be understood by those skilled in the art that the sentiment analysis text is mixed with characteristic key information. When the tester sees the characteristic key information, the tester's audio and video are collected, the test image video and the test target audio are analyzed, the magnitude of the emotion fluctuation is determined, and a test emotion fluctuation index is generated. The accuracy of the sentiment analysis text is determined by comparing the test emotion fluctuation index with the emotion fluctuation index. If the sentiment analysis text cannot accurately analyze the emotion fluctuation, the collection and analysis is performed again, and a new sentiment analysis text is generated for testing until the test emotion offset is less than or equal to the set second threshold.

[0150] Reference Fig.10 As shown, a multimodal data fusion sentiment analysis system is used to implement the multimodal data fusion sentiment analysis method as described above, including:

[0151] A data storage module, the data storage module is used to store target image video and target audio;

[0152] An image and video feature extraction module, which is used to perform image preprocessing and feature extraction on the target image and video, and obtain image and video frames, expression change features, and expression change time;

[0153] An audio feature extraction module, which is used to perform conversion processing, audio preprocessing, and feature extraction processing on the target audio to obtain an audio digital signal, a pure target audio, and an audio decibel change time;

[0154] A first emotion change information generating module, which analyzes and processes audio feature related information according to expression change time to obtain first emotion change information;

[0155] A second emotion change information generating module, wherein the second emotion change information generating module analyzes and processes image feature related information through audio decibel change time to obtain a second emotion change information generating module;

[0156] A correction module, wherein the correction module is used to correct the first emotion change information and the second emotion change information to obtain the first emotion change correction information and the second emotion change correction information;

[0157] A feature key information determination module, wherein the feature key information determination module is used to perform feature extraction processing on the first emotion change correction information and the second emotion change correction information to obtain feature key information;

[0158] A text generation module, wherein the text generation module generates sentiment analysis text according to feature key information;

[0159] A test module, wherein the test module performs an emotional test analysis on the tester according to the emotional analysis text, and obtains a test image video and a test target audio;

[0160] The text accuracy determination module analyzes and judges the test image video and the test target audio to determine the accuracy of the sentiment analysis text.

[0161] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A sentiment analysis method for multimodal data fusion, characterized in that: include: Obtain target image video and target audio; Perform feature extraction processing on the target image video to obtain image feature related information, wherein the image feature related information includes expression change features and expression change time corresponding to the expression change features; Perform feature extraction processing on the target audio to obtain audio feature related information, wherein the audio feature related information includes an audio decibel change feature and an audio decibel change time corresponding to the audio decibel change feature; Analyze and process the audio feature related information according to the expression change time to obtain the first emotion change information; Performing correction processing on the first emotion change information to obtain first emotion change correction information; Analyze and process the image feature related information according to the audio decibel change time to obtain second emotion change information; Correcting the second emotion change information to obtain second emotion change correction information; Perform feature extraction processing according to the first emotion change correction information and the second emotion change correction information to obtain feature key information; Generate sentiment analysis text based on key feature information, conduct test sentiment analysis on the tester, and obtain test image video and test target audio; Analyze and process the test image video and test target audio to determine the accuracy of the sentiment analysis text.

2. The sentiment analysis method of multimodal data fusion according to claim 1, characterized in that: The feature extraction process of the target image video to obtain information related to the image features specifically includes the following steps: Perform image preprocessing on the target image video through a face recognition algorithm to obtain image video frames; Perform feature extraction on image and video frames to obtain expression change features; The target image video is searched and processed according to the expression change characteristics to obtain the expression change time.

3. The sentiment analysis method of multimodal data fusion according to claim 2, characterized in that: The feature extraction process of the target audio to obtain audio feature related information specifically includes the following steps: Convert the target audio to obtain an audio digital signal corresponding to the target audio; Performing audio preprocessing on the audio digital signal; The audio digital information is converted from the time domain to the frequency domain through the fast Fourier transform algorithm, the audio digital information is filtered and noise-reduced in the frequency domain, and the processed audio digital signal is converted from the frequency domain to the time domain through the inverse Fourier transform algorithm to obtain pure target audio; Make comparative judgments based on pure target audio; If the decibel of the pure target audio is greater than or less than the set threshold range, it is determined that the emotion has changed, and the audio decibel change feature is obtained; If the decibel level of the pure target audio is within the set threshold range, it is determined that the emotion has not changed, and the decibel feature of the audio is not recorded; The target audio is searched and processed according to the audio decibel change characteristics to obtain the audio decibel change time.

4. The sentiment analysis method of multimodal data fusion according to claim 3 is characterized in that: The step of analyzing and processing the audio feature related information according to the expression change time to obtain the first emotion change information specifically includes the following steps: Traverse the expression change time to obtain the discontinuous points of expression change time; Performing data grouping processing on the expression change time according to the discontinuous points of the expression change time to obtain the expression change time group; A time series search is performed on the audio feature related information according to the expression change time group, and the audio decibel change characteristics of the audio feature related information are intercepted by taking the start time of the expression change time group as the first interception point, and the end time of the expression change time group as the first cutoff point to stop intercepting the audio decibel change characteristics and obtain the first emotion change information.

5. The sentiment analysis method of multimodal data fusion according to claim 4 is characterized in that: The correcting process of the first emotion change information to obtain the first emotion change correction information specifically includes the following steps: Determine and process the audio decibel change feature in the first emotion change information; If the decibel in the audio decibel change feature at the first interception point is greater than or less than the set threshold interval, the audio decibel change feature is expanded and intercepted, and the audio decibel change feature of the set time before the first interception point is obtained; if the decibel in the audio decibel change feature of the set time before the first interception point is greater than or less than the set threshold interval, it is determined that the expression change is caused by the audio decibel change, and the first emotion change correction information is output; If the decibel in the audio decibel change feature of the first interception point is within the set threshold interval, the audio decibel change feature is traversed with the first interception point as the starting point, and the audio decibel change feature of the set time after the first interception point is obtained. If the decibel in the audio decibel change feature of the set time after the first interception point is greater than or less than the set threshold interval, it is determined that the audio decibel change is caused by the expression change, and the first emotion change correction information is output; If the decibel in the audio decibel change feature within a set time after the first interception point is within a set threshold range, it is determined that the expression change cannot cause the audio decibel change, and the first emotion change correction information is output.

6. The sentiment analysis method of multimodal data fusion according to claim 1, characterized in that: The step of analyzing and processing the image feature related information according to the audio decibel change time to obtain the second emotion change information specifically includes the following steps: Traverse the audio decibel change time to obtain the discontinuous points of the audio decibel change time; Performing data grouping processing on the audio decibel change time according to the discontinuous points of the audio decibel change time to obtain an audio decibel change time group; A time series search is performed on the image feature related information according to the audio decibel change time group, and the expression change characteristics of the image feature related information are intercepted by taking the start time of the audio decibel change time group as the second interception point. The end time of the audio decibel change time group is taken as the second cutoff point to stop intercepting the expression change characteristics and obtain the second emotion change information.

7. The sentiment analysis method of multimodal data fusion according to claim 1, characterized in that: The correcting process of the second emotion change information to obtain the second emotion change correction information specifically includes the following steps: Calculate and process the first emotion change correction information and the second emotion change information to obtain the corrected second emotion change information; Among them, the specific calculation model for obtaining and correcting the second emotion change information is: AND C =And S -(AND S ∩E F ); In the formula, E C To correct the second emotion change information; E F Correction information for the first emotion change; E S It is the second emotion change information; Determine and process the expression change feature in the corrected second emotion change information; If the difference between the expression change feature at the second interception point and the set expression feature is greater than the set first threshold, the expression change feature is expanded and intercepted to obtain the expression change feature at the set time before the second interception point. If the difference between the expression change feature at the set time before the second interception point and the set expression feature is greater than the set first threshold, it is determined that the audio decibel change is caused by the expression change, and the second emotion change correction information is output; If the difference between the expression change feature at the second interception point and the set expression feature is less than the set first threshold, the expression change feature is traversed with the second interception point as the starting point to obtain the expression change feature at the set time after the second interception point. If the difference between the expression change feature at the set time after the second interception point and the set expression feature is greater than the set first threshold, it is determined that the expression change is caused by the audio decibel change, and the second emotion change correction information is output; If the difference between the expression change feature at the set time after the second interception point and the set expression feature is less than the set first threshold, it is determined that the audio decibel change cannot cause the expression change, and the second emotion change correction information is output.

8. The sentiment analysis method of multimodal data fusion according to claim 1, characterized in that: The step of performing feature extraction processing according to the first emotion change correction information and the second emotion change correction information to obtain feature key information specifically includes the following steps: Performing audio analysis on the first emotion change correction information and the second emotion change correction information to determine emotion change keywords; Analyze and process the emotion change keywords to determine feature key information, where the feature key information includes image information and keyword information; The first emotion change correction information and the second emotion change correction information are traversed and processed according to the characteristic key information to obtain the emotion fluctuation index.

9. The sentiment analysis method of multimodal data fusion according to claim 1, characterized in that: The step of analyzing and processing the test image video and the test target audio to determine the accuracy of the sentiment analysis text specifically includes the following steps: Perform feature extraction processing on the test image video and the test target audio to obtain the test expression change features and the test audio decibel change features; Analyze and process the test expression change characteristics and the test audio decibel change characteristics to obtain the test emotion fluctuation index; Calculate and process the emotional fluctuation index and the test emotional fluctuation index to obtain the test emotional offset; Conduct judgment and processing on the test emotion offset; If the test sentiment offset is greater than the set second threshold, the output sentiment analysis text cannot accurately analyze the sentiment fluctuations; If the test sentiment offset is less than or equal to the set second threshold, the output sentiment analysis text can accurately analyze the sentiment fluctuations.

10. A multimodal data fusion sentiment analysis system, used to implement the multimodal data fusion sentiment analysis method according to any one of claims 1 to 9, characterized in that: include: A data storage module, the data storage module is used to store target image video and target audio; An image and video feature extraction module, which is used to perform image preprocessing and feature extraction on the target image and video, and obtain image and video frames, expression change features, and expression change time; An audio feature extraction module, which is used to perform conversion processing, audio preprocessing, and feature extraction processing on the target audio to obtain an audio digital signal, a pure target audio, and an audio decibel change time; A first emotion change information generating module, which analyzes and processes audio feature related information according to expression change time to obtain first emotion change information; A second emotion change information generating module, wherein the second emotion change information generating module analyzes and processes image feature related information through audio decibel change time to obtain a second emotion change information generating module; A correction module, wherein the correction module is used to correct the first emotion change information and the second emotion change information to obtain the first emotion change correction information and the second emotion change correction information; A feature key information determination module, wherein the feature key information determination module is used to perform feature extraction processing on the first emotion change correction information and the second emotion change correction information to obtain feature key information; A text generation module, wherein the text generation module generates sentiment analysis text according to feature key information; A test module, wherein the test module performs an emotional test analysis on the tester according to the emotional analysis text, and obtains a test image video and a test target audio; The text accuracy determination module analyzes and judges the test image video and the test target audio to determine the accuracy of the sentiment analysis text.