A method, apparatus, device and medium for multi-modal sentiment analysis

Abstract

Embodiments of the present application provide a multi-modal sentiment analysis method, device, equipment and medium, the method comprising: extracting a text feature vector corresponding to a text to be analyzed, wherein the text to be analyzed is obtained by converting the speech to be analyzed; fusing the text feature vector and a speech feature vector of the speech to be analyzed to obtain a fusion vector; inputting the fusion vector into a multi-modal sentiment analysis model, and predicting the sentiment of the fusion vector by the multi-modal sentiment analysis model to obtain the probability of at least one sentiment label corresponding to the speech to be analyzed. Through some embodiments of the present application, the text feature vector and the speech feature vector can be fused to obtain an accurate sentiment label, so that targeted services can be provided to the user.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of user sentiment analysis, and particularly relate to a multi-modal sentiment analysis method, device, equipment and medium. BACKGROUND

[0002] In related technologies, in the scene of electric call-out, user sentiment analysis is generally performed after converting speech into text and then performing text sentiment recognition, which leads to the loss of some user speech information, and the speech-to-text conversion in the upstream also has recognition errors, thus causing a certain negative impact on the accuracy of sentiment recognition.

[0003] Therefore, how to improve the accuracy of user sentiment analysis has become a problem to be solved. SUMMARY

[0004] Embodiments of the present application provide a multi-modal sentiment analysis method, device, equipment and medium, which can at least fuse a text feature vector and a speech feature vector to obtain an accurate sentiment label, so as to provide targeted services for users.

[0005] In a first aspect, the present application provides a multi-modal sentiment analysis method, comprising: extracting a text feature vector corresponding to a text to be analyzed, wherein the text to be analyzed is obtained by converting a speech to be analyzed; fusing the text feature vector and a speech feature vector of the speech to be analyzed to obtain a fusion vector; inputting the fusion vector into a multi-modal sentiment analysis model, and performing sentiment analysis prediction on the fusion vector through the multi-modal sentiment analysis model to obtain a probability corresponding to at least one sentiment label of the speech to be analyzed.

[0006] Therefore, unlike the method of directly using the text to be analyzed for user sentiment analysis in related technologies, the present application fuses the text feature vector and the speech feature vector of the speech to be analyzed to obtain a fusion vector, and then uses the fusion vector for sentiment analysis prediction, so as to obtain an accurate sentiment label and provide targeted services for users.

[0007] In combination with the first aspect, in an embodiment of the present application, the fusing the text feature vector and the speech feature vector of the speech to be analyzed to obtain a fusion vector comprises: performing vector splicing on the text feature vector and the speech feature vector to obtain a fusion vector, wherein the vector splicing manner at least includes adding the text feature vector and the speech feature vector.

[0008] Therefore, by splicing the text feature vector and the speech feature vector, the embodiment of the present application can use the features representing the text and the features representing the tone in the process of sentiment analysis, so as to obtain an accurate sentiment label.

[0009] With reference to the first aspect, in an embodiment of the present application, before the fusion vector is obtained by fusing the text feature vector and the speech feature vector of the speech to be analyzed, the method further comprises: inputting the speech to be analyzed into a speech extraction model, and performing convolution operation and pooling operation on the speech to be analyzed by the speech extraction model to obtain the speech feature vector.

[0010] Therefore, by performing convolution operation and pooling operation on the speech to be analyzed, the embodiment of the present application can obtain an accurate speech feature vector.

[0011] With reference to the first aspect, in an embodiment of the present application, after the probability corresponding to the at least one sentiment label of the speech to be analyzed is obtained, the method further comprises: obtaining an electroencephalogram conversion file corresponding to the speech to be analyzed, wherein the electroencephalogram conversion file is the brain wave data of the relevant person collected in the process of generating the speech to be analyzed; based on the electroencephalogram conversion file, the probability corresponding to the at least one sentiment label is checked to obtain a target probability corresponding to the at least one sentiment label; and the target probability is used to confirm the target sentiment label of the speech to be analyzed.

[0012] Therefore, by checking the sentiment label based on the brain wave data, the embodiment of the present application can obtain a more accurate sentiment label.

[0013] With reference to the first aspect, in an embodiment of the present application, after the probability corresponding to the at least one sentiment label of the speech to be analyzed is obtained, the method further comprises: obtaining an electroencephalogram conversion file corresponding to the speech to be analyzed, wherein the electroencephalogram conversion file is the brain wave data of the relevant person collected in the process of generating the speech to be analyzed; based on the electroencephalogram conversion file, the probability corresponding to the at least one sentiment label is checked to obtain a target probability corresponding to the at least one sentiment label; and the target probability is used to confirm the target sentiment label of the speech to be analyzed.

[0014] Therefore, by checking the sentiment label based on the brain wave data, the embodiment of the present application can obtain a more accurate sentiment label.

[0015] In a second aspect, the present application provides a device for multi-modal sentiment analysis, the device comprising: a vector extraction module configured to extract a text feature vector corresponding to a text to be analyzed, wherein the text to be analyzed is obtained by converting a speech to be analyzed; a vector fusion module configured to fuse the text feature vector with a speech feature vector of the speech to be analyzed to obtain a fusion vector; and a sentiment analysis module configured to input the fusion vector into a multi-modal sentiment analysis model, and to perform sentiment analysis prediction on the fusion vector by the multi-modal sentiment analysis model to obtain a probability corresponding to at least one sentiment label of the speech to be analyzed.

[0016] In combination with the second aspect, in an embodiment of the present application, the vector fusion module is further configured to perform vector splicing on the text feature vector and the speech feature vector to obtain the fusion vector, wherein the vector splicing manner comprises at least adding the text feature vector and the speech feature vector.

[0017] In combination with the second aspect, in an embodiment of the present application, the vector extraction module is further configured to input the speech to be analyzed into a speech extraction model, and to perform convolution operation and pooling operation on the speech to be analyzed by the speech extraction model to obtain the speech feature vector.

[0018] In combination with the second aspect, in an embodiment of the present application, the sentiment analysis module is further configured to obtain an electroencephalogram conversion file corresponding to the speech to be analyzed, wherein the electroencephalogram conversion file is electroencephalogram data of a relevant person collected in a process of generating the speech to be analyzed; to verify the probability corresponding to the at least one sentiment label based on the electroencephalogram conversion file to obtain a target probability corresponding to the at least one sentiment label; and to confirm a target sentiment label of the speech to be analyzed by the target probability.

[0019] In combination with the second aspect, in an embodiment of the present application, the sentiment analysis module is further configured to obtain an eye movement conversion file corresponding to the speech to be analyzed, wherein the eye movement conversion file is eye movement data of a relevant person collected in a process of generating the speech to be analyzed; to verify the probability corresponding to the at least one sentiment label based on the eye movement conversion file to obtain a target probability corresponding to the at least one sentiment label; and to confirm a target sentiment label of the speech to be analyzed by the target probability.

[0020] In a third aspect, the present application provides an electronic device, comprising: a processor, a memory and a bus; the processor is connected with the memory through the bus, the memory stores a computer program, and the computer program is executed by the processor to realize the method according to any one of the embodiments of the first aspect.

[0021] In a fourth aspect, the present application provides a computer readable storage medium, having stored thereon a computer program, which when executed by a computer, implements the method according to any one of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A system composition schematic diagram of a multi-modal sentiment analysis according to an embodiment of the present application is shown;

[0023] Figure 2 One of the method flowcharts of a multi-modal sentiment analysis according to an embodiment of the present application is shown;

[0024] Figure 3 The second method flowchart of a multi-modal sentiment analysis according to an embodiment of the present application is shown;

[0025] Figure 4 The third method flowchart of a multi-modal sentiment analysis according to an embodiment of the present application is shown;

[0026] Figure 5 The fourth method flowchart of a multi-modal sentiment analysis according to an embodiment of the present application is shown;

[0027] Figure 6 The fifth method flowchart of a multi-modal sentiment analysis according to an embodiment of the present application is shown;

[0028] Figure 7 A device composition schematic diagram of a multi-modal sentiment analysis according to an embodiment of the present application is shown;

[0029] Figure 8 An electronic device composition schematic diagram according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings for the embodiments of the present application to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all embodiments of the present application. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0031] The embodiments of the present application can be applied to the scene of analyzing user emotions. In order to solve the problems in the background art, in some embodiments of the present application, the fusion vector of the text feature vector and the speech feature vector is used to perform emotion analysis prediction. For example, in some embodiments of the present application, first, the text feature vector corresponding to the text to be analyzed is extracted, then the text feature vector and the speech feature vector are fused to obtain a fusion vector, and finally the fusion vector is input into a multi-modal emotion analysis model to obtain at least one emotion label.

[0032] The method steps in the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0033] Figure 1 The structural diagram of the multi-modal emotion analysis system in some embodiments of the present application is provided, which includes a user 110 and an outbound server 120. Specifically, during the communication between the user 110 and the outbound server 120, the outbound server 120 obtains the user call, then fuses the text vector and the speech vector corresponding to the user call to obtain a fusion vector, performs emotion analysis prediction on the fusion vector, and obtains the probability of at least one emotion label of the speech to be analyzed.

[0034] Unlike the embodiments of the present application, in the related art, in the scene of electric sales outbound, user emotion analysis is generally performed after speech-to-text, and then text emotion recognition is performed, which leads to the loss of some user speech information, and the speech-to-text in the upstream also has recognition errors, thus causing a certain negative impact on the accuracy of emotion recognition. The present application uses a fusion vector to perform emotion analysis prediction, thereby obtaining an accurate emotion label probability.

[0035] The implementation process of the multi-modal emotion analysis provided by the present application will be described below.

[0036] At least to solve the problems in the background art, such as Figure 2 As shown in the figure, some embodiments of the present application provide a multi-modal emotion analysis method, which includes:

[0037] S210, extracting a text feature vector corresponding to a text to be analyzed.

[0038] It should be noted that the text to be analyzed is obtained by converting the speech to be analyzed. During the communication with the user, the direct call (i.e. the speech to be analyzed) between the outbound server and the user is converted in real time to obtain the text to be analyzed.

[0039] It can be understood that the method used to convert the speech to be analyzed into the text to be analyzed is the same as the method used to convert the speech into text in the related art, which will not be described herein.

[0040] In one embodiment of this application, after the speech to be analyzed is converted into text to be analyzed in real time, the text to be analyzed is input into a text extraction model, and the text extraction model extracts the features of the text to be analyzed to obtain a text feature vector.

[0041] Specifically, the text extraction model in this application is established using the BERT model. After the text to be analyzed is input into the BERT model, the BERT model performs feature processing on the textual information in the text to be analyzed and obtains the text feature vector.

[0042] For example, such as Figure 3 As shown, the text to be analyzed is input into the BERT model 310, and the text feature vector is obtained after the BERT model 310 calculates it.

[0043] In one embodiment of this application, the speech to be analyzed is input into a speech extraction model, and the speech extraction model performs convolution and pooling operations on the speech to be analyzed to obtain speech feature vectors.

[0044] In other words, after extracting the text feature vectors from the text to be analyzed, the speech to be analyzed also needs to be input into the speech extraction model. In the speech extraction model, the speech to be analyzed is subjected to alternating convolution and pooling operations to obtain speech feature vectors. Specifically, the speech to be analyzed is convolved by the convolution kernel in the speech extraction model, and the speech to be analyzed is pooled by the pooling layer in the speech extraction model after the convolution operation. This process is repeated alternately.

[0045] Specifically, this application uses a convolutional neural network (CNN) model to process the speech to be analyzed, in order to capture and extract temporal information. The specific process involves the input speech to be analyzed undergoing multiple convolution and pooling operations to transform it into a new vector, which serves as the speech feature vector.

[0046] For example, such as Figure 4 As shown, firstly, the speech to be analyzed is input into convolutional layer 410, where convolution is performed. Then, the convolutionally processed speech is input into pooling layer 420 for pooling to obtain a first pooling vector. Next, the first pooling vector is input into convolutional layer 410 again for convolution to obtain a second pooling vector. Finally, the second pooling vector is input into global pooling layer 430 for global pooling to obtain the speech feature vector.

[0047] It can be understood that the present application does not limit the execution order of the steps of extracting the text feature vector and the steps of extracting the speech feature vector. The text feature vector can be extracted first during the conversation with the user, or the speech feature vector can be extracted first during the conversation with the user.

[0048] S220, fusing the text feature vector and the speech feature vector of the speech to be analyzed to obtain a fusion vector.

[0049] That is, the text feature vector and the speech feature vector are spliced to obtain a fusion vector. In the present application, the splicing method of the text feature vector and the speech feature vector includes the following three methods:

[0050] In an embodiment of the present application, the first method of vector splicing is to add the vectors in the same position of the text feature vector and the speech feature vector. For example, the first vector in the text feature vector is added to the first vector in the speech feature vector, the second vector in the text feature vector is added to the second vector in the speech feature vector, and so on to obtain the fusion vector.

[0051] It can be understood that the text feature vector and the speech feature vector are both vector groups, and each vector group includes multiple vectors.

[0052] In an embodiment of the present application, the second method of vector splicing is to directly splice the text feature vector and the speech feature vector. For example, the last vector in the text feature vector is spliced with the first vector in the speech feature vector to obtain the fusion vector. Alternatively, the last vector in the speech feature vector is directly spliced with the first vector in the text feature vector to obtain the fusion vector.

[0053] In an embodiment of the present application, the third method of vector splicing is to merge the text feature vector and the speech feature vector into a vector group to realize vector splicing. For example, the first vector of the text feature vector is merged with the first vector of the speech feature vector, and the second vector of the text feature vector is merged with the second vector of the speech feature vector, and so on to obtain the fusion vector.

[0054] S230, inputting the fusion vector into the multi-modal sentiment analysis model, performing sentiment analysis and prediction on the fusion vector through the multi-modal sentiment analysis model to obtain the probability corresponding to at least one sentiment label of the speech to be analyzed.

[0055] In other words, after obtaining the fused vector of text feature vector and speech feature vector, the fused vector is input into a multimodal sentiment analysis model based on the Feed Forward Neural Network (FFN) model. The FFN model is used to analyze and predict user sentiment and obtain the probability corresponding to at least one sentiment label.

[0056] Understandably, at least one sentiment label can be a label indicating positive user sentiment, a label indicating neutral user sentiment, or a label indicating negative user sentiment.

[0057] For example, such as Figure 5 As shown, the speech feature vector and text feature vector are fused to obtain a fused vector. This fused vector is then input into the FFN model 510, which outputs the probabilities of a user's emotion being positive, neutral, and negative. Specifically, the feedforward neural network consists of two layers: the input layer is the fused vector concatenated from the speech and text feature vectors, and the output layer is the emotion label prediction layer, which is also the output layer of the entire model, yielding the output probabilities of the three emotions. Therefore, it can be seen that the model considers both audio and text information during the prediction process. The fusion of multiple information sources can extract more user features, thus enabling a more accurate judgment of the user's emotion.

[0058] In one embodiment of this application, after obtaining the probabilities corresponding to at least one sentiment label, the sentiment label with the highest probability value can be used as the user's real-time sentiment analysis result. For example, if the probability value of positive is 90%, the probability value of neutral is 10%, and the probability value of negative is 0% among at least one sentiment label, then the user's current mood is confirmed to be positive.

[0059] In one embodiment of this application, when the probabilities of each emotion tag in at least one emotion tag are relatively close, the EEG conversion file or eye-tracking conversion file corresponding to the speech to be analyzed is used to further analyze the user's emotion and obtain an accurate emotion tag. For example, if the probabilities of each emotion tag in at least one emotion tag are: a positive probability of 33%, a neutral probability of 30%, and a negative probability of 37%, then the EEG conversion file or eye-tracking conversion file is used to determine the user's emotion tag.

[0060] It is understood that the probabilities of each sentiment tag are relatively close, that is, the probabilities of each sentiment tag differ from each other within 10% or within 20%, and can be limited according to the actual situation. The embodiments of this application are not limited to this.

[0061] As an embodiment of the present application, the brain wave conversion file corresponding to the voice to be analyzed is used to further analyze the emotion of the user, including:

[0062] S1, obtaining a brain wave conversion file corresponding to the voice to be analyzed.

[0063] It should be noted that the brain wave conversion file is the brain wave data of the user collected in the process of generating the voice to be analyzed, or the brain wave data of the agent in the process of communication between the agent and the user. It can be understood that, under the condition that the user agrees to collect, the brain wave data of the user is collected by the collection device on the user terminal during the user's call, close to the top of the user's ear. Under the condition that the agent agrees to collect, the collection device is worn for the agent to collect the brain wave data of the agent, and the emotion label of the user is fed back through the brain wave data of the agent.

[0064] S2, checking the probability corresponding to at least one emotion label based on the brain wave conversion file to obtain a target probability corresponding to at least one emotion label.

[0065] Before obtaining the brain wave conversion file, the brain wave emotion analysis model needs to be trained, and the brain wave training set data and the label are input. The brain wave emotion analysis model to be trained continuously learns the brain wave features corresponding to each label to obtain a trained brain wave emotion analysis model.

[0066] After obtaining the brain wave emotion analysis model, the real-time brain wave data of the agent or the user is input into the brain wave emotion analysis model, and the brain wave feature analysis is performed through the brain wave emotion analysis model to obtain a target probability corresponding to at least one emotion label.

[0067] S3, confirming the target emotion label of the voice to be analyzed through the target probability.

[0068] That is, the emotion label corresponding to the maximum value of the target probability of at least one emotion label is selected as the target emotion label of the user at present.

[0069] As an embodiment of the present application, the eye movement conversion file corresponding to the voice to be analyzed is used to further analyze the emotion of the user, including:

[0070] S1, obtaining an eye movement conversion file corresponding to the voice to be analyzed.

[0071] It should be noted that the eye movement conversion file is the eye movement data of the user collected in the process of generating the speech to be analyzed or the agent in the process of communicating with the user. It can be understood that, under the condition that the user agrees to collect, the eye movement data of the user watching the screen and the pupil scaling data in the process of the user calling are tracked through the collection device on the user terminal. Under the condition that the agent agrees to collect, the eye movement data of the agent watching the screen and the pupil scaling data on the computer used by the agent are tracked, and the emotional label of the user is fed back through the eye movement data of the agent.

[0072] S2, based on the eye movement conversion file, the probability corresponding to at least one emotional label is checked to obtain the target probability corresponding to at least one emotional label.

[0073] Before the above-mentioned obtaining of the eye movement data, the eye movement emotion analysis model needs to be trained, and the eye movement training set data and the label are input, the eye movement features corresponding to each label are continuously learned in the eye movement emotion analysis model to be trained, and the trained eye movement emotion analysis model is obtained.

[0074] After obtaining the eye movement emotion analysis model, the real-time eye movement data of the agent or the user is input into the eye movement emotion analysis model, the eye movement feature analysis is performed through the eye movement emotion analysis model, and the target probability corresponding to at least one emotional label is obtained.

[0075] S3, confirming the target emotional label of the speech to be analyzed through the target probability.

[0076] That is, the emotional label corresponding to the maximum value of the target probability of at least one emotional label is selected as the target emotional label of the user at present.

[0077] As a specific embodiment of the present application, in the electric sales outbound call scene, the emotion analysis function is mostly realized in the natural language understanding module, mainly for the processing of text, and the present application combines voice and text information, so that the outbound server can more accurately judge the emotional state of the user. The present application constructs a dual-modal emotion analysis model of voice and text, splices the features of the two modalities, inputs them into the downstream classification model, and performs information fusion.

[0078] As shown in Figure 6 The speech to be analyzed is input into the speech recognition model 610 to generate the text to be analyzed, the text to be analyzed is input into the BERT model 310 to output the text feature vector. At the same time, the speech to be analyzed is input into the CNN model 620 to output the speech feature vector, and then the text feature vector and the speech feature vector are fused to obtain a fusion vector, the fusion vector is input into the FFN model 510 to obtain at least one emotional label of user emotion analysis.

[0079] That is, the speech recognition model is to recognize speech as text data, and a third-party speech recognition service can be used. The text data is subjected to feature extraction by a BERT model to obtain a text feature vector, and the audio data is subjected to feature extraction by a CNN model to obtain a speech feature vector. Then, the two vectors are spliced to obtain a fusion vector of the dual modalities, which is input into a downstream FFN model to obtain at least one sentiment label of user emotion analysis.

[0080] Therefore, the multi-modal model can maximize the advantages of modal fusion, achieving an effect of 1 plus 1 greater than 2. In particular, for the sarcasm tactics in user dialogue, the multi-modal fusion model can effectively determine the negative emotion of the user.

[0081] For example, in the e-sell scenario, when the user shows a relatively negative attitude, the user will communicate with the outbound server in a negative tone. At this time, the user's emotion cannot be correctly determined through text, and the outbound server will continue to broadcast according to the preset process. This will promote the user's more negative emotion, thereby increasing the risk of complaints. For example, the user says "good good" at a certain moment in a relatively negative tone. The correct emotion cannot be determined through text, but the information of the speech is combined to increase the probability of correct determination by the outbound server.

[0082] Single modalities are easily affected by other factors, such as noise factors of current sound / background sound in audio, leading to model misjudgment; and text is affected by the error of the upstream speech recognition result, leading to model misjudgment. Therefore, the multi-modal fusion modeling method, which combines text and speech features, can improve the accuracy of user emotion analysis.

[0083] The above describes a specific embodiment of a multi-modal sentiment analysis method of the present application, and the following will describe a multi-modal sentiment analysis device of the present application.

[0084] As shown in Figure 7 Some embodiments of the present application provide a multi-modal sentiment analysis device 700, which comprises a vector extraction module 710, a vector fusion module 720, and a sentiment analysis module 730.

[0085] The vector extraction module 710 is configured to extract a text feature vector corresponding to the text to be analyzed, wherein the text to be analyzed is obtained by converting the speech to be analyzed; the vector fusion module 720 is configured to fuse the text feature vector and a speech feature vector of the speech to be analyzed to obtain a fusion vector; and the sentiment analysis module 730 is configured to input the fusion vector into a multi-modal sentiment analysis model, perform sentiment analysis prediction on the fusion vector through the multi-modal sentiment analysis model, and obtain a probability corresponding to at least one sentiment label of the speech to be analyzed.

[0086] In an embodiment of the present application, the vector fusion module 720 is further configured to perform vector splicing on the text feature vector and the speech feature vector to obtain a fusion vector, where the vector splicing manner at least includes adding the text feature vector and the speech feature vector.

[0087] In an embodiment of the present application, the vector extraction module 710 is further configured to input the speech to be analyzed into a speech extraction model, and perform convolution operation and pooling operation on the speech to be analyzed by the speech extraction model to obtain the speech feature vector.

[0088] In an embodiment of the present application, the sentiment analysis module 730 is further configured to obtain an electroencephalogram conversion file corresponding to the speech to be analyzed, where the electroencephalogram conversion file is electroencephalogram data of a relevant person collected in a process of generating the speech to be analyzed; check a probability corresponding to the at least one sentiment label based on the electroencephalogram conversion file to obtain a target probability corresponding to the at least one sentiment label; and confirm a target sentiment label of the speech to be analyzed through the target probability.

[0089] In an embodiment of the present application, the sentiment analysis module 730 is further configured to obtain an eye movement conversion file corresponding to the speech to be analyzed, where the eye movement conversion file is eye movement data of a relevant person collected in a process of generating the speech to be analyzed; check a probability corresponding to the at least one sentiment label based on the eye movement conversion file to obtain a target probability corresponding to the at least one sentiment label; and confirm a target sentiment label of the speech to be analyzed through the target probability.

[0090] In the embodiments of the present application, Figure 7 the modules shown in the figures can realize Figures 1 to 6 the processes in the method embodiments. Figure 7 the operations and / or functions of the modules in the figures, respectively, are to realize Figures 1 to 6 the corresponding flows in the method embodiments. For details, refer to the descriptions in the method embodiments, and the detailed descriptions are appropriately omitted here to avoid repetition.

[0091] As Figure 8 shown, the embodiments of the present application provide an electronic device 800, which includes a processor 810, a memory 820 and a bus 830, the processor is connected with the memory through the bus, the memory stores computer readable instructions, when the computer readable instructions are executed by the processor, the computer readable instructions are used to realize the method in any one of all the above embodiments, for details, refer to the descriptions in the method embodiments, and the detailed descriptions are appropriately omitted here to avoid repetition.

[0092] The bus is configured to enable direct connection and communication among the components. In an embodiment of the present application, the processor can be an integrated circuit chip, which has the ability to process signals. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), or the like. The processor can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor can implement or execute the various methods, steps, and logical block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0093] The memory can be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or the like. The memory stores computer readable instructions, which, when executed by the processor, can perform the methods described in the above embodiments.

[0094] It can be understood that Figure 8 The structure shown is only schematic, and can further include more or fewer components than those shown, or have a different configuration of components than those shown. Figure 8 The components shown in the figures can be implemented in hardware, software, or a combination thereof. Figure 8 The components shown in the figures can be implemented in hardware, software, or a combination thereof. Figure 8 The components shown in the figures can be implemented in hardware, software, or a combination thereof.

[0095] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is executed by a server to implement the method described in any one of the above embodiments. For details, refer to the description of the method embodiments. To avoid repetition, the detailed description is appropriately omitted here.

[0096] The above descriptions are only the preferred embodiment of the present application, but not intended to limit the present application. Any modification, equivalent replacement and improvement made within the principle and technical scope of the present application should be included in the protection scope of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, thus, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0097] The above descriptions are only the specific embodiments of the present application, but the protection scope of the present application is not limited to this. Any modification, equivalent replacement and improvement made within the technical range disclosed by the present application can be easily thought by any person skilled in the art, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for multimodal sentiment analysis, characterized in that, The method includes: Extract the text feature vector corresponding to the text to be analyzed, wherein the text to be analyzed is obtained by converting the speech to be analyzed; The text feature vector is fused with the speech feature vector of the speech to be analyzed to obtain a fused vector; The fused vector is input into a multimodal sentiment analysis model, and the multimodal sentiment analysis model is used to perform sentiment analysis and prediction on the fused vector to obtain the probability of at least one sentiment label of the speech to be analyzed. After obtaining the probability corresponding to at least one emotion label of the speech to be analyzed, the method further includes: Obtain the EEG conversion file corresponding to the speech to be analyzed, wherein the EEG conversion file is the EEG data of relevant personnel collected during the generation of the speech to be analyzed; Based on the EEG conversion file, the probability corresponding to the at least one emotion label is verified to obtain the target probability corresponding to the at least one emotion label; The target sentiment label of the speech to be analyzed is determined by the target probability. or, Obtain an eye-tracking conversion file corresponding to the speech to be analyzed, wherein the eye-tracking conversion file is eye movement data of relevant personnel collected during the generation of the speech to be analyzed; Based on the eye-tracking conversion file, the probability corresponding to the at least one emotion label is verified to obtain the target probability corresponding to the at least one emotion label; The target sentiment label of the speech to be analyzed is determined by the target probability.

2. The method according to claim 1, characterized in that, The step of fusing the text feature vector with the speech feature vector of the speech to be analyzed to obtain a fused vector includes: The text feature vector and the speech feature vector are concatenated to obtain a fused vector, wherein the concatenation method includes at least adding the text feature vector and the speech feature vector.

3. The method according to any one of claims 1-2, characterized in that, Before fusing the text feature vector with the speech feature vector of the speech to be analyzed to obtain a fused vector, the method further includes: The speech to be analyzed is input into the speech extraction model, and the speech extraction model performs convolution and pooling operations on the speech to be analyzed to obtain the speech feature vector.

4. A device for multimodal sentiment analysis, characterized in that, The device includes: The vector extraction module is configured to extract the text feature vector corresponding to the text to be analyzed, wherein the text to be analyzed is obtained by converting the speech to be analyzed; The vector fusion module is configured to fuse the text feature vector with the speech feature vector of the speech to be analyzed to obtain a fused vector; The sentiment analysis module is configured to input the fusion vector into a multimodal sentiment analysis model, and perform sentiment analysis and prediction on the fusion vector through the multimodal sentiment analysis model to obtain the probability of at least one sentiment label of the speech to be analyzed. The sentiment analysis module is further configured to: acquire an EEG conversion file corresponding to the speech to be analyzed, wherein the EEG conversion file is EEG data of relevant personnel collected during the generation of the speech to be analyzed; verify the probability of the at least one sentiment label based on the EEG conversion file to obtain the target probability of the at least one sentiment label; and confirm the target sentiment label of the speech to be analyzed through the target probability. Alternatively, the sentiment analysis module is further configured to: acquire an eye-tracking conversion file corresponding to the speech to be analyzed, wherein the eye-tracking conversion file is eye movement data of relevant personnel collected during the generation of the speech to be analyzed; verify the probability corresponding to the at least one sentiment label based on the eye-tracking conversion file to obtain the target probability corresponding to the at least one sentiment label; and confirm the target sentiment label of the speech to be analyzed through the target probability.

5. The apparatus according to claim 4, characterized in that, The vector fusion module is also configured to: The text feature vector and the speech feature vector are concatenated to obtain a fused vector, wherein the concatenation method includes at least adding the text feature vector and the speech feature vector.

6. The apparatus according to any one of claims 4-5, characterized in that, The vector extraction module is further configured to: The speech to be analyzed is input into the speech extraction model, and the speech extraction model performs convolution and pooling operations on the speech to be analyzed to obtain the speech feature vector.

7. An electronic device, characterized in that, include: Processor, memory, and bus; The processor is connected to the memory via the bus, and the memory stores a computer program that, when executed by the processor, can implement the method as described in any one of claims 1-3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method as described in any one of claims 1-3.

Images