Emotion analysis method and device, equipment, medium and program product

Abstract

The invention provides an emotion analysis method which can be applied to the technical fields of financial science and technology and artificial intelligence. The sentiment analysis method comprises the steps of determining multi-modal dialogue data of a user based on dialogue voice data of the user under the condition that user permission is obtained; based on the multi-modal dialogue data, determining multi-modal input data used for sentiment analysis, the multi-modal input data being obtained by deleting a previous preset number of statements of the multi-modal dialogue data and then reordering the remaining dialogue data according to dialogue weights; the dialogue weight is determined according to the similarity between every two dialogue text sub-data in the remaining dialogue data; and inputting the multi-modal input data into the sentiment analysis model to obtain sentiment classification of the user. The invention further provides an emotion analysis device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the fields of financial technology and artificial intelligence technology, specifically to the field of natural language processing, and more specifically to a sentiment analysis method, device, equipment, medium and program product. Background Art

[0002] With the rapid development of artificial intelligence technology, the application of natural language processing technology in the financial field is becoming more and more extensive. In the financial field, customer service and marketing have become important means for banks to improve customer satisfaction and achieve performance growth.

[0003] However, traditional banking marketing activities often rely on manual operations and lack data analysis and user sentiment tracking, which makes it difficult for marketers to accurately focus on key information in user feedback, resulting in poor sentiment analysis results and poor user experience. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a sentiment analysis method, apparatus, device, medium and program product.

[0005] According to a first aspect of the present disclosure, a sentiment analysis method is provided, comprising: with the permission of the user, determining the multimodal conversation data of the user based on the conversation voice data of the user, wherein the user's conversation includes a plurality of sentences, the conversation voice data includes conversation voice sub-data corresponding to the plurality of sentences respectively, and the multimodal conversation data is determined based on the conversation voice data and conversation text data corresponding to the conversation voice data; determining multimodal input data for sentiment analysis based on the multimodal conversation data, wherein the multimodal input data is obtained by deleting a first preset number of sentences of the multimodal conversation data and reordering the remaining conversation data according to conversation weights, and the conversation weights are determined based on the similarity between every two conversation text sub-data in the remaining conversation data; and inputting the multimodal input data into a sentiment analysis model to obtain the user's sentiment classification.

[0006] According to an embodiment of the present disclosure, based on multimodal conversation data, multimodal input data for sentiment analysis is determined, including: determining a preset number based on the number of multiple sentences in the multimodal conversation data; determining the remaining conversation data based on the preset number and the multimodal conversation data; determining the similarity between every two conversation text sub-data in the conversation text data corresponding to the remaining conversation data, wherein the conversation text sub-data is determined based on the conversation voice sub-data, and the similarity includes cosine similarity and element similarity, and the element similarity is determined based on the number of identical elements in the conversation text sub-data; determining the conversation weight of each conversation text sub-data in the conversation text sub-data corresponding to the remaining conversation data based on the similarity; and reordering the multiple conversation text sub-data in the remaining conversation data and the conversation voice sub-data corresponding to the conversation text sub-data respectively based on the conversation weight to obtain the multimodal input data.

[0007] According to an embodiment of the present disclosure, determining the similarity between every two dialogue text sub-data in the dialogue text data corresponding to the remaining dialogue data includes: respectively determining the semantic coding of the dialogue text sub-data corresponding to each sentence; for each dialogue text sub-data, determining the element similarity between the dialogue text sub-data and the dialogue text sub-data corresponding to the multiple other sentences in the dialogue based on the total number of elements and the number of identical elements in the dialogue text sub-data corresponding to the dialogue text sub-data and the multiple other sentences, wherein the elements include vocabulary; determining the cosine similarity between the dialogue text sub-data and the dialogue text sub-data corresponding to the multiple other sentences based on the semantic coding of the dialogue text sub-data and the semantic coding corresponding to the multiple other sentences; and determining the similarity between the dialogue text sub-data and the dialogue text sub-data corresponding to the multiple other sentences based on multiple element similarities and multiple cosine similarities.

[0008] According to an embodiment of the present disclosure, based on the conversation weights, multiple conversation text sub-data in the remaining conversation data and conversation voice sub-data respectively corresponding to the conversation text sub-data are reordered to obtain multimodal input data, including: based on the conversation weights of the multiple conversation text sub-data respectively, the multiple conversation weights are sorted according to the size relationship to obtain the arranged conversation weights; according to the arranged conversation weights, the multiple conversation text sub-data and the conversation voice sub-data respectively corresponding to the conversation text sub-data are reordered so that the sorted multiple conversation text sub-data correspond to the arranged conversation weights; and based on the reordered multiple conversation text sub-data and the multiple conversation voice sub-data, the multimodal input data is determined.

[0009] According to an embodiment of the present disclosure, the sentiment analysis model includes a discrete wavelet transform module, a multi-head attention mechanism module, a bidirectional long short-term memory network module and a feature fusion module; the multimodal input data is input into the sentiment analysis model to obtain the user's sentiment classification, including: inputting the multimodal input data into the discrete wavelet transform module to obtain low-frequency feature data and high-frequency feature data; reconstructing the low-frequency feature data to obtain reconstructed low-frequency feature data; inputting the reconstructed low-frequency feature data into the multi-head attention mechanism module to obtain the attention feature of the multimodal input data; inputting the reconstructed low-frequency feature data and the high-frequency feature data into the bidirectional long short-term memory network module to obtain the time series feature of the multimodal input data; inputting the attention feature and the time series feature into the feature fusion module to obtain the fusion feature; and obtaining the user's sentiment classification based on the fusion feature.

[0010] According to an embodiment of the present disclosure, based on the user's conversation voice data, determining the user's multimodal conversation data includes: based on the conversation voice data, determining the conversation text data corresponding to the conversation voice data; and based on the conversation voice data and the conversation text data, determining the multimodal conversation data.

[0011] According to an embodiment of the present disclosure, a sentiment analysis model is trained in the following manner: based on sample conversation voice data of a sample conversation, sample multimodal conversation data is determined, wherein the sample conversation includes a plurality of sample sentences, the sample conversation voice data includes sample conversation voice sub-data corresponding to the plurality of sample data respectively, and the sample multimodal conversation data is determined based on the sample conversation voice data and the sample conversation text data corresponding to the sample conversation voice data; based on the sample multimodal conversation data, sample multimodal intermediate data for sentiment analysis is determined, wherein the sample multimodal intermediate data is obtained by deleting a first preset number of sample sentences of the sample multimodal conversation data and sorting the remaining sample conversation data according to the sample conversation. The weights of the sample conversations are obtained by reordering the weights, and the sample conversation weights are determined according to the similarity between every two sample conversation text sub-data in the remaining sample conversation data; the sample multimodal intermediate data are randomly masked to obtain sample multimodal input data; the sample multimodal input data are input into the initial sentiment analysis model to obtain the predicted sentiment classification of the sample conversation; when the predicted sentiment classification does not match the sentiment classification label corresponding to the sample conversation, the model parameters of the initial sentiment analysis model are adjusted based on the predicted sentiment classification and the sentiment classification label by using the projected gradient descent method; and when the predicted sentiment classification matches the sentiment classification label, the sentiment analysis model is determined based on the current model parameters.

[0012] A second aspect of the present disclosure provides a sentiment analysis device, comprising: a dialogue determination module, for determining a user's multimodal dialogue data based on the user's dialogue voice data, wherein the user's dialogue includes multiple sentences, the dialogue voice data includes dialogue voice sub-data corresponding to the multiple sentences respectively, and the multimodal dialogue data is determined based on the dialogue voice data and dialogue text data corresponding to the dialogue voice data; an input determination module, for determining multimodal input data for sentiment analysis based on the multimodal dialogue data, wherein the multimodal input data is obtained by deleting a first preset number of sentences in the multimodal dialogue data and reordering the remaining dialogue data according to dialogue weights, and the dialogue weights are determined based on the similarity between every two dialogue text sub-data in the remaining dialogue data; and a sentiment classification module, for inputting the multimodal input data into a sentiment analysis model to obtain the user's sentiment classification.

[0013] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0014] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0015] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0016] According to the embodiments of the present disclosure, by constructing multimodal data, the modality of user conversation data is enriched, so that different emphases can be provided on the same data, enriching data features for better feature extraction, and by preprocessing the multimodal data, the user's scattered emotional expressions can be concentrated, so as to better analyze the user's emotions and improve the accuracy of emotional analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0018] Figure 1 The application scenario diagram of the sentiment analysis method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown;

[0019] Figure 2 The flowchart of the sentiment analysis method according to the embodiment of the present disclosure is schematically shown;

[0020] Figure 3 Schematically shows a data flow diagram of sentiment analysis according to an embodiment of the present disclosure;

[0021] Figure 4 A structural block diagram of a sentiment analysis device according to an embodiment of the present disclosure is schematically shown; and

[0022] Figure 5 A block diagram of an electronic device suitable for implementing a sentiment analysis method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0024] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0025] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0027] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0028] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0029] An embodiment of the present disclosure provides a sentiment analysis method, comprising: with the permission of the user, determining the user's multimodal conversation data based on the user's conversation voice data, wherein the user's conversation includes multiple sentences, the conversation voice data includes conversation voice sub-data corresponding to the multiple sentences respectively, and the multimodal conversation data is determined based on the conversation voice data and conversation text data corresponding to the conversation voice data; based on the multimodal conversation data, determining multimodal input data for sentiment analysis, wherein the multimodal input data is obtained by deleting a first preset number of sentences in the multimodal conversation data and reordering the remaining conversation data according to conversation weights, and the conversation weights are determined based on the similarity between every two conversation text sub-data in the remaining conversation data; and inputting the multimodal input data into a sentiment analysis model to obtain the user's sentiment classification.

[0030] Figure 1 The application scenario diagram of the sentiment analysis method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown.

[0031] like Figure 1As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0032] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0033] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0034] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0035] It should be noted that the sentiment analysis method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the sentiment analysis device provided in the embodiment of the present disclosure can generally be set in the server 105. The sentiment analysis method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the sentiment analysis device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0036] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0037] The following will be based on Figure 1 The scene described by Figure 2~Figure 5 The sentiment analysis method of the disclosed embodiment is described in detail.

[0038] Figure 2 The flowchart of the sentiment analysis method according to the embodiment of the present disclosure is schematically shown.

[0039] like Figure 2 As shown, the sentiment analysis method of this embodiment includes operations S210 to S230.

[0040] In operation S210, with the permission of the user, multimodal conversation data of the user is determined based on the conversation voice data of the user.

[0041] According to the embodiments of the present disclosure, during the voice communication with the user, it is necessary to analyze the user's conversation voice data to determine the user's emotions, so as to select appropriate words according to the user's emotions for more effective communication. Before obtaining the user's conversation voice data, the user can be prompted by voice to determine whether the user allows the acquisition of his or her conversation voice data. If the user allows the data acquisition, the user's conversation voice data is analyzed using the sentiment analysis method.

[0042] According to an embodiment of the present disclosure, a user's conversation includes multiple sentences, the conversation voice data includes conversation voice sub-data corresponding to the multiple sentences respectively, and the multimodal conversation data is determined based on the conversation voice data and conversation text data corresponding to the conversation voice data, including at least two modes: voice and text.

[0043] In operation S220 , multimodal input data for sentiment analysis is determined based on the multimodal conversation data.

[0044] According to an embodiment of the present disclosure, multimodal input data is obtained by deleting the first preset number of sentences of the multimodal dialogue data and reordering the remaining dialogue data according to dialogue weights, wherein, when performing user sentiment analysis based on historical dialogues of multiple users, it is found that the first few sentences in the user's historical dialogues usually do not have a strong emotional tendency. Therefore, when performing sentiment analysis, a preset number can be determined based on the experience of sentiment analysis of historical dialogues, and the first preset number of sentences of the multimodal dialogue data can be deleted, and the other sentences are retained as the remaining dialogue sentences. The dialogue weight is determined based on the similarity between each two dialogue text sub-data in the remaining dialogue data.

[0045] In operation S230, the multimodal input data is input into a sentiment analysis model to obtain a sentiment classification of the user.

[0046] According to the embodiments of the present disclosure, by constructing multimodal data, the modality of user conversation data is enriched, so that different emphases can be provided on the same data, enriching data features for better feature extraction, and by preprocessing the multimodal data, the user's scattered emotional expressions can be concentrated, so as to better analyze the user's emotions and improve the accuracy of emotional analysis.

[0047] According to an embodiment of the present disclosure, based on multimodal conversation data, multimodal input data for sentiment analysis is determined, including: determining a preset number based on the number of multiple sentences in the multimodal conversation data; determining the remaining conversation data based on the preset number and the multimodal conversation data; determining the similarity between every two conversation text sub-data in the conversation text data corresponding to the remaining conversation data, wherein the conversation text sub-data is determined based on the conversation voice sub-data, and the similarity includes cosine similarity and element similarity, and the element similarity is determined based on the number of identical elements in the conversation text sub-data; determining the conversation weight of each conversation text sub-data in the conversation text sub-data corresponding to the remaining conversation data based on the similarity; and reordering the multiple conversation text sub-data in the remaining conversation data and the conversation voice sub-data corresponding to the conversation text sub-data respectively based on the conversation weight to obtain the multimodal input data.

[0048] According to an embodiment of the present disclosure, when the user's conversation is long, the number of sentences without strong emotional tendencies in the conversation may be greater, so the preset number can be determined according to the number of sentences in the multimodal conversation data, and preferably, the preset number can be 2. The first preset number of sentences in the multimodal conversation data are deleted to obtain the remaining conversation data.

[0049] According to an embodiment of the present disclosure, the cosine similarity and element similarity between every two dialogue text sub-data in the remaining dialogue data are calculated respectively, and the dialogue weight of each dialogue text sub-data is further determined. The multiple dialogue text sub-data in the remaining dialogue data are sorted in descending order according to the dialogue weight, and are respectively combined with the corresponding dialogue speech sub-data to obtain multimodal input data.

[0050] According to an embodiment of the present disclosure, after the reordering is completed, the conversation voice sub-data and conversation text sub-data corresponding to the preset number of sentences may be added to the end of the reordered data to obtain multimodal input data.

[0051] According to the embodiments of the present disclosure, different similarity calculation methods are used to more comprehensively judge the similarity between every two sentences, thereby improving the accuracy of sentence similarity judgment through two methods: the same words and similar semantics, thereby improving the reordering effect.

[0052] According to an embodiment of the present disclosure, determining the similarity between every two dialogue text sub-data in the dialogue text data corresponding to the remaining dialogue data includes: respectively determining the semantic coding of the dialogue text sub-data corresponding to each sentence; for each dialogue text sub-data, determining the element similarity between the dialogue text sub-data and the dialogue text sub-data corresponding to the multiple other sentences in the dialogue based on the total number of elements and the number of identical elements in the dialogue text sub-data corresponding to the dialogue text sub-data and the multiple other sentences, wherein the elements include vocabulary; determining the cosine similarity between the dialogue text sub-data and the dialogue text sub-data corresponding to the multiple other sentences based on the semantic coding of the dialogue text sub-data and the semantic coding corresponding to the multiple other sentences; and determining the similarity between the dialogue text sub-data and the dialogue text sub-data corresponding to the multiple other sentences based on multiple element similarities and multiple cosine similarities.

[0053] According to an embodiment of the present disclosure, for each dialogue text sub-data, the element similarity between the current dialogue text sub-data and the dialogue text sub-data is determined according to the total number of elements in the current dialogue text sub-data, the total number of elements in other dialogue text sub-data for which similarity needs to be calculated, and the number of identical elements between the current dialogue text sub-data and the dialogue text sub-data for which similarity needs to be calculated, as shown in formula (1):

[0054] (1)

[0055] Where i is the ranking of the current dialogue text sub-data in the multiple sentences in the multimodal input data, j is the ranking of the dialogue text sub-data with similarity calculated with the current dialogue text sub-data in the multiple sentences in the multimodal input data, α is the scaling factor, Count(t i , t j ) is the number of identical elements in the i-th and j-th dialogue text sub-data, |t i | is the total number of elements in the ith dialogue text sub-data, |t j | is the total number of elements in the j-th dialogue text sub-data.

[0056] According to an embodiment of the present disclosure, the BERT model is used to determine the embedded representation of the dialogue text sub-data corresponding to each sentence, and then the embedded representation is further processed using a temporal convolutional network to capture the temporal dependency in the sentence and obtain the semantic encoding of the sentence, wherein the dilation convolution process in the temporal convolutional network is shown in formula (2):

[0057] (2)

[0058] Among them, K' is the size of the expanded convolution kernel, K is the original convolution kernel size, and R is the expansion rate. Multiple expanded convolutions can be used to perform convolution operations on the embedded representation of the text sub-data, and after residual connection, multiple outputs are spliced ​​together to obtain the semantic encoding of the dialogue text sub-data.

[0059] According to an embodiment of the present disclosure, for each dialogue text sub-data, the cosine similarity between the semantic encodings of the current dialogue text sub-data and other dialogue text sub-data is calculated, as shown in formula (3):

[0060] (3)

[0061] Wherein, β is a scaling factor, preferably, β=1-α, Ei and Ej are the semantic encodings of the i-th sentence and the j-th sentence determined by the BERT model and the dilated convolution, respectively, and |·| is the length of the vector.

[0062] According to an embodiment of the present disclosure, after determining the element similarity and cosine similarity between the current dialogue text sub-data and all other dialogue text sub-data, the similarity between the dialogue text sub-data and other multiple dialogue text sub-data can be determined using formula (4):

[0063] (4)

[0064] Wherein, n is the number of sentences in the dialogue text data, and k is a preset number.

[0065] According to an embodiment of the present disclosure, after the similarity is determined, the conversation weight of each conversation text sub-data in the conversation text sub-data corresponding to the remaining conversation data can be determined based on the similarity. For each conversation text sub-data, the conversation weight of the conversation text sub-data can be determined by formula (5):

[0066] (5)

[0067] In order to prevent the similarity from being 0, which would result in the denominator being 0, a smaller constant ε is set.

[0068] According to an embodiment of the present disclosure, based on the conversation weights, multiple conversation text sub-data in the remaining conversation data and conversation voice sub-data respectively corresponding to the conversation text sub-data are reordered to obtain multimodal input data, including: based on the conversation weights of the multiple conversation text sub-data respectively, the multiple conversation weights are sorted according to the size relationship to obtain the arranged conversation weights; according to the arranged conversation weights, the multiple conversation text sub-data and the conversation voice sub-data respectively corresponding to the conversation text sub-data are reordered so that the sorted multiple conversation text sub-data correspond to the arranged conversation weights; and based on the reordered multiple conversation text sub-data and the multiple conversation voice sub-data, the multimodal input data is determined.

[0069] According to an embodiment of the present disclosure, the conversation weights of multiple conversation text sub-data are sorted in descending order, and the conversation text sub-data and the conversation voice sub-data corresponding to the conversation text sub-data are re-sorted in the same order as the sorted conversation weights to determine the multimodal input data.

[0070] According to an embodiment of the present disclosure, by reordering, the order of sentences with higher dialogue weights, that is, sentences with stronger user emotion expression, can be adjusted to a front position, thereby improving the accuracy of sentiment analysis based on multimodal input data.

[0071] According to an embodiment of the present disclosure, the sentiment analysis model includes a discrete wavelet transform module, a multi-head attention mechanism module, a bidirectional long short-term memory network module and a feature fusion module; the multimodal input data is input into the sentiment analysis model to obtain the user's sentiment classification, including: inputting the multimodal input data into the discrete wavelet transform module to obtain low-frequency feature data and high-frequency feature data; reconstructing the low-frequency feature data to obtain reconstructed low-frequency feature data; inputting the reconstructed low-frequency feature data into the multi-head attention mechanism module to obtain the attention feature of the multimodal input data; inputting the reconstructed low-frequency feature data and the high-frequency feature data into the bidirectional long short-term memory network module to obtain the time series feature of the multimodal input data; inputting the attention feature and the time series feature into the feature fusion module to obtain the fusion feature; and obtaining the user's sentiment classification based on the fusion feature.

[0072] According to the embodiments of the present disclosure, the Discrete Wavelet Transform (DWT) is a multi-resolution analysis based on the continuous wavelet change. It generally adopts the method of hashing the time shift factor τ to the positive integer domain, that is, the signal sampling frequency, and binary hashing the scale factor s. The discrete binary wavelet basis function is converted into two mutually orthogonal filters to achieve the decomposition of the input signal into multiple groups of high-frequency channels and low-frequency channels, and the step-by-step discrete wavelet transform formula is obtained. The calculation of the i-th level low-frequency information and high-frequency information of the DWT is shown in formulas (6) and (7), respectively:

[0073] (6)

[0074] (7)

[0075] Where i is the number of layers, t is the time information of discrete wavelet transform, T is the maximum value of the time information, n is the recursive term index, g[·] is the low-frequency filter, and h[·] is the high-frequency filter.

[0076] According to the embodiments of the present disclosure, when the signal is decomposed, the scale factor is halved for each level up, the time resolution is improved, and the frequency resolution is reduced. At the same time, the DWT can reconstruct the signal through inverse transformation. The process of reconstructing the signal can be represented by Inverse Discrete Wavelet Transform (IDWT). The IDWT process from level i+1 to level i is shown in formula (8):

[0077] (8)

[0078] According to an embodiment of the present disclosure, in the process of signal reconstruction using IDWT, the influence of low-frequency feature data on sentiment polarity can be highlighted, thereby obtaining reconstructed low-frequency feature data to improve the interpretability and scalability of sentiment analysis.

[0079] According to the embodiments of the present disclosure, in addition to implementing DWT and IDWT, the filter may also include a feature transformation layer. After extracting the word vector features in the time domain to a more complex time-frequency domain through DWT, the features may be reorganized in the feature transformation layer, and then the features may be restored to the word vector space in the time domain using IDWT. In this process, feature learning may be performed by updating the feature parameters of the feature transformation layer. The process of the feature transformation layer is shown in formulas (9) and (10):

[0080] (9)

[0081] (10)

[0082] Where c is the scale factor, m is the index value under the scale factor, and x[c,·] is the original signal at scale c.

[0083] According to an embodiment of the present disclosure, the reconstructed low-frequency feature data is concatenated by linear layer transformation to obtain Q vector, K vector and V vector that can represent the relationship between words, and the above vectors are input into the multi-head attention mechanism module to obtain the attention features of the multimodal input data, as shown in formula (11):

[0084] (11)

[0085] in, is the dimension of the K vector, preferably, You can take 64.

[0086] According to an embodiment of the present disclosure, a single attention mechanism matrix can be obtained by matrix operation, and then a matrix representation of a multi-head attention mechanism can be obtained by continuous splicing, and then a forward propagation operation is performed to reduce the dimension of the spliced ​​matrix representation, as shown in formula (12):

[0087] (12)

[0088] Among them, W C and W M is the weight matrix parameter, B C and B M is the bias parameter, H i The calculation of is shown in formula (13):

[0089] (13)

[0090] Where f(·) is the attention mechanism calculation as shown in formula (11), W Ki , W Qi , W Vi are the weight parameters of the i-th attention mechanism matrix respectively.

[0091] According to an embodiment of the present disclosure, the reconstructed low-frequency feature data and high-frequency feature data are input into a bidirectional long short-term memory network module to obtain a time series feature H-LSTM of the multimodal input data.

[0092] According to an embodiment of the present disclosure, the attention feature Multi-H and the temporal feature H-LSTM are input into the feature fusion module and concatenated to obtain the fusion feature, as shown in formula (14):

[0093] (14)

[0094] According to the embodiments of the present disclosure, the sentiment analysis model may further include a feature mining module. After obtaining the fusion feature, the feature mining module may be used to further mine the fusion feature. When the input time series is X and the shape of Fusion-MH is (B, T, C), N convolution kernels of different sizes are defined, and the size of each convolution kernel is k. n , where n=1, 2, ..., N, B is the batch size, T is the length of the time series, C is the feature dimension, and the initial weight of each convolution kernel is W n , the shape is (k n , C, C'), with a bias of b n , the shape is (C', 1), where C' is the feature dimension of the fusion feature after convolution. For the nth convolution kernel, the output Y of the convolution kernel is n It can be expressed as formula (15):

[0095] (15)

[0096] Among them, Y n The shape is (B, Tk n +1, C'), i is the data subscript dimension, j is the subscript of the convolution kernel at a certain position, W n is the parameter weight, b n is the bias. Causal convolution is used in the construction of the convolution kernel. Since the current time step and the previous time step are taken into account, the output length will be reduced by k. n -1. After obtaining N convolution kernels, the N convolution kernels are concatenated, as shown in formula (16):

[0097] (16)

[0098] Where axis=2 means vector concatenation is performed in the second dimension. The concatenation result Y all The dimension is (T-max(k n )+1,N×C').

[0099] According to an embodiment of the present disclosure, the feature after the temporal convolution is enhanced by using the enhanced attention mechanism, and the Y output by the convolution can be all Perform feature encoding and output the attention score of each feature, as shown in formula (17):

[0100] (17)

[0101] Among them, F is Y all The shape of the flattened two-dimensional matrix F is (T-max(k n )+1,N×C'),Wa is the weight of the fully connected layer, b a is the bias of the fully connected layer, Score is the attention score matrix, and the shape of Score is (T-max(k n )+1,1). By calculating the attention score, we can get the attention-weighted output Z en , as shown in formula (18):

[0102] (18)

[0103] Among them, Score[i,:] is the attention score matrix of the i-th element calculated by the scoring function.

[0104] According to an embodiment of the present disclosure, after obtaining the attention-weighted output, a linear layer can be used to perform calculations such as formula (19) to determine the user's sentiment classification score:

[0105] (19)

[0106] Among them, W M is the weight of the linear layer, b M is the bias of the linear layer.

[0107] According to an embodiment of the present disclosure, the softmax function can be used to process the user's emotion classification score to obtain the user's emotion classification Label-out, as shown in formula (20):

[0108] (20)

[0109] Among them, W L is the weight of the classifier, b L is the bias of the classifier. User sentiment classification can be a multi-classification task, for example, including five categories: negative, slightly negative, neutral, slightly positive, and positive.

[0110] According to the embodiments of the present disclosure, by performing feature extraction in different dimensions, the expression angle of the feature can be improved, thereby making the feature more accurate.

[0111] According to an embodiment of the present disclosure, based on the user's conversation voice data, determining the user's multimodal conversation data includes: based on the conversation voice data, determining the conversation text data corresponding to the conversation voice data; and based on the conversation voice data and the conversation text data, determining the multimodal conversation data.

[0112] According to an embodiment of the present disclosure, a text-to-speech conversion method is used to convert conversation voice data into corresponding conversation text data, and the conversation voice data is combined with the conversation text data to determine multimodal conversation data.

[0113] According to the embodiments of the present disclosure, by determining the corresponding conversation text data through the conversation voice data and forming multimodal conversation data, the feature expression of the input data can be enriched, so that more accurate and comprehensive features can be obtained in the subsequent feature extraction, so as to improve the accuracy of sentiment analysis.

[0114] Figure 3 The data flow diagram of sentiment analysis according to an embodiment of the present disclosure is schematically shown.

[0115] like Figure 3 As shown, with the user's permission, the user's conversation voice data is obtained and converted into conversation text data, the conversation voice data and the conversation text data are combined to obtain multimodal conversation data, the multimodal conversation data is preprocessed to obtain multimodal input data, and the multimodal input data is input into the sentiment analysis model.

[0116] The sentiment analysis model uses the discrete wavelet transform module to process the multimodal input data to obtain low-frequency feature data and high-frequency feature data, further reconstructs the low-frequency feature data to obtain reconstructed low-frequency feature data, inputs the reconstructed low-frequency data into the multi-head attention mechanism module to obtain attention features, and inputs the reconstructed low-frequency feature data and high-frequency feature data into the long short-term memory network module to obtain time series features. The attention features and time series features are respectively input into the feature fusion module to obtain fusion features, and the fusion features are further mined using the feature mining module. The processed fusion features are input into the linear layer for processing, and the data output by the linear layer is input into the classifier to obtain the sentiment classification results for the dialogue speech data.

[0117] According to an embodiment of the present disclosure, a sentiment analysis model is trained in the following manner: based on sample conversation voice data of a sample conversation, sample multimodal conversation data is determined, wherein the sample conversation includes a plurality of sample sentences, the sample conversation voice data includes sample conversation voice sub-data corresponding to the plurality of sample data respectively, and the sample multimodal conversation data is determined based on the sample conversation voice data and the sample conversation text data corresponding to the sample conversation voice data; based on the sample multimodal conversation data, sample multimodal intermediate data for sentiment analysis is determined, wherein the sample multimodal intermediate data is obtained by deleting a first preset number of sample sentences of the sample multimodal conversation data and sorting the remaining sample conversation data according to the sample conversation. The weights of the sample conversations are obtained by reordering the weights, and the sample conversation weights are determined according to the similarity between every two sample conversation text sub-data in the remaining sample conversation data; the sample multimodal intermediate data are randomly masked to obtain sample multimodal input data; the sample multimodal input data are input into the initial sentiment analysis model to obtain the predicted sentiment classification of the sample conversation; when the predicted sentiment classification does not match the sentiment classification label corresponding to the sample conversation, the model parameters of the initial sentiment analysis model are adjusted based on the predicted sentiment classification and the sentiment classification label by using the projected gradient descent method; and when the predicted sentiment classification matches the sentiment classification label, the sentiment analysis model is determined based on the current model parameters.

[0118] According to an embodiment of the present disclosure, a random masking operation can be performed by a masked autoencoder (MAE). For example, 15% of the sentences can be randomly selected from the sample multimodal intermediate data, 80% of which are represented by special characters, 10% are replaced by other characters, and 10% remain unchanged.

[0119] According to an embodiment of the present disclosure, Projected Gradient Descent (PGD) is an adversarial training method that adds perturbations, which is used to generate adversarial samples for text and voice bimodal data, so that the model can obtain higher robustness and transfer learning ability during the training process. The PGD adversarial learning method searches for better adversarial samples through multiple small-step iterations. In each iteration, PGD updates the adversarial samples based on the current gradient information and limits the size of the perturbation to a certain range.

[0120] According to the embodiments of the present disclosure, when using PGD for adversarial training, it is necessary to calculate the forward loss of the current model on the input data and obtain the gradient through back propagation. The specific steps are: initialize the adversarial sample as the input data; for each iteration, calculate the perturbation according to the current gradient and update the adversarial sample; if the perturbation exceeds the allowed range, project it back to the space within the allowed range. The a+1th iteration operation process is shown in formula (21):

[0121] (twenty one)

[0122] in, It means that the adversarial sample is projected into the allowed range x+S, S is the state space, sgn(·) is the sign function, L(θ, x, y) is the loss function used to optimize the initial sentiment analysis model, θ is the model parameter of the initial sentiment analysis model, x is the sample multimodal input data, y is the label information of the sample multimodal input data, and lr is the learning rate.

[0123] According to an embodiment of the present disclosure, the adversarial loss value L is calculated using the generated adversarial sample adv (θ,x adv ,y), and update the model parameters based on the model loss and adversarial loss value of the initial sentiment analysis model. The total loss of the model L total (θ,x,x adv ,y) as shown in formula (22):

[0124] (twenty two)

[0125] Among them, λ is the weight coefficient, which can be used to balance the model loss and adversarial loss. After determining the total loss, the model parameters can be updated using gradient descent, as shown in formula (23):

[0126] (twenty three)

[0127] Where η is the learning rate of the total loss.

[0128] According to the embodiments of the present disclosure, the input data is processed by a masked autoencoder and adversarial loss is introduced in the loss function, so that the feature expression of the data used by the training model is reduced, thereby reducing the model's requirements for input data, and enabling the model to still provide correct classification results when faced with blurred input data. Therefore, the robustness and generalization ability of the trained model when faced with different input data can be improved.

[0129] Based on the above sentiment analysis method, the present disclosure also provides a sentiment analysis device. Figure 4 The device is described in detail.

[0130] Figure 4 The structural block diagram of the sentiment analysis device according to an embodiment of the present disclosure is schematically shown.

[0131] like Figure 4 As shown, the sentiment analysis device 400 of this embodiment includes a dialogue determination module 410, an input determination module 420 and a sentiment classification module 430.

[0132] The dialogue determination module 410 is used to determine the multimodal dialogue data of the user based on the dialogue voice data of the user, wherein the dialogue of the user includes multiple sentences, the dialogue voice data includes dialogue voice sub-data corresponding to the multiple sentences respectively, and the multimodal dialogue data is determined according to the dialogue voice data and the dialogue text data corresponding to the dialogue voice data. In one embodiment, the dialogue determination module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0133] The input determination module 420 is used to determine the multimodal input data for sentiment analysis based on the multimodal dialogue data, wherein the multimodal input data is obtained by deleting the first preset number of sentences of the multimodal dialogue data and reordering the remaining dialogue data according to the dialogue weight, and the dialogue weight is determined according to the similarity between each two dialogue text sub-data in the remaining dialogue data. In one embodiment, the input determination module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0134] The sentiment classification module 430 is used to input the multimodal input data into the sentiment analysis model to obtain the sentiment classification of the user. In one embodiment, the sentiment classification module 430 can be used to perform the operation S230 described above, which will not be repeated here.

[0135] According to an embodiment of the present disclosure, the input determination module 420 includes a quantity determination submodule, a data determination submodule, a similarity determination submodule, a weight determination submodule and a data sorting submodule.

[0136] The quantity determination submodule is used to determine a preset quantity based on the quantity of multiple sentences in the multimodal dialogue data.

[0137] The data determination submodule is used to determine the remaining dialogue data based on a preset quantity and the multimodal dialogue data.

[0138] The similarity determination submodule is used to determine the similarity between every two dialogue text sub-data in the dialogue text data corresponding to the remaining dialogue data, wherein the dialogue text sub-data is determined based on the dialogue speech sub-data, and the similarity includes cosine similarity and element similarity, and the element similarity is determined based on the number of identical elements in the dialogue text sub-data.

[0139] The weight determination submodule is used to determine the conversation weight of each conversation text sub-data in the conversation text sub-data corresponding to the remaining conversation data based on the similarity.

[0140] The data sorting submodule is used to reorder the multiple dialogue text sub-data and the dialogue voice sub-data respectively corresponding to the dialogue text sub-data in the remaining dialogue data based on the dialogue weights to obtain multimodal input data.

[0141] According to an embodiment of the present disclosure, the similarity determination submodule includes an encoding determination unit, a first similarity determination unit, a second similarity determination unit, and a similarity determination unit.

[0142] The coding determination unit is used to determine the semantic coding of the dialogue text sub-data corresponding to each sentence respectively.

[0143] The first similarity determination unit is used to determine, for each dialogue text sub-data, the element similarity between the dialogue text sub-data and the dialogue text sub-data corresponding to multiple other sentences in the dialogue based on the total number of elements and the number of identical elements in the dialogue text sub-data corresponding to the dialogue text sub-data and the multiple other sentences, wherein the elements include vocabulary.

[0144] The second similarity determination unit is used to determine the cosine similarity of the dialogue text sub-data and the dialogue text sub-data corresponding to the other sentences based on the semantic coding of the dialogue text sub-data and the semantic coding corresponding to the other sentences.

[0145] The similarity determination unit is used to determine the similarity between the dialogue text sub-data and the dialogue text sub-data corresponding to multiple other sentences based on multiple element similarities and multiple cosine similarities.

[0146] According to an embodiment of the present disclosure, the data sorting submodule includes a weight sorting unit, a data sorting unit and a data determining unit.

[0147] The weight sorting unit is used to sort the multiple dialogue weights according to the size relationship based on the dialogue weights of the multiple dialogue text sub-data, so as to obtain the arranged dialogue weights.

[0148] The data sorting unit is used to reorder the multiple dialogue text sub-data and the dialogue voice sub-data corresponding to the dialogue text sub-data according to the arranged dialogue weights, so that the sorted multiple dialogue text sub-data correspond to the arranged dialogue weights.

[0149] The data determination unit is used to determine the multimodal input data based on the reordered multiple dialogue text sub-data and the multiple dialogue voice sub-data.

[0150] According to an embodiment of the present disclosure, the sentiment classification module 430 includes a discrete wavelet transform submodule, a data reconstruction submodule, a first data input submodule, a second data input submodule, a feature fusion submodule and a sentiment classification submodule.

[0151] The discrete wavelet transform submodule is used to input the multimodal input data into the discrete wavelet transform module to obtain low-frequency feature data and high-frequency feature data.

[0152] The data reconstruction submodule is used to reconstruct the low-frequency feature data to obtain reconstructed low-frequency feature data.

[0153] The first data input submodule is used to input the reconstructed low-frequency feature data into the multi-head attention mechanism module to obtain the attention features of the multimodal input data.

[0154] The second data input submodule is used to input the reconstructed low-frequency feature data and high-frequency feature data into the bidirectional long short-term memory network module to obtain the time series characteristics of the multimodal input data.

[0155] The feature fusion submodule is used to input the attention feature and the temporal feature into the feature fusion module to obtain the fused feature.

[0156] The sentiment classification submodule is used to obtain the user's sentiment classification based on the fusion features.

[0157] According to an embodiment of the present disclosure, the dialogue determination module 410 includes a text determination submodule and a dialogue determination submodule.

[0158] The text determination submodule is used to determine the dialogue text data corresponding to the dialogue voice data based on the dialogue voice data.

[0159] The dialogue determination submodule is used to determine the multimodal dialogue data based on the dialogue voice data and the dialogue text data.

[0160] According to an embodiment of the present disclosure, the sentiment analysis device 400 also includes a sample dialogue determination module, a sample data determination module, a random mask module, a data input module, a parameter adjustment module and a model determination module.

[0161] A sample conversation determination module is used to determine sample multimodal conversation data based on sample conversation voice data of a sample conversation, wherein the sample conversation includes multiple sample sentences, the sample conversation voice data includes sample conversation voice sub-data corresponding to the multiple sample data respectively, and the sample multimodal conversation data is determined based on the sample conversation voice data and the sample conversation text data corresponding to the sample conversation voice data.

[0162] The sample data determination module is used to determine the sample multimodal intermediate data for sentiment analysis based on the sample multimodal conversation data, wherein the sample multimodal intermediate data is obtained by deleting the first preset number of sample sentences of the sample multimodal conversation data and reordering the remaining sample conversation data according to the sample conversation weights, and the sample conversation weights are determined based on the similarity between every two sample conversation text sub-data in the remaining sample conversation data.

[0163] The random masking module is used to perform a random masking operation on the sample multimodal intermediate data to obtain the sample multimodal input data.

[0164] The data input module is used to input the sample multimodal input data into the initial sentiment analysis model to obtain the predicted sentiment classification of the sample conversation.

[0165] The parameter adjustment module is used to adjust the model parameters of the initial sentiment analysis model by using projected gradient descent based on the predicted sentiment classification and the sentiment classification label when the predicted sentiment classification does not match the sentiment classification label corresponding to the sample conversation.

[0166] The model determination module is used to determine the sentiment analysis model based on the current model parameters when the predicted sentiment classification matches the sentiment classification label.

[0167] According to an embodiment of the present disclosure, any multiple modules of the dialogue determination module 410, the input determination module 420 and the emotion classification module 430 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the dialogue determination module 410, the input determination module 420 and the emotion classification module 430 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the dialogue determination module 410, the input determination module 420 and the emotion classification module 430 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0168] Figure 5 A block diagram of an electronic device suitable for implementing a sentiment analysis method according to an embodiment of the present disclosure is schematically shown.

[0169] like Figure 5 As shown, the electronic device 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0170] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0171] According to an embodiment of the present disclosure, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0172] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0173] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0174] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the sentiment analysis method provided by the embodiment of the present disclosure.

[0175] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 501. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0176] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0177] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0178] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0179] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0180] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0181] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A sentiment analysis method, characterized in that: The method comprises: With the permission of the user, determining the multimodal conversation data of the user based on the conversation voice data of the user, wherein the conversation of the user includes a plurality of sentences, the conversation voice data includes conversation voice sub-data corresponding to the plurality of sentences respectively, and the multimodal conversation data is determined based on the conversation voice data and the conversation text data corresponding to the conversation voice data; Based on the multimodal conversation data, determining multimodal input data for sentiment analysis, wherein the multimodal input data is obtained by deleting a first preset number of sentences of the multimodal conversation data and reordering the remaining conversation data according to conversation weights, wherein the conversation weights are determined according to the similarity between every two conversation text sub-data in the remaining conversation data; and The multimodal input data is input into a sentiment analysis model to obtain the sentiment classification of the user.

2. The method according to claim 1, characterized in that The step of determining multimodal input data for sentiment analysis based on the multimodal dialogue data includes: Determining the preset number based on the number of the plurality of sentences in the multimodal conversation data; Determining the remaining conversation data based on the preset number and the multimodal conversation data; Determine the similarity between every two dialogue text sub-data in the dialogue text data corresponding to the remaining dialogue data, wherein the dialogue text sub-data is determined based on the dialogue speech sub-data, and the similarity includes cosine similarity and element similarity, and the element similarity is determined based on the number of identical elements in the dialogue text sub-data; Based on the similarity, determining a conversation weight of each of the conversation text sub-data in the conversation text sub-data corresponding to the remaining conversation data; and Based on the dialogue weight, the plurality of dialogue text sub-data in the remaining dialogue data and the dialogue voice sub-data respectively corresponding to the dialogue text sub-data are reordered to obtain the multimodal input data.

3. The method according to claim 2, characterized in that The determining of the similarity between each two of the dialogue text sub-data in the dialogue text data corresponding to the remaining dialogue data comprises: respectively determining the semantic coding of the dialogue text sub-data corresponding to each of the sentences; For each of the dialogue text sub-data, Determining the element similarity between the dialogue text sub-data and the dialogue text sub-data corresponding to the other sentences in the dialogue based on the total number of elements and the number of identical elements in the dialogue text sub-data and the dialogue text sub-data corresponding to the other sentences, wherein the elements include vocabulary; Determining the cosine similarity between the dialogue text sub-data and the dialogue text sub-data corresponding to the other sentences based on the semantic coding of the dialogue text sub-data and the semantic coding corresponding to the other sentences; and Based on the plurality of element similarities and the plurality of cosine similarities, similarities between the dialogue text sub-data and the dialogue text sub-data corresponding to the plurality of other sentences are determined.

4. The method according to claim 2, characterized in that: The step of reordering the plurality of dialogue text sub-data and the dialogue voice sub-data respectively corresponding to the dialogue text sub-data in the remaining dialogue data based on the dialogue weight to obtain the multimodal input data includes: Based on the respective conversation weights of the plurality of conversation text sub-data, the plurality of conversation weights are sorted according to size to obtain sorted conversation weights; According to the arranged conversation weights, reordering the plurality of conversation text sub-data and the conversation voice sub-data respectively corresponding to the conversation text sub-data so that the ordered plurality of conversation text sub-data correspond to the arranged conversation weights; and The multimodal input data is determined based on the reordered plurality of dialogue text sub-data and the plurality of dialogue voice sub-data.

5. The method according to claim 1, characterized in that The sentiment analysis model includes a discrete wavelet transform module, a multi-head attention mechanism module, a bidirectional long short-term memory network module and a feature fusion module; The step of inputting the multimodal input data into a sentiment analysis model to obtain the sentiment classification of the user includes: Inputting the multimodal input data into the discrete wavelet transform module to obtain low-frequency feature data and high-frequency feature data; Reconstructing the low-frequency feature data to obtain reconstructed low-frequency feature data; Inputting the reconstructed low-frequency feature data into the multi-head attention mechanism module to obtain the attention features of the multimodal input data; Inputting the reconstructed low-frequency feature data and the high-frequency feature data into the bidirectional long short-term memory network module to obtain the time series features of the multimodal input data; Inputting the attention feature and the temporal feature into the feature fusion module to obtain a fusion feature; and Based on the fused features, a sentiment classification of the user is obtained.

6. The method according to claim 1, characterized in that The determining, based on the conversation voice data of the user, the multimodal conversation data of the user comprises: Based on the conversation voice data, determining the conversation text data corresponding to the conversation voice data; and The multimodal conversation data is determined based on the conversation voice data and the conversation text data.

7. The method according to claim 1, characterized in that The sentiment analysis model is trained in the following way: Determining sample multimodal dialogue data based on sample dialogue voice data of a sample dialogue, wherein the sample dialogue includes a plurality of sample sentences, the sample dialogue voice data includes sample dialogue voice sub-data corresponding to the plurality of sample data respectively, and the sample multimodal dialogue data is determined based on the sample dialogue voice data and sample dialogue text data corresponding to the sample dialogue voice data; Based on the sample multimodal conversation data, determine sample multimodal intermediate data for sentiment analysis, wherein the sample multimodal intermediate data is obtained by deleting a first preset number of sample sentences of the sample multimodal conversation data and reordering the remaining sample conversation data according to sample conversation weights, wherein the sample conversation weights are determined according to the similarity between every two sample conversation text sub-data in the remaining sample conversation data; Performing a random masking operation on the sample multimodal intermediate data to obtain sample multimodal input data; Inputting the sample multimodal input data into an initial sentiment analysis model to obtain a predicted sentiment classification of the sample conversation; In the case where the predicted sentiment classification does not match the sentiment classification label corresponding to the sample conversation, adjusting the model parameters of the initial sentiment analysis model by projected gradient descent based on the predicted sentiment classification and the sentiment classification label; and In a case where the predicted sentiment classification matches the sentiment classification label, the sentiment analysis model is determined based on current model parameters.

8. A sentiment analysis device, characterized in that: The device comprises: a dialogue determination module, configured to determine multimodal dialogue data of the user based on dialogue voice data of the user, wherein the dialogue of the user includes a plurality of sentences, the dialogue voice data includes dialogue voice sub-data corresponding to the plurality of sentences respectively, and the multimodal dialogue data is determined based on the dialogue voice data and dialogue text data corresponding to the dialogue voice data; an input determination module, configured to determine multimodal input data for sentiment analysis based on the multimodal dialogue data, wherein the multimodal input data is obtained by deleting a first preset number of sentences of the multimodal dialogue data and reordering the remaining dialogue data according to dialogue weights, wherein the dialogue weights are determined according to the similarity between every two dialogue text sub-data in the remaining dialogue data; and The sentiment classification module is used to input the multimodal input data into the sentiment analysis model to obtain the sentiment classification of the user.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images