Multi-modal data processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a multi-modal data processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring multi-modal data, wherein the multi-modal data comprises text data and voice data; inputting the text data and the voice data into an emotional state model, and receiving a predicted emotional result output by the emotional state model; predicting the emotion characterization degree of the emotion result; the emotional state model is determined based on a five-stage emotion migration path and a hidden Markov model; and determining a response strategy according to the predicted emotion result. By adopting the method, the accuracy of the response strategy can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a multimodal data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of artificial intelligence technology, sentiment analysis technology has emerged. Sentiment analysis technology refers to the process of analyzing, processing and extracting subjective data with emotional colors using natural language processing and data mining technology.

[0003] In traditional technology, emotion recognition is performed through unimodal data or multimodal data. When using unimodal data, this method is limited by the semantic ambiguity of the text, context dependency gaps, and noise interference from the speech environment. When using multimodal data, it faces bottlenecks such as feature temporal misalignment, cross-modal conflicts, and rigid static weight allocation.

[0004] However, current emotion recognition methods have a high risk of emotion misjudgment, resulting in a low accuracy rate of response strategies based on incorrect emotion analysis results. Summary of the Invention

[0005] Based on this, it is necessary to provide a multimodal data processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of answering in response to the above technical problems.

[0006] In a first aspect, the present application provides a multimodal data processing method, the method comprising:

[0007] Acquiring multimodal data, wherein the multimodal data includes text data and voice data;

[0008] Inputting the text data and the speech data into an emotional state model, and receiving a predicted emotional result output by the emotional state model; the predicted emotional result represents the degree of emotion; the emotional state model is determined based on a five-stage emotion migration path and a hidden Markov model;

[0009] A response strategy is determined based on the predicted emotional result.

[0010] In one embodiment, the method further comprises:

[0011] Aligning the timestamps of the text data and the voice data obtained by the alignment extraction layer to obtain aligned text data and aligned voice data; extracting the emotional features of the aligned text data to obtain a text emotional feature vector; extracting the emotional features of the aligned voice data to obtain a voice emotional feature vector;

[0012] Obtaining a fusion emotion feature vector by performing a hidden Markov model on the text emotion feature vector and the speech emotion feature vector;

[0013] The fused emotion feature vector is processed based on the five-stage emotion migration path to obtain the predicted emotion result.

[0014] In one embodiment, the aligned text data includes current text data of a current time period, and extracting the sentiment features of the aligned text data to obtain a text sentiment feature vector includes:

[0015] Acquire historical text data; the historical text data is text data in a target time period prior to the current text data in the current time period;

[0016] Extracting an initial sentiment feature vector of the current text data based on the historical text data and the current text data;

[0017] Calculating the cosine similarity between the initial sentiment feature vector and each standard sentiment feature vector in the sentiment dictionary;

[0018] The text emotion feature vector corresponding to the initial emotion feature vector is determined according to each of the cosine similarities; the text emotion feature is a standard emotion feature vector corresponding to a maximum value of the cosine similarities.

[0019] In one embodiment, extracting the emotional features of the aligned speech data to obtain the speech emotional feature vector includes:

[0020] Performing noise reduction processing on the voice data to obtain noise-reduced voice data;

[0021] Extract the Mel-frequency cepstral coefficient features of the denoised speech data to obtain a feature matrix;

[0022] Processing the feature matrix through a target command to obtain a prosodic feature vector; wherein the prosodic feature vector represents an acoustic attribute in speech; the acoustic attribute includes fundamental frequency, energy, and duration;

[0023] Processing the feature matrix through voiceprint embedding to obtain a voiceprint embedding feature vector; the voiceprint embedding feature vector represents the pronunciation style;

[0024] The prosody feature vector and the voiceprint embedding feature vector are concatenated to obtain a speech emotion feature vector.

[0025] In one embodiment, the obtaining of a fused emotion feature vector by performing a hidden Markov model on the text emotion feature vector and the speech emotion feature vector includes:

[0026] The text sentiment feature vector is processed through a dual-stream LSTM time series to output the text hidden state; the text hidden state represents the semantic coherence of the text;

[0027] The speech emotion feature vector is processed through a dual-stream LSTM time series to output the speech hidden state; the speech hidden state represents the change of speech emotion;

[0028] Through the attention mechanism, the confidence weight of the text hidden state and the confidence weight of the speech hidden state are calculated;

[0029] A fused emotion feature vector is obtained based on the confidence weight of the text hidden state and the confidence weight of the speech hidden state.

[0030] In one embodiment, the five-stage emotion transition path includes a path formed by a calm state, a focused state, an anxious state, an angry state, and a satisfied state; and processing the fused emotion feature vector based on the five-stage emotion transition path to obtain the predicted emotion result includes:

[0031] Calculating the probability that the fused emotional feature vector is located in each state of the five-stage emotional migration path;

[0032] Determine the target state path based on the probability and the preset five-stage emotion transition constraints;

[0033] Based on the target state path of the fused emotion feature vector, a predicted emotion result is determined.

[0034] In a second aspect, the present application further provides a multimodal data processing device, the device comprising:

[0035] An acquisition module, configured to acquire multimodal data, wherein the multimodal data includes text data and voice data;

[0036] An input receiving module, configured to input the text data and the speech data into an emotional state model and receive a predicted emotional result output by the emotional state model; the predicted emotional result represents the degree of emotion; the emotional state model is determined based on a five-stage emotion migration path and a hidden Markov model;

[0037] The response module is used to determine a response strategy based on the predicted emotion result.

[0038] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0040] In a fifth aspect, the present application also provides a computer program product, comprising a computer program, which implements the steps of the above method when executed by a processor.

[0041] The above-mentioned multimodal data processing method, apparatus, computer equipment, computer-readable storage medium and computer program product first obtain multimodal data, which includes text data and voice data; obtain more data information to improve the accuracy of emotion recognition, and avoid the inaccuracy of emotion recognition caused by relying on single data; secondly, input the text data and voice data into the emotional state model determined based on the five-stage emotion migration path and the hidden Markov model, resolve acoustic-semantic conflicts and capture hidden information, thereby improving the accuracy of emotion recognition; finally, based on the predicted emotional results, determine the response strategy corresponding to the predicted emotional results, thereby improving the accuracy of the response strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 FIG1 is an application environment diagram of a multimodal data processing method in one embodiment;

[0044] Figure 2 1 is a flow chart of a multimodal data processing method in one embodiment;

[0045] Figure 3 A schematic diagram of a process for processing multimodal data by an emotional state model in one embodiment;

[0046] Figure 4 Schematic diagram of the process of generating a text sentiment feature vector in one embodiment;

[0047] Figure 5 A schematic diagram of a process for extracting a speech emotion feature vector in one embodiment;

[0048] Figure 6 Schematic diagram of a process for extracting a fused emotion feature vector in one embodiment;

[0049] Figure 7A schematic diagram of a process for processing a fused emotion feature vector based on a five-stage emotion migration path to obtain a predicted emotion result in one embodiment;

[0050] Figure 8 is a structural block diagram of a multimodal data processing device in one embodiment;

[0051] Figure 9 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0053] The multimodal data processing method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The emotional state model can be deployed on the terminal 102 or on the server 104. If the emotional state model can be deployed on the terminal 102, the terminal 102 obtains multimodal data, and the multimodal data includes text data and voice data; the text data and voice data are input into the emotional state model, and the predicted emotional result output by the emotional state model is received; the predicted emotional result represents the degree of emotion; the emotional state model is determined based on the five-stage emotion migration path and the hidden Markov model; according to the predicted emotional result, the response strategy is determined. If the emotional state model can be deployed on server 104, terminal 102 acquires multimodal data, including text and voice data. The text and voice data are input into the emotional state model deployed on server 104, and terminal 102 receives the predicted emotional outcome output by the emotional state model. The predicted emotional outcome represents the degree of emotion. The emotional state model is determined based on the five-stage emotion transition path and the Hidden Markov Model. Terminal 102 determines a response strategy based on the predicted emotional outcome. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart car devices, and projectors. Portable wearable devices can include smart watches, smart bracelets, head-mounted devices, and the like. Head-mounted devices can include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, and the like. Server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0054] In an exemplary embodiment, Figure 2 As shown, a multimodal data processing method is provided, which is described by taking the method applied to a terminal as an example, and includes the following steps S202 to S206. Among them:

[0055] Step S202: Acquire multimodal data, where the multimodal data includes text data and voice data.

[0056] Among them, multimodal data can also include facial micro-expressions, such as raised eyebrows, upturned corners of the mouth, etc.; physiological data, such as heartbeat, etc.

[0057] Optionally, in a customer service scenario, the terminal acquires multimodal data, including text and voice data. The voice data includes the intonation (high volume / trembling): "Where! Is! My! Order!?"; the text data includes the customer service chat box on the terminal display: "Can you help me check the status of my order?"

[0058] Optionally, while the terminal acquires the user's text data and voice data, it also acquires the user's facial image through an image acquisition device and acquires the user's heartbeat through a physiological acquisition device.

[0059] Step S204: input the text data and voice data into the emotional state model, and receive the predicted emotional result output by the emotional state model.

[0060] The predicted emotion result represents the degree of emotion, which can include calmness, concern, anxiety, anger, and satisfaction. The degree of emotion can be divided into other levels, such as the degree of anxiety, including mild anxiety, moderate anxiety, and severe anxiety, and the degree of calmness, including mild calmness, moderate calmness, and severe calmness.

[0061] The affective state model is based on a five-stage emotional transition path and a hidden Markov model. The five-stage emotional transition path is a conceptual framework that describes the dynamic changes in a user's emotional state during specific events or situations. Stage 1: Calm: The individual is neutral or relaxed, with no particular emotional fluctuations. They are unaware of or haven't yet paid attention to a specific event or issue. Stage 2: Attention: The individual begins to notice information, a problem, a need, or a potential change. Their attention is captured, generating interest or initial alertness. Stage 3: Anxiety: Concern deepens, accompanied by uncertainty, worry, or tension. The individual begins to consider the severity of the problem, its possible consequences, and their ability to resolve or control the situation, becoming uneasy about the uncertainty and potential negative outcomes. Stage 4: Anger: When anxiety fails to subside and the individual believes the problem is caused by external factors (such as others' mistakes, system failures, unfair treatment, or obstacles) and that these factors are avoidable or should be corrected, anxiety escalates into anger. This is a more intense negative emotion, accompanied by frustration, blame, and confrontational tendencies. Stage 5: Satisfaction: The problem is effectively resolved, needs are met, misunderstandings are clarified, fairness is restored, or a positive outcome occurs. Negative emotions (anxiety, anger) dissipate, replaced by relief, joy, or satisfaction. The individual feels that the problem has been closed.

[0062] A Hidden Markov Model (HMM) is a probabilistic graphical model used to model time series data. It assumes that the system is a Markov process, but the system's state is not directly observable (i.e., "hidden") and can only be inferred from the observed output sequence.

[0063] Optionally, the terminal trains the initial emotional state model based on sample data, which includes sample text and sample speech, based on a five-stage emotion migration path and a hidden Markov model, to obtain a trained emotional state model.

[0064] Optionally, the terminal inputs the acquired text data and voice data into an emotional state model, and the emotional state model processes the text data and voice data and outputs a processing result, that is, a predicted emotional result.

[0065] Step S206: Determine a response strategy based on the predicted emotion result.

[0066] Optionally, based on the predicted emotional outcome, for example, anxiety, the terminal's response strategy might be to expedite the response and provide text and / or voice output to appease the user. For anger, the response strategy might be to initiate an apology while eliminating solutions to trigger severe anger. For satisfaction, the response strategy might be to request positive reviews and recommend relevant information.

[0067] In the above-mentioned multimodal data processing method, first, multimodal data is obtained, and the multimodal data includes text data and voice data; more data information is obtained to improve the accuracy of emotion recognition, and the inaccuracy of emotion recognition caused by relying on single data is avoided; secondly, the text data and voice data are input into the emotional state model determined based on the five-stage emotion migration path and the hidden Markov model, so as to resolve the acoustic-semantic conflict and capture hidden information, thereby improving the accuracy of emotion recognition; finally, based on the predicted emotional results, the response strategy corresponding to the predicted emotional results is determined, thereby improving the accuracy of the response strategy.

[0068] In an exemplary embodiment, Figure 3 As shown, the emotional state model processes multimodal data including steps S302 to S308.

[0069] Step S302 , aligning the acquired timestamps of the text data and the voice data through an alignment extraction layer to obtain aligned text data and aligned voice data.

[0070] Both text data and speech data are time series data. The emotional state model includes an alignment extraction layer, a hidden Markov model, and a five-stage emotion migration path.

[0071] Optionally, the terminal first needs to align the timestamps of the text data and the voice data, and align the timestamps of the text paragraphs and the corresponding voice segments through dynamic time warping to obtain aligned text data and aligned voice data.

[0072] Step S304: extract the emotional features of the aligned text data to obtain a text emotional feature vector; extract the emotional features of the aligned speech data to obtain a speech emotional feature vector.

[0073] Optionally, the terminal may extract the emotional features of the aligned text data and the emotional features of the aligned speech data in parallel to obtain a text emotional feature vector and a speech emotional feature vector. The terminal may also first extract the emotional features of the aligned text data to obtain a text emotional feature vector. The terminal may also first extract the emotional features of the aligned speech data to obtain a speech emotional feature vector.

[0074] Step S306: Obtain a fused emotion feature vector by performing a hidden Markov model on the text emotion feature vector and the speech emotion feature vector.

[0075] Among them, since the text emotion feature vector and the speech emotion feature vector are time series, the fused emotion feature vector is also a time series.

[0076] Optionally, the terminal calculates the confidence of the text emotion feature vector and the speech emotion feature vector through a hidden Markov model, and obtains a fused emotion feature vector according to the confidence.

[0077] Confidence can be used to handle situations where there are inconsistencies between speech and text data, such as when the text data says "I'm fine" but the speech data says "sobbing and shaky breathing." By adjusting the confidence levels of the text and speech data, a fused emotional feature vector is determined.

[0078] Step S308: Process the fused emotion feature vector based on the five-stage emotion migration path to obtain a predicted emotion result.

[0079] Optionally, based on the five-stage emotion transition path and the fused emotion feature vector, the terminal calculates the probability that each time-step feature in the fused emotion feature vector belongs to the five-stage emotion transition path. Based on the probabilities, the terminal generates an initial state path, forcing the generated initial state path to conform to the five-stage transition rules, and ultimately outputs a target state path. For example, given the fused emotion feature vector [t=1: text "How's the order?", t=2: text "Too slow!", t=3: text "I'll fix it now!" + voice tremor], the terminal calculates the probability that the state at t=1 belongs to the five-stage emotion transition path. For example, if Pt1 represents the probability of the first time-step feature, indicating an attention state; Pt2 represents the probability of the second time-step feature, indicating an anxiety state; and Pt3 represents the probability of the third time-step feature, indicating an anger state. The resulting path is attention-anxiety-anger, which conforms to the constraints of the five-stage emotion transition path. Therefore, the final target state path is attention-anxiety-anger. If the generated path does not conform to the constraints of the five-stage emotion transition path, the path needs to be adjusted to determine the target state path. Based on the target state path, the terminal calculates the feature strength, duration, and transition depth to ultimately determine the predicted emotion.

[0080] In this embodiment, by aligning text data and voice data, hidden features can be obtained through a hidden Markov model, and finally the predicted emotional result is determined based on the five-stage emotion migration path and the fused emotional feature vector, which can improve the accuracy of the predicted emotional result.

[0081] In an exemplary embodiment, Figure 4 As shown, the aligned text data includes the current text data of the current time period, and the emotional features of the aligned text data are extracted to obtain the text emotional feature vector, including steps S402 to S408.

[0082] Step S402: Acquire historical text data.

[0083] The historical text data refers to the text data in the target time period before the current text data in the current time period.

[0084] Optionally, the terminal divides the text data into current text data and historical text data according to time. The historical text data is text data in a target time period before the current text data in the current time period.

[0085] Step S404: extracting an initial sentiment feature vector of the current text data based on the historical text data and the current text data.

[0086] Optionally, the terminal extracts an initial sentiment feature vector of the current text data using BERT embedding based on historical text data and current text data. The initial sentiment feature vector of the current text data can be better extracted through semantic context understanding.

[0087] Step S406 , calculating the cosine similarity between the initial sentiment feature vector and each standard sentiment feature vector in the sentiment dictionary.

[0088] Optionally, the terminal adds the initial emotion feature vector through the emotion dictionary. The terminal calculates the cosine similarity between the initial emotion feature vector and each standard emotion feature vector in the emotion dictionary.

[0089] Step S408: determining the text sentiment feature vector corresponding to the initial sentiment feature vector according to each cosine similarity.

[0090] Among them, the text sentiment feature is the standard sentiment feature vector corresponding to the maximum value of cosine similarity.

[0091] Optionally, the terminal obtains the cosine similarity value between the initial emotion feature vector and each standard emotion feature vector in the emotion dictionary, and uses the standard emotion feature vector corresponding to the maximum value as the text emotion feature vector corresponding to the initial emotion feature vector.

[0092] In this embodiment, the initial sentiment feature vector is extracted by BERT embedding and enhanced using the sentiment dictionary to obtain the text sentiment feature vector, which can improve the accuracy of the text sentiment feature vector.

[0093] In an exemplary embodiment, Figure 5 As shown, extracting the emotional features of the aligned speech data to obtain the speech emotional feature vector includes steps S502 to S510.

[0094] Step S502: performing noise reduction processing on the aligned speech data to obtain noise-reduced speech data.

[0095] Optionally, the terminal obtains the aligned voice data, such as a .wav file, and performs noise reduction on the aligned voice data using spectral subtraction, Wiener filtering, wavelet denoising, or a neural network model such as RNNoise or DCCRN. This noise reduction preserves the voice content and reduces background noise.

[0096] Step S504: extract Mel-frequency cepstral coefficient features from the denoised speech data to obtain a feature matrix.

[0097] Among them, MFCC (Mel-Frequency Cepstral Coefficient) is the core feature in speech signal processing. Its essence is the sound "fingerprint" extracted by imitating the auditory characteristics of the human ear.

[0098] Optionally, the terminal enhances the audio component of the noise-reduced speech data through a high-pass filter, frames, adds windows, converts the time domain signal into a frequency domain spectrum, performs a Mel filter bank, takes the logarithm, and performs a discrete cosine transform to obtain MFCC coefficients, and finally obtains a time series MFCC feature matrix.

[0099] Step S506: Process the feature matrix using the target command to obtain a prosodic feature vector.

[0100] A prosodic feature vector is a statistical summary of the prosodic characteristics of an entire speech signal (such as a sentence or a conversation). It captures the emotional expression pattern of speech at a macroscopic scale and represents the acoustic properties of speech, including fundamental frequency, energy, and duration.

[0101] The target command can be OpenSMILE. OPENsmile (full name: open Speech and Music Interpretation by Large Space Extraction) is an open source toolkit focusing on audio signal processing and feature extraction, mainly used in speech recognition, sentiment computing, music information retrieval and other fields.

[0102] Optionally, the terminal extracts the feature matrix using OpenSMILE to obtain a prosodic feature vector, which includes fundamental frequency, energy, and duration. The fundamental frequency (F0) physically represents the auditory pitch of the sound (unit: Hz). Statistical features include the F0 mean, which represents the overall pitch level; the F0 range, which represents the amplitude of pitch variation; the F0 standard deviation, which represents the degree of pitch fluctuation; and the F0 slope, which represents the upward / downward trend of pitch. Energy physically represents the loudness or intensity of the speech (unit: dB). Statistical features include the energy mean, which represents the overall volume; the energy dynamic range, which represents the contrast between strong and weak parts; and the high energy ratio, which represents the duration of the loudest parts. Duration physically represents the temporal structure of the speech segment. Statistical features include speech rate, which represents the speaking rate; pause frequency, which represents the number of interruptions of silence; and the proportion of long pauses, which represent periods of thought or suppressed emotion.

[0103] Step S508: Process the feature matrix through voiceprint embedding to obtain a voiceprint embedding feature vector; the voiceprint embedding feature vector represents the pronunciation style.

[0104] The voiceprint embedding, also known as the voiceprint embedding model, is pre-trained. The voiceprint embedding feature vector represents the user's pronunciation style.

[0105] Optionally, the terminal voiceprint embedding performs frame processing on the feature matrix, extracts features of each frame, and finally outputs a voiceprint embedding feature vector.

[0106] Step S510: concatenate the prosody feature vector and the voiceprint embedding feature vector to obtain a speech emotion feature vector.

[0107] Optionally, the terminal concatenates the prosody feature vector and the voiceprint embedding feature vector, performs z-score normalization on the concatenated vector, and uses PCA or an autoencoder to compress the dimension of the normalized vector to obtain a speech emotion feature vector.

[0108] In this embodiment, the speech emotion feature vector is obtained by splicing the prosody feature vector and the voiceprint embedding feature vector, so that the speech emotion feature vector acquires the acoustic properties and the pronunciation style of the user, making the extracted speech emotion feature vector more accurate.

[0109] In an exemplary embodiment, Figure 6 As shown, the text emotion feature vector and the speech emotion feature vector are processed by the hidden Markov model to obtain a fusion emotion feature vector, including steps S602 to S608.

[0110] Step S602: Process the text sentiment feature vector through the dual-stream LSTM time series and output the text hidden state.

[0111] Step S604: Process the speech emotion feature vector through the dual-stream LSTM time sequence and output the speech hidden state.

[0112] The text hidden state represents the semantic coherence of the text, and the speech hidden state represents the emotional changes of the speech.

[0113] The dual-stream LSTM time series model is a deep learning architecture that integrates two independent LSTM networks and is specifically designed to process multimodal time series data (such as text + voice, video + sensor, etc.).

[0114] Optionally, the terminal processes the text sentiment feature vector through LSTM stream 1 to capture semantic reinforcement and semantic coherence, and processes the speech sentiment feature vector through LSTM stream 2 to capture pitch spikes and tremor patterns.

[0115] Step S606: Calculate the confidence weight of the text hidden state and the confidence weight of the speech hidden state through the attention mechanism.

[0116] Step S608: Obtain a fused emotion feature vector based on the confidence weight of the text hidden state and the confidence weight of the speech hidden state.

[0117] Among them, the attention mechanism includes one of the soft attention mechanism, hard attention mechanism, self-attention mechanism, multi-head attention mechanism, and spatial attention mechanism.

[0118] Optionally, the attention mechanism calculates confidence weights for the text hidden state and the speech hidden state. For example, if a user sneers and says "no problem," the terminal calculates a confidence weight of 0.3 for the text hidden state (literal positive) and 0.7 for the speech hidden state (falling tone + breathy sound). Therefore, the confidence weights for the text hidden state are 0.3, and 0.7 for the speech hidden state. Based on the confidence weights for the text hidden state and the speech hidden state, the terminal generates a fused sentiment feature vector.

[0119] In this embodiment, hidden features of speech and text are obtained through a hidden Markov model, and a fused emotional feature vector is obtained based on the hidden features of speech and text, which can mine more information from speech and text. The fused emotional feature vector will carry more information so that the predicted emotional results can be accurately output based on the fused emotional feature vector.

[0120] In an exemplary embodiment, Figure 7 As shown, the five-stage emotion transition path includes the path formed by the calm state, the concerned state, the anxious state, the angry state, and the satisfied state; processing the fused emotion feature vector based on the five-stage emotion transition path to obtain the predicted emotion result includes steps S702 to S706. Among them:

[0121] Step S702: Calculate the probability that the fused emotional feature vector is in each state in the five-stage emotional transition path.

[0122] Step S704: Determine the target state path based on the probability and the preset five-stage emotion transition constraints.

[0123] Step S706: Determine the predicted emotion result based on the fused emotion feature vector and the target state path.

[0124] Optionally, the emotional feature vector [t=1: text "How's the order?", t=2: text "Too slow," t=3: text "I'll fix it!" + voice tremor] is fused to calculate the probability of belonging to the five-stage emotional transition path at t=1. For example, Pt1 represents the probability of the first time step feature, indicating an attention state; Pt2 represents the probability of the second time step feature, indicating an anxiety state; and Pt3 represents the probability of the third time step feature, indicating an anger state. The resulting path is attention-anxiety-anger, which meets the constraints of the five-stage emotional transition path. Therefore, the final target state path is attention-anxiety-anger. Based on the target state path and the fused emotional feature vector, the terminal calculates the feature strength, duration, and transition depth to ultimately determine the predicted emotional outcome. The constraints of the five-stage emotional transition path include prohibiting skipping across stages (e.g., calm → anxious) and allowing a small probability of backtracking (e.g., anxiety → attention with a 20% probability).

[0125] S0 represents calmness, S1 represents concern, S2 represents anxiety, S3 represents anger, and S4 represents satisfaction. Consider the target state path S_path (e.g., ['S0', 'S1', 'S2', 'S2', 'S3']) and the current state (i.e., the last state, such as 'S3'). The Euclidean distance between the feature vector at the last time step (i.e., the last vector in the entire fused emotion feature vector) and the baseline feature vector for the calm state (S0) is used, then normalized by dividing by the maximum distance to obtain the feature strength. The baseline feature vector for the calm state (S0) is preset.

[0126] Starting from the current state, traverse the state path in reverse, count the number of times the current state appears consecutively (until a different state is encountered), and then divide the number of continuous steps by the maximum duration (for example, 5) for normalization to obtain the state duration, and limit the state duration to the range of [0,1].

[0127] Calculate the number of transition steps (i.e., the number of state changes) from the most recent calm state (S0) in the path to the current state (S_T). Specifically, first find the most recent calm state in the state path (i.e., the last occurrence of S0), then calculate the number of state transitions (i.e., the number of state changes) from this S0 position to the current state position. Then, normalize this number of steps by the maximum possible depth (e.g., 4, since there are at most 4 steps from S0 to S4) to obtain the transition depth.

[0128] The feature strength, state duration and migration depth are weighted to obtain the predicted sentiment results.

[0129] In this embodiment, the fused emotion feature vector is processed based on the five-stage emotion migration path to obtain a predicted emotion result, which can improve the accuracy of the predicted emotion result.

[0130] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0131] Based on the same inventive concept, embodiments of the present application also provide a multimodal data processing device for implementing the multimodal data processing method described above. The implementation solution provided by this device is similar to the implementation solution described in the above method. Therefore, the specific limitations of one or more multimodal data processing device embodiments provided below can be found in the above-mentioned limitations of the multimodal data processing method and will not be repeated here.

[0132] In an exemplary embodiment, Figure 8 As shown, a multimodal data processing device is provided, including: an acquisition module 801, an input receiving module 802 and a response module 803, wherein:

[0133] The acquisition module 801 is used to acquire multimodal data, where the multimodal data includes text data and voice data.

[0134] The input receiving module 802 is used to input text data and voice data into the emotional state model and receive the predicted emotional result output by the emotional state model; the predicted emotional result represents the degree of emotion; the emotional state model is determined based on the five-stage emotion migration path and the hidden Markov model.

[0135] The response module 803 is used to determine a response strategy based on the predicted emotion result.

[0136] In an exemplary embodiment, a multimodal data processing device is provided, which also includes a model processing module for aligning the timestamps of text data and voice data obtained through an alignment extraction layer to obtain aligned text data and aligned voice data; extracting the emotional features of the aligned text data to obtain a text emotional feature vector; extracting the emotional features of the aligned voice data to obtain a voice emotional feature vector; obtaining a fused emotional feature vector by processing the text emotional feature vector and the voice emotional feature vector through a hidden Markov model; and processing the fused emotional feature vector based on a five-stage emotion migration path to obtain a predicted emotional result.

[0137] In an exemplary embodiment, the aligned text data includes the current text data of the current time period, and the model processing module is also used to obtain historical text data; the historical text data is the text data in the target time period before the current text data of the current time period; based on the historical text data and the current text data, the initial sentiment feature vector of the current text data is extracted; the cosine similarity between the initial sentiment feature vector and each standard sentiment feature vector in the sentiment dictionary is calculated; based on each cosine similarity, the text sentiment feature vector corresponding to the initial sentiment feature vector is determined; the text sentiment feature is based on the standard sentiment feature vector corresponding to the maximum value of the cosine similarity.

[0138] In an exemplary embodiment, the model processing module is also used to perform noise reduction processing on the speech data to obtain noise-reduced speech data; extract the Mel-frequency cepstral coefficient features of the noise-reduced speech data to obtain a feature matrix; process the feature matrix through target commands to obtain a prosodic feature vector; wherein the prosodic feature vector represents the acoustic properties in the speech; the acoustic properties include fundamental frequency, energy and duration; process the feature matrix through voiceprint embedding to obtain a voiceprint embedded feature vector; the voiceprint embedded feature vector represents the pronunciation style; and concatenate the prosodic feature vector and the voiceprint embedded feature vector to obtain a speech emotion feature vector.

[0139] In an exemplary embodiment, the model processing module is also used to process the text emotion feature vector through a dual-stream LSTM time sequence and output a text hidden state; the text hidden state represents the semantic coherence of the text; the speech emotion feature vector is processed through a dual-stream LSTM time sequence and output a speech hidden state; the speech hidden state represents the change in speech emotion; through the attention mechanism, the confidence weight of the text hidden state and the confidence weight of the speech hidden state are calculated; based on the confidence weight of the text hidden state and the confidence weight of the speech hidden state, a fused emotion feature vector is obtained.

[0140] In an exemplary embodiment, the five-stage emotion transition path includes paths formed by a calm state, a focused state, an anxious state, an angry state, and a satisfied state; the model processing module is also used to calculate the probability that the fused emotion feature vector is located in each state in the five-stage emotion transition path; determine the target state path based on the probability and the preset five-stage emotion transition constraints; and determine the predicted emotion result based on the target state path of the fused emotion feature vector.

[0141] Each module in the multimodal data processing device described above may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0142] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store multimodal data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multimodal data processing method is implemented.

[0143] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0144] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0145] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0146] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0147] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0148] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0149] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A multimodal data processing method, characterized in that: The method comprises: Acquiring multimodal data, wherein the multimodal data includes text data and voice data; Inputting the text data and the speech data into an emotional state model, and receiving a predicted emotional result output by the emotional state model; the predicted emotional result represents the degree of emotion; the emotional state model is determined based on a five-stage emotion migration path and a hidden Markov model; A response strategy is determined based on the predicted emotional result.

2. The method according to claim 1, characterized in that The method further comprises: Aligning the timestamps of the text data and the voice data obtained by the alignment extraction layer to obtain aligned text data and aligned voice data; extracting the emotional features of the aligned text data to obtain a text emotional feature vector; extracting the emotional features of the aligned voice data to obtain a voice emotional feature vector; Obtaining a fusion emotion feature vector by performing a hidden Markov model on the text emotion feature vector and the speech emotion feature vector; The fused emotion feature vector is processed based on the five-stage emotion migration path to obtain the predicted emotion result.

3. The method according to claim 2, characterized in that The aligned text data includes current text data of a current time period, and extracting the sentiment features of the aligned text data to obtain a text sentiment feature vector includes: Acquire historical text data; the historical text data is text data in a target time period prior to the current text data in the current time period; Extracting an initial sentiment feature vector of the current text data based on the historical text data and the current text data; Calculating the cosine similarity between the initial sentiment feature vector and each standard sentiment feature vector in the sentiment dictionary; The text emotion feature vector corresponding to the initial emotion feature vector is determined according to each of the cosine similarities; the text emotion feature is a standard emotion feature vector corresponding to a maximum value of the cosine similarities.

4. The method according to claim 2, characterized in that The step of extracting the emotional features of the aligned speech data to obtain a speech emotional feature vector includes: Performing noise reduction processing on the voice data to obtain noise-reduced voice data; Extract the Mel-frequency cepstral coefficient features of the denoised speech data to obtain a feature matrix; Processing the feature matrix through a target command to obtain a prosodic feature vector; wherein the prosodic feature vector represents an acoustic attribute in speech; the acoustic attribute includes fundamental frequency, energy, and duration; Processing the feature matrix through voiceprint embedding to obtain a voiceprint embedding feature vector; the voiceprint embedding feature vector represents the pronunciation style; The prosody feature vector and the voiceprint embedding feature vector are concatenated to obtain a speech emotion feature vector.

5. The method according to claim 2, characterized in that The method of obtaining a fused emotion feature vector by performing a hidden Markov model on the text emotion feature vector and the speech emotion feature vector comprises: The text sentiment feature vector is processed through a dual-stream LSTM time series to output the text hidden state; the text hidden state represents the semantic coherence of the text; The speech emotion feature vector is processed through a dual-stream LSTM time series to output the speech hidden state; the speech hidden state represents the change of speech emotion; Through the attention mechanism, the confidence weight of the text hidden state and the confidence weight of the speech hidden state are calculated; A fused emotion feature vector is obtained based on the confidence weight of the text hidden state and the confidence weight of the speech hidden state.

6. The method according to claim 2, characterized in that The five-stage emotional migration path includes the path formed by the calm state, the concerned state, the anxious state, the angry state, and the satisfied state; The processing of the fused emotion feature vector based on the five-stage emotion migration path to obtain the predicted emotion result includes: Calculating the probability that the fused emotional feature vector is located in each state of the five-stage emotional migration path; Determine the target state path based on the probability and the preset five-stage emotion transition constraints; Based on the fused emotion feature vector and the target state path, a predicted emotion result is determined.

7. A multimodal data processing device, characterized in that: The device comprises: An acquisition module, configured to acquire multimodal data, wherein the multimodal data includes text data and voice data; An input receiving module, configured to input the text data and the speech data into an emotional state model and receive a predicted emotional result output by the emotional state model; the predicted emotional result represents the degree of emotion; the emotional state model is determined based on a five-stage emotion migration path and a hidden Markov model; The response module is used to determine a response strategy based on the predicted emotion result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images