Speech behavior sequence recognition and man-machine conversation action generation method based on MCE model

Abstract

The invention discloses a speech behavior sequence recognition and man-machine conversation action generation method based on an MCE model. The method comprises the following steps of 1, recognizing and extracting multi-modal data; 2, extracting an identity ID feature and an emotion label feature; 3, extracting text features; 4, extracting audio features; 5, dialogue action features are extracted; step 6, generating fusion features through a multi-condition encoder model; 7, extracting local and global information of the fusion features through a multi-condition cross attention mechanism, and generating optimized features; and 8, generating a speech action matched with the speech content based on the optimized features, and adjusting the speech action by adopting a reconstruction loss function and a speech action speed loss function. According to the method, the action sequence which is highly consistent with the speech content and is natural and smooth is generated end to end through the MCE model and the multi-condition cross attention mechanism, and integration of speech behavior recognition and action generation is achieved.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a method for speech behavior sequence recognition and human-computer dialogue action generation based on the MCE model. Background Technology

[0002] With the continuous development of artificial intelligence and human-computer interaction technologies, speech behavior sequence recognition and human-computer dialogue action generation technologies based on multimodal data have received increasing attention. Most existing speech behavior recognition technologies rely on single-modal data, such as relying solely on text or audio signals for speech and behavior analysis. However, in practical applications, the following problems often exist:

[0003] Traditional methods neglect the fusion of multimodal data, resulting in weak comprehensive understanding of information such as emotion, identity, and language, and failing to fully explore the potential relationships between multiple modalities. Existing behavior generation models often rely solely on limited text or audio features, failing to accurately capture key behavioral features such as the speaker's body language and facial expressions, leading to inconsistent and unnatural generated actions. In multimodal data processing, different modalities have different dimensions and information densities, making effective alignment and fusion of these data a technical challenge. While traditional deep learning models have made breakthroughs in processing temporal data, they suffer from low processing efficiency in scenarios involving long-term, multimodal data fusion and struggle to capture long-distance dependencies in complex contexts. These issues pose numerous challenges to existing speech behavior recognition and human-computer dialogue action generation technologies in practical applications, especially in scenarios requiring high real-time performance and accuracy, where model performance is unstable, impacting user experience and system effectiveness.

[0004] Therefore, how to provide a speech behavior sequence recognition and human-computer dialogue action generation method based on the MCE model is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a method for speech behavior sequence recognition and human-computer dialogue action generation based on the MCE model. This invention utilizes multimodal data fusion, the MCE model, and a multi-conditional cross-attention mechanism to optimize the generation of speech actions, and adjusts the accuracy and naturalness of action generation through reconstruction loss function and speech action speed loss function.

[0006] The speech behavior sequence recognition and human-computer dialogue action generation method based on the MCE model according to embodiments of the present invention includes the following steps: Step 1: Identify and extract multimodal data, which includes identity ID data, sentiment tag data, text data, audio data, and dialogue action data; Step 2: Input the identity ID data and the sentiment tag data into the one-hot encoding module to extract the encoded identity ID features and sentiment tag features; Step 3: Input the text data into the FastText encoder for dimensionality reduction, and then adjust it using the DA-BERT model to extract text features; Step 4: Input the audio data into the DA-BERT model for feature extraction and dimensionality reduction to extract audio features; Step 5: Extract dialogue action features by continuously extracting frames from the dialogue action data and performing masking and padding. Step Six: Input the identity ID features, sentiment tag features, text features, audio features, and dialogue action features into the multi-condition encoder model, perform feature encoding, transformation, and alignment, and obtain the fused features; Step 7: Extract local and global information of the fused features through a multi-condition cross-attention mechanism to generate optimized features; Step 8: Generate speech actions that match the speech content based on the optimized features, and adjust the speech actions using a reconstruction loss function and a speech action speed loss function.

[0007] Optionally, the identification and extraction of multimodal data specifically includes: Identify and extract identity ID data and sentiment tag data from the speech scene; the identity ID data represents the speaker's unique identifier; the sentiment tag data represents the speaker's emotional state, and a sentiment analysis model is used to analyze the speech content and generate sentiment tags; Identify and extract text data from the speech content; the text data refers to the statements or content spoken by the speaker. Identify and extract audio data from the audio signal of the speech; the audio data includes the speaker's voice features; Identify and extract dialogue action data from the speaker's movements; the dialogue action data includes the speaker's body language.

[0008] Optionally, step two involves inputting the identity ID data and the sentiment tag into a one-hot encoding module to extract the encoded identity ID features and sentiment tag features, specifically: The identity ID data is input into the one-hot encoding unit of the one-hot encoding module, which maps each type of identity ID into a binary vector with "1" in the corresponding position and "0" in the rest. The binary vector is then mapped into a multi-dimensional identifier feature vector by the feature mapping unit of the one-hot encoding module. The identifier feature vector is used to represent the speaker's identity information. The emotion tag data is input into the same one-hot encoding module, and the one-hot encoding unit maps each emotion tag to a binary vector with "1" at the corresponding position and "0" at the rest. The binary vector is then mapped to a multi-dimensional emotion feature vector by the feature mapping unit. The emotion feature vector is used to characterize the speaker's emotion category.

[0009] Optionally, step three involves inputting the text data into the FastText encoder for dimensionality reduction and then adjusting it using the DA-BERT model to extract text features. Specifically: The text data is input into the FastText encoder and subjected to low-dimensional mapping to obtain the text features after preliminary dimensionality reduction; the text features are vector representations of the text content. The text features after initial dimensionality reduction are input into the DA-BERT model, and Masked Language Modeling and Next Sentence Prediction pre-training are performed sequentially. A cross-modal adapter is embedded after each Transformer layer of the DA-BERT model; the adapter includes sampling, activation, and upsampling structures. The final three self-attention sublayers of the DA-BERT model are replaced with a multi-conditional cross-attention mechanism to output text features from the DA-BERT model.

[0010] Optionally, step four involves inputting the audio data into the DA-BERT model for feature extraction and dimensionality reduction, specifically extracting audio features: Audio data is input into the audio encoder of the DA-BERT model, which has been pre-trained for domain-adaptive DA-BERT model and pre-trained for the speech scenario using Masked Audio Modeling. A lightweight cross-modal adapter is embedded in each Transformer layer of the DA-BERT model to fuse prior information from textual or sentiment modalities. In the last three layers of the DA-BERT model, the standard self-attention sublayer is replaced with an audio-driven multi-head cross-attention sublayer; the multi-head cross-attention sublayer uses the text hidden vector as the query and the audio hidden vector as the key, and further aligns the temporal and emotional information of the text and audio through a multi-condition cross-attention mechanism. The DA-BERT model outputs a high-dimensional audio feature vector; the audio features include frequency, pitch, speech rate, and emotional change information, which are used to express the speaker's audio characteristics. The audio features are then subjected to dimensionality reduction processing and mapped to a low-dimensional feature space to obtain the dimensionality-reduced audio features.

[0011] Optionally, step five: extracting dialogue action features from the dialogue action data by extracting consecutive frames and performing masking and padding, specifically: The dialogue action data is continuously extracted using motion detection or keyframe extraction techniques, and the frame data related to the speech content is selected. The extracted continuous frame data is masked and padded to fill in missing or missing information, generating complete dialogue action features, and the dialogue action features are normalized.

[0012] Optionally, step six involves inputting the identity ID features, sentiment tag features, text features, audio features, and dialogue action features into a multi-condition encoder model for feature encoding, transformation, and alignment to obtain fused features. Specifically: The identity ID features, sentiment tag features, text features, audio features, and dialogue action features are input into the MCE model, and linear projection is applied to each to map them to the same hidden dimension. Add conditional embedding vectors corresponding to each feature to identify the source modality, and add them to the projected features to generate conditional features; The conditional features are input in parallel to the Transformer encoder unit, each unit including a self-attention layer and a feedforward network, for extracting the interaction relationship between features in a global scope; In the attention calculation of each Transformer encoder unit, a multi-conditional cross-attention mechanism is adopted to fuse the conditional features of different modalities as queries, keys, and values ​​to complete the alignment and information supplementation between modalities. The conditional output after passing through the Transformer encoder unit is concatenated or weighted and summed, and a fusion feature is generated through a linear mapping layer. The fusion feature comprehensively represents the information of each modality and has an aligned and consistent representation.

[0013] Optionally, step seven: extracting local and global information of the fused features through a multi-condition cross-attention mechanism to generate optimized features, specifically: The fused features are mapped to query, key, and value vectors respectively, and then replicated into several subspace representations through a multi-head attention mechanism. Attention weights are calculated using the query vector and the key vector of the corresponding header, and the value vector is weighted and summed to extract local dependency information. The outputs of all attention heads are concatenated and a global fusion is formed through linear mapping. By applying feedforward networks and residual connections to global fusion, optimized features containing both local and global information are obtained.

[0014] Optionally, step eight: generating speech actions that match the speech content based on the optimized features, and adjusting the speech actions using a reconstruction loss function and a speech action speed loss function, specifically: Based on the optimized features input into the speech action generation module, speech actions that match the speech content are generated. In the generated speech actions, the difference between the generated actions and the target speech actions is calculated using the reconstruction loss function, and the actions are optimized accordingly. By using the speech action speed loss function, the deviation between the speed of the generated action and the actual speech rhythm is calculated, the speed of the action is adjusted, and the optimized speech action is output.

[0015] Optional, also includes: A training dataset was generated by collecting a dataset containing speech videos, audio, and text, and manually annotating it. Each data sample includes speech content, audio features, and related speech action labels. The training dataset is divided into a training set, a validation set, and a test set. The speech actions are evaluated using the validation set and the test set, and the parameters of the MCE model are adjusted based on the evaluation results.

[0016] The beneficial effects of this invention are: This invention addresses the shortcomings of existing speech behavior recognition and human-computer dialogue action generation methods by applying multimodal data fusion based on the MCE model and a multi-conditional cross-attention mechanism. These shortcomings include insufficient processing of single-modal information, unnatural generated actions, and low efficiency of multimodal data fusion. The invention comprehensively processes multimodal data, including identity IDs, sentiment tags, text, audio, and dialogue actions. Through one-hot encoding, FastText dimensionality reduction, BERT model tuning, and the fusion of multi-conditional encoder models, the accuracy of speech behavior recognition and the naturalness of human-computer interaction are improved. When generating speech actions, the reconstruction loss function and the speech action speed loss function are combined to precisely adjust the actions, ensuring that the generated speech actions are highly consistent with the speech content and are smooth and natural. Furthermore, the optimization of speech action generation using multimodal data effectively avoids the shortcomings of single-modality methods and the lack of realism and interactivity in action generation. Ultimately, natural and accurate speech actions can be generated in complex scenarios, improving the smoothness and accuracy of human-computer interaction and enhancing the user experience. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0018] Figure 1The flowchart shows the speech behavior sequence recognition and human-computer dialogue action generation method based on the MCE model proposed in this invention. Figure 2 This is a schematic diagram of the speech behavior sequence recognition and human-computer dialogue action generation method based on the MCE model proposed in this invention; Figure 3 This is a module interaction diagram of the speech behavior sequence recognition and human-computer dialogue action generation method based on the MCE model proposed in this invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0020] refer to Figure 1-3 A method for speech behavior sequence recognition and human-computer dialogue action generation based on the MCE model includes the following steps: Step 1: Identify and extract multimodal data, which includes identity ID data, sentiment tag data, text data, audio data, and dialogue action data; Step 2: Input the identity ID data and the sentiment tag data into the one-hot encoding module to extract the encoded identity ID features and sentiment tag features; Step 3: Input the text data into the FastText encoder for dimensionality reduction, and then adjust it using the DA-BERT model to extract text features; Step 4: Input the audio data into the DA-BERT model for feature extraction and dimensionality reduction to extract audio features; Step 5: Extract dialogue action features by continuously extracting frames from the dialogue action data and performing masking and padding. Step Six: Input the identity ID features, sentiment tag features, text features, audio features, and dialogue action features into the multi-condition encoder model, perform feature encoding, transformation, and alignment, and obtain the fused features; Step 7: Extract local and global information of the fused features through a multi-condition cross-attention mechanism to generate optimized features; Step 8: Generate speech actions that match the speech content based on the optimized features, and adjust the speech actions using a reconstruction loss function and a speech action speed loss function.

[0021] The feature extraction of the identity ID first involves one-hot encoding and embedding, followed by linear transformation to meet the model's dimensionality requirements. The formula is as follows: ; ; Where ID_embedding represents the embedding layer function, and Linear represents the linear function. This represents the identity ID label of the i-th sample. This represents a vector containing the identity ID. Let represent the feature vector obtained after further linear mapping, and let i represent the i-th input data sample.

[0022] The extraction of emotional features follows the same method as the processing of identity IDs, using the following formula:

[0023] ; Where Emo_Embedding represents the embedding function, and Linear represents the linear function. This represents the sentiment label of the i-th sample. Indicates emotion tags The corresponding embedding vector, This represents the emotional characteristics after linear mapping.

[0024] The dialogue action first randomly extracts consecutive frames as context with a certain probability, then embeds and maps them to a high-dimensional space through linear transformation, and then completes the frame using mask superposition. Finally, the prior information of the mask and the context knowledge are used to guide the generation of the remaining frames. ; ; Pose_Embedding is used to perform feature mapping encoding on the original pose data, mapping the key point coordinates into a high-dimensional representation; This indicates that starting from frame i, 34 × mr pose keypoint features are extracted consecutively; mr is a pre-set frame extraction ratio. This represents the processed pose feature vector. Is the same Zero tensors of the same shape, Mask_token represents a learnable mask vector.

[0025] This invention achieves the synergistic effect of multiple modal information by uniformly identifying and extracting identity ID data, sentiment tag data, text data, audio data, and dialogue action data, avoiding the information loss and inaccurate recognition problems caused by single modality processing. After extracting features for each modality using a one-hot encoding module, a FastText encoder, and a DA-BERT model, they are uniformly mapped to the same latent space, ensuring consistency in the representation dimension and distribution of different modal features, laying the foundation for subsequent fusion. A multi-condition encoder model encodes, transforms, and aligns the features of each modality, generating fused features, thus realizing... Deep interaction and supplementation of cross-modal information significantly improves the efficiency of multimodal fusion and the completeness of fused features. The multi-conditional cross-attention mechanism focuses on both local dependencies and global correlations when extracting fused features, enabling optimized features to capture fine-grained intermodal interactions while taking into account the global context, thus enhancing the model's ability to perceive complex speech scenarios. The speech actions generated based on optimized features are highly matched with the speech content, and are adjusted by combining the reconstruction loss function and the speech action speed loss function, achieving a dual improvement in the accuracy and fluency of generated actions, significantly improving the naturalness of human-computer dialogue interaction and user experience.

[0026] In this embodiment, the identification and extraction of multimodal data specifically includes: Identify and extract identity ID data and sentiment tag data from the speech scene; the identity ID data represents the speaker's unique identifier; the sentiment tag data represents the speaker's emotional state, and a sentiment analysis model is used to analyze the speech content and generate sentiment tags; Identify and extract text data from the speech content; the text data refers to the statements or content spoken by the speaker. Identify and extract audio data from the audio signal of the speech; the audio data includes the speaker's voice features; Identify and extract dialogue action data from the speaker's movements; the dialogue action data includes the speaker's body language.

[0027] By deploying cameras and sensing devices to detect and track the speaker's face, and combining this with facial recognition or smart tagging technology to acquire and encode identity ID data, a unique identifier for the speaker is achieved. Simultaneously, acoustic and facial features are extracted from the speaker's speech signal and facial expressions, respectively. A deep neural network emotion classifier and an expression analysis model are used to fuse multi-channel emotion prediction results, generating emotion tag data that accurately reflects the speaker's emotional state. Further, speech recognition is performed on the speech audio to obtain time-aligned text transcription. The text is then processed through sentence segmentation, noise reduction, and word segmentation to extract clean and consistent text data. At the audio data level, frame-level acoustic features such as Mel-frequency cepstral coefficients are extracted from the original audio stream, and these features are normalized and silence removed to obtain audio data reflecting the speaker's pitch, speech rate, and emotional changes. Positional information of key nodes such as the upper body, arms, and head is acquired using deep video or skeletal tracking devices. Data frames are continuously extracted according to a set frame rate, and a temporal masking completion algorithm is used to fill in missing frames or occlusions, generating complete and consistent dialogue action data. The multimodal data extraction method in this invention not only ensures the accuracy and consistency of various features, but also effectively resists interference such as noise and occlusion, laying a solid foundation for subsequent unified feature encoding and fusion processing, and improving the robustness and accuracy of speech behavior recognition and action generation.

[0028] The sentiment analysis model is primarily used to extract sentiment label data from the speaker's speech signals and facial expressions. Specifically, the speaker's audio signal is first framed, and acoustic features such as Mel-frequency cepstral coefficients, energy, pitch, and speech rate curves are extracted. Simultaneously, these acoustic features are input into a pre-trained deep neural network sentiment classifier. End-to-end acoustic sentiment recognition is performed through multi-layer convolutional and recurrent structures, outputting the sentiment probability distribution for each frame or each sentence. Further, facial image sequences captured by a camera are input into a facial expression analysis module based on keypoint detection and convolutional neural networks. By locating facial keypoints and extracting expression feature vectors, the corresponding expression sentiment category is output through a multi-layer perceptron. Finally, the acoustic sentiment probability distribution and the expression analysis results are weighted and fused according to preset weights to obtain the speaker's comprehensive sentiment label at that moment. This approach considers both emotional changes at the speech level and the intuitive perception of facial expressions, ensuring the accuracy and robustness of the sentiment label data.

[0029] In this embodiment, step two, which involves inputting the identity ID data and the sentiment tag into a one-hot encoding module to extract the encoded identity ID features and sentiment tag features, specifically includes: The identity ID data is input into the one-hot encoding unit of the one-hot encoding module, which maps each type of identity ID into a binary vector with "1" in the corresponding position and "0" in the rest. The binary vector is then mapped into a multi-dimensional identifier feature vector by the feature mapping unit of the one-hot encoding module. The identifier feature vector is used to represent the speaker's identity information. The emotion tag data is input into the same one-hot encoding module, and the one-hot encoding unit maps each emotion tag to a binary vector with "1" at the corresponding position and "0" at the rest. The binary vector is then mapped to a multi-dimensional emotion feature vector by the feature mapping unit. The emotion feature vector is used to characterize the speaker's emotion category.

[0030] First, the speaker's identity ID and sentiment label are input into the encoding unit of the same one-hot encoding module. Through category mapping, each identity or sentiment category is converted into a binary vector with "1" at the corresponding position and "0" at the rest. Simultaneously, the binary vector output by the encoding unit is converted into a fixed-dimensional continuous vector by the feature mapping unit, generating a multi-dimensional identifier feature vector representing the speaker's identity information and a multi-dimensional sentiment feature vector representing the speaker's sentiment category. Furthermore, the feature mapping uses the same parameterization structure to process both types of vectors, ensuring that identity and sentiment features have a consistent distribution and scale within the same feature space. This unified mapping process achieves efficient vectorization of discrete labels, enhancing the alignment and comparability between features during subsequent multimodal fusion, thereby significantly improving the model's recognition accuracy and fusion robustness for identity and sentiment information.

[0031] In this embodiment, step three: inputting the text data into the FastText encoder for dimensionality reduction and adjusting it using the DA-BERT model to extract text features, specifically: The text data is input into the FastText encoder and subjected to low-dimensional mapping to obtain the text features after preliminary dimensionality reduction; the text features are vector representations of the text content. The text features after initial dimensionality reduction are input into the DA-BERT model, and Masked Language Modeling and Next Sentence Prediction pre-training are performed sequentially. A cross-modal adapter is embedded after each Transformer layer of the DA-BERT model; the adapter includes sampling, activation, and upsampling structures. The final three self-attention sublayers of the DA-BERT model are replaced with a multi-conditional cross-attention mechanism to output text features from the DA-BERT model.

[0032] The multi-conditional cross-attention model of the MCE speech action generation model is still based on multi-head attention, and its formula is: ; ; in, Represents the multi-head attention function, Contact This means concatenating the attention results of all heads together, where Q, K, and V represent the query vector, key vector, and value vector, respectively. Let represent the output of the j-th attention head, `Attention` represent the computation of a single attention head, and `h` be the number of attention heads. The dimension of each head is... , Where is the number of modalities, and 32 is the fixed hidden dimension for each modality. This represents the output transformation matrix. Therefore, the formula for calculating attention is as follows:

[0033] ; in, This represents the local attention weight matrix. This represents the global attention weight matrix. And for... These are weights specific to multi-condition cross-attention, and their calculation formula is as follows:

[0034] ; ; Where Q represents the query vector, with dimension 1. , This represents the key vector in the nth local region. This represents the similarity between the query and the nth key region. This is a scaling factor to prevent the inner product result from becoming too large. By normalizing the attention weights within each region, Contact concatenates the attention weights of multiple local regions to obtain a complete local attention matrix.

[0035] In this embodiment, step four: inputting the audio data into the DA-BERT model for feature extraction and dimensionality reduction, specifically extracting audio features, involves: Audio data is input into the audio encoder of the DA-BERT model, which has been pre-trained for domain-adaptive DA-BERT model and pre-trained for the speech scenario using Masked Audio Modeling. A lightweight cross-modal adapter is embedded in each Transformer layer of the DA-BERT model to fuse prior information from textual or sentiment modalities. In the last three layers of the DA-BERT model, the standard self-attention sublayer is replaced with an audio-driven multi-head cross-attention sublayer; the multi-head cross-attention sublayer uses the text hidden vector as the query and the audio hidden vector as the key, and further aligns the temporal and emotional information of the text and audio through a multi-condition cross-attention mechanism. The DA-BERT model outputs a high-dimensional audio feature vector; the audio features include frequency, pitch, speech rate, and emotional change information, which are used to express the speaker's audio characteristics. The audio features are then subjected to dimensionality reduction processing and mapped to a low-dimensional feature space to obtain the dimensionality-reduced audio features.

[0036] This implementation uses a FastText encoder to perform initial dimensionality reduction on the text and combines it with a DA-BERT model that has been adaptively pre-trained on general corpora and speech-specific corpora. By utilizing cross-modal adapters and multi-conditional cross-attention mechanisms, the sensitivity of text features to speech context and emotional semantics is enhanced, achieving fine alignment between text and other modalities. Simultaneously, domain-adaptive pre-training and cross-modal adapters are also used in the audio channel, and audio-driven multi-conditional cross-attention is introduced at the end, making the audio features more accurate in expressing frequency, pitch, speech rate, and emotional changes, and maintaining temporal and semantic consistency with the text hidden vectors. The final output text and audio features have high contextual relevance and cross-modal complementarity, providing a solid and robust feature foundation for subsequent multimodal fusion and action generation.

[0037] In this embodiment, step five: extracting dialogue action features from the dialogue action data by extracting consecutive frames and performing masking and padding, specifically involves: The dialogue action data is continuously extracted using motion detection or keyframe extraction techniques, and the frame data related to the speech content is selected. The extracted continuous frame data is masked and padded to fill in missing or missing information, generating complete dialogue action features, and the dialogue action features are normalized.

[0038] This implementation combines continuous frame extraction with mask completion processing to effectively acquire and reconstruct the speaker's action information at key moments, improving the completeness and accuracy of action features. Motion detection or keyframe extraction techniques ensure that only frames closely related to the speech content are extracted, avoiding the introduction of noise from irrelevant frames. Simultaneously, a temporal mask-based completion algorithm is used to interpolate and fill in missing or occluded key points, achieving seamless recovery of missing action parts. Normalization processing is performed on the reconstructed consistent action sequence to unify the scale and distribution between different frames, ensuring that the generated dialogue action features have stable numerical characteristics and good alignment in subsequent multimodal fusion processes. This significantly improves the robustness of action recognition and the coherence of generated actions.

[0039] In this embodiment, step six: inputting the identity ID features, sentiment tag features, text features, audio features, and dialogue action features into the multi-condition encoder model, performing feature encoding, transformation, and alignment to obtain fused features, specifically: The identity ID features, sentiment tag features, text features, audio features, and dialogue action features are input into the MCE model, and linear projection is applied to each to map them to the same hidden dimension. Add conditional embedding vectors corresponding to each feature to identify the source modality, and add them to the projected features to generate conditional features; The conditional features are input in parallel to the Transformer encoder unit, each unit including a self-attention layer and a feedforward network, for extracting the interaction relationship between features in a global scope; In the attention calculation of each Transformer encoder unit, a multi-conditional cross-attention mechanism is adopted to fuse the conditional features of different modalities as queries, keys, and values ​​to complete the alignment and information supplementation between modalities. The conditional output after passing through the Transformer encoder unit is concatenated or weighted and summed, and a fusion feature is generated through a linear mapping layer. The fusion feature comprehensively represents the information of each modality and has an aligned and consistent representation.

[0040] This implementation significantly improves the interaction efficiency and complementary effects of multimodal information by uniformly encoding, conditionalizing, and aligning the features of each modality through a multi-conditional encoder model. Each modality feature is first mapped to a unified hidden dimension via linear projection, achieving consistency in the representation space for data from different sources. The conditional embedding vector is added to the projected features, providing an explicit identifier for each modality, enabling the encoder to distinguish and maintain the independent attributes of each modality. Conditional features are input in parallel into stacked Transformer encoder units, which extract the interaction relationships between modalities globally through self-attention layers and feedforward networks. Within the encoder, a multi-conditional cross-attention mechanism fuses queries, keys, and values ​​from different modalities, effectively aligning key information between modalities and supplementing complementary features. The fused features generated by concatenation or weighted summation followed by linear mapping not only comprehensively reflect the semantic and temporal attributes of each modality but also maintain high consistency in scale and distribution, providing a complete, accurate, and robust input foundation for subsequent speech action generation.

[0041] In this embodiment, step seven: extracting local and global information of the fused features through a multi-condition cross-attention mechanism to generate optimized features, specifically includes: The fused features are mapped to query, key, and value vectors respectively, and then replicated into several subspace representations through a multi-head attention mechanism. Attention weights are calculated using the query vector and the key vector of the corresponding header, and the value vector is weighted and summed to extract local dependency information. The outputs of all attention heads are concatenated and a global fusion is formed through linear mapping. By applying feedforward networks and residual connections to global fusion, optimized features containing both local and global information are obtained.

[0042] This implementation linearly maps the fused features into query, key, and value vectors, and replicates them into multiple subspace representations, thus supporting multi-head parallel computation. Each attention head calculates attention weights based on the query vector and the corresponding key vector, and performs a weighted summation on the value vector to obtain fine-grained local dependency information. The multi-head outputs are concatenated and linearly mapped to generate a global fused representation, ensuring that information from each subspace is effectively converged. A feedforward network and residual connections are used to perform nonlinear transformations and information preservation on the global fusion, generating optimized features containing both local and global semantic dependencies. This significantly enhances the model's hierarchical understanding of multimodal fused features while ensuring the stability and integrity of information transmission, providing high-quality optimized input for subsequent action generation.

[0043] In this embodiment, step eight: generating speech actions that match the speech content based on the optimized features, and adjusting the speech actions using a reconstruction loss function and a speech action speed loss function, specifically includes: Based on the optimized features input into the speech action generation module, speech actions that match the speech content are generated. In the generated speech actions, the difference between the generated actions and the target speech actions is calculated using the reconstruction loss function, and the actions are optimized accordingly. By using the speech action speed loss function, the deviation between the speed of the generated action and the actual speech rhythm is calculated, the speed of the action is adjusted, and the optimized speech action is output.

[0044] The formula for the reconstruction loss based on speech actions is:

[0045] Where Loss represents the total loss function during model training. This represents the Huber loss function, used to measure the positional difference between the generated action and the target action; This represents the velocity loss function, used to measure the deviation between the velocity variation of the generated motion across consecutive frames and the actual motion velocity; Used to control the impact on reconstruction accuracy Used to control the smoothness and consistency of the generated actions in terms of speed.

[0046] In addition, a loss function was designed to optimize the speed of speech gesture changes. This function is used to train the consistency between the generated gestures and real gestures and the semantics of the dialogue. It also helps to adjust the speed of the speech gestures generated by the model later based on the hardware conditions of the humanoid robot in human-computer interaction, such as the degrees of freedom of the hand joints. The formula is as follows: ; in, This represents the loss function, used to measure the difference between two vectors; This represents the velocity vector used to generate the action sequence. This represents the velocity vector of the actual motion sequence.

[0047] This implementation achieves a high degree of consistency between actions and semantic content by performing end-to-end modeling on the feature-driven speech action generation module. It also incorporates dual loss functions for fine-tuning the generated results. The reconstruction loss function accurately measures the differences between the generated actions and the actual speech actions in terms of keypoint positions and motion trajectories, guiding the model to continuously adjust and improve the accuracy of action reproduction. The speech action speed loss function compares the time dimension of action execution with the speech rate, quantifying the degree of deviation in speed to dynamically correct the action rhythm, ensuring that the output action remains natural in the spatial dimension and conforms to the speech rhythm requirements in the temporal dimension. This significantly improves the accuracy and fluency of action generation, providing a solid guarantee for high-quality human-computer dialogue interaction.

[0048] In this embodiment, it also includes: A training dataset was generated by collecting a dataset containing speech videos, audio, and text, and manually annotating it. Each data sample includes speech content, audio features, and related speech action labels. The training dataset is divided into a training set, a validation set, and a test set. The speech actions are evaluated using the validation set and the test set, and the parameters of the MCE model are adjusted based on the evaluation results.

[0049] This implementation constructs a multimodal labeled dataset containing speech videos, audio, and text, and accurately labels each sample with speech action tags, providing high-quality supervision information for MCE model training. The dataset is proportionally divided into training, validation, and test sets. The validation set is used to dynamically evaluate the model's performance on unseen data and guide hyperparameter optimization, while the test set is used for final performance verification, ensuring that the model not only possesses strong learning capabilities but also maintains stable recognition and generation effects in real-world scenarios. This significantly improves the model's generalization ability and robustness, laying a data foundation for efficient and reliable speech behavior recognition and action generation.

[0050] Example 1: To verify the feasibility of this invention in practice, it was applied to a "Future Speech Assistant" system test held in June 2025 at a university's intelligent interaction laboratory. The laboratory, approximately 150 square meters in size, was equipped with multiple 4K cameras, a 16-channel microphone array, and two depth skeletal tracking devices. All participants wore RFID tags to obtain their identity IDs. The test period ran from June 1st to June 15th, inviting 30 graduate students and 10 faculty members from different colleges as speakers. The topics covered included artificial intelligence, environmental protection, and educational innovation, with each speech lasting five to seven minutes. The experiment aimed to address the problems of low accuracy, unnatural motion generation, and insufficient real-time performance in traditional single-modal recognition. This invention, based on a multimodal fusion and multi-conditional cross-attention mechanism using the MCE model, achieved high-precision recognition of speech behavior sequences and natural, fluent motion generation.

[0051] In this scenario, the speaker's face is first detected and identified using RFID reading to obtain their ID data. Simultaneously, acoustic and facial emotion features are extracted from the acoustic signal and facial expression, respectively, and fused using a classification model to output emotion label data. Speech recognition is performed on the audio of the speech to obtain high-quality text transcription, which is then preliminarily dimensionality-reduced using FastText. Mel-frequency cepstral coefficients are extracted from the original audio frames, and silent segments are removed to generate audio features. Keyframes are continuously acquired using a skeletal tracking device, and a temporal masking algorithm is used to generate dialogue action features. These five types of features are then processed through one-hot encoding, FastText+DA-BERT, DA-BERT audio branching, action frame extraction, and masking, before being uniformly input into a multi-condition encoder model for encoding, transformation, and alignment. A multi-condition cross-attention mechanism is used to extract local and global information. Finally, the action generation module outputs an action sequence highly matching the speech content under constraints of reconstruction loss and velocity loss. The entire process runs in real-time on a laboratory server, with an average processing latency of 40 milliseconds per frame, meeting the requirements for real-time interaction.

[0052] Table 1 Evaluation Table of Multimodal Fusion Effect

[0053] To quantitatively evaluate the beneficial effects of this invention, three indicators were selected: behavior recognition accuracy, action naturalness score (out of 5), and system response latency. This invention was then compared with two control methods (text + audio fusion only, and traditional LSTM + rule engine method). Table 1, embedded in the above paragraphs, accurately reflects the overall performance of each configuration during the 15-day testing period.

[0054] Table 1, the multimodal fusion performance evaluation table, shows that the text and audio-based fusion method achieved an accuracy rate of 88.1% in behavior recognition across 90 presentations by 30 speakers, with an average action naturalness score of 3.3 and an average system response latency of 75 milliseconds. Adding emotion tags further improved the accuracy rate to 91.0%, the naturalness score to 3.7, and the latency to 68 milliseconds. By comprehensively employing five modal fusions—identity, emotion, text, audio, and action—the invention achieved an accuracy rate of 95.5%, an action naturalness score of 4.5, and a further reduction in system response latency to 42 milliseconds, fully demonstrating the advantages of deep multimodal fusion.

[0055] To further verify the improvement effect of this invention compared to the traditional LSTM+rule engine method, a "Performance Comparison Table of This Invention and Traditional Methods" was obtained on the same dataset. Table 2 shows that the comparison method has a recognition accuracy of only 82.5%, an action naturalness score of 3.0, and a system response latency of 110 milliseconds. In the same scenario, this invention achieves a recognition accuracy of 95.5%, an action naturalness score of 4.5, and a latency of 42 milliseconds, representing improvements of 13 percentage points, 1.5 points, and nearly 70 milliseconds respectively, significantly enhancing user experience and system reliability.

[0056] Table 2 Performance Comparison of the Invention and Traditional Methods

[0057] Analysis of the table above shows that this invention, through multimodal feature fusion based on the MCE model and multi-condition cross-attention mechanism, not only significantly improves the accuracy of speech behavior recognition and the naturalness of action generation, but also ensures the real-time response capability of the system, solving the technical bottleneck of traditional single-modal and shallow fusion methods in balancing accuracy and real-time performance.

[0058] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for speech behavior sequence recognition and human-computer dialogue action generation based on the MCE model, characterized in that, Includes the following steps: Step 1: Identify and extract multimodal data, which includes identity ID data, sentiment tag data, text data, audio data, and dialogue action data; Step 2: Input the identity ID data and the sentiment tag data into the one-hot encoding module to extract the encoded identity ID features and sentiment tag features; Step 3: Input the text data into the FastText encoder for dimensionality reduction, and then adjust it using the DA-BERT model to extract text features; Step 4: Input the audio data into the DA-BERT model for feature extraction and dimensionality reduction to extract audio features; Step 5: Extract dialogue action features by continuously extracting frames from the dialogue action data and performing masking and padding. Step Six: Input the identity ID features, sentiment tag features, text features, audio features, and dialogue action features into the multi-condition encoder model, perform feature encoding, transformation, and alignment, and obtain the fused features; Step 7: Extract local and global information of the fused features through a multi-condition cross-attention mechanism to generate optimized features; Step 8: Generate speech actions that match the speech content based on the optimized features, and adjust the speech actions using a reconstruction loss function and a speech action speed loss function.

2. The method for speech behavior sequence recognition and human-computer dialogue action generation based on the MCE model according to claim 1, characterized in that, The identification and extraction of multimodal data specifically includes: Identify and extract identity ID data and sentiment tag data from the speech scene; the identity ID data represents the speaker's unique identifier; the sentiment tag data represents the speaker's emotional state, and a sentiment analysis model is used to analyze the speech content and generate sentiment tags; Identify and extract text data from the speech content; the text data refers to the statements or content spoken by the speaker. Identify and extract audio data from the audio signal of the speech; the audio data includes the speaker's voice features; Identify and extract dialogue action data from the speaker's movements; the dialogue action data includes the speaker's body language.

3. The method for speech behavior sequence recognition and human-computer dialogue action generation based on the MCE model according to claim 2, characterized in that, Step two: Input the identity ID data and the sentiment tag into the one-hot encoding module to extract the encoded identity ID features and sentiment tag features, specifically: The identity ID data is input into the one-hot encoding unit of the one-hot encoding module, which maps each type of identity ID into a binary vector with "1" in the corresponding position and "0" in the rest. The binary vector is then mapped into a multi-dimensional identifier feature vector by the feature mapping unit of the one-hot encoding module. The identifier feature vector is used to represent the speaker's identity information. The emotion tag data is input into the same one-hot encoding module, and the one-hot encoding unit maps each emotion tag to a binary vector with "1" at the corresponding position and "0" at the rest. The binary vector is then mapped to a multi-dimensional emotion feature vector by the feature mapping unit. The emotion feature vector is used to characterize the speaker's emotion category.

4. The method for speech behavior sequence recognition and human-computer dialogue action generation based on the MCE model according to claim 3, characterized in that, Step three: The text data is input into the FastText encoder for dimensionality reduction, and then adjusted using the DA-BERT model to extract text features, specifically: The text data is input into the FastText encoder and subjected to low-dimensional mapping to obtain the text features after preliminary dimensionality reduction; the text features are vector representations of the text content. The text features after initial dimensionality reduction are input into the DA-BERT model, and Masked Language Modeling and Next Sentence Prediction pre-training are performed sequentially. A cross-modal adapter is embedded after each Transformer layer of the DA-BERT model; the adapter includes sampling, activation, and upsampling structures. The final three self-attention sublayers of the DA-BERT model are replaced with a multi-conditional cross-attention mechanism to output text features from the DA-BERT model.

5. The method for speech behavior sequence recognition and human-computer dialogue action generation based on the MCE model according to claim 4, characterized in that, Step four: Input the audio data into the DA-BERT model for feature extraction and dimensionality reduction. Specifically, the audio features are extracted as follows: Audio data is input into the audio encoder of the DA-BERT model, which has been pre-trained for domain-adaptive DA-BERT model and pre-trained for the speech scenario using Masked Audio Modeling. A lightweight cross-modal adapter is embedded in each Transformer layer of the DA-BERT model to fuse prior information from textual or sentiment modalities. In the last three layers of the DA-BERT model, the standard self-attention sublayer is replaced with an audio-driven multi-head cross-attention sublayer; the multi-head cross-attention sublayer uses the text hidden vector as the query and the audio hidden vector as the key, and further aligns the temporal and emotional information of the text and audio through a multi-condition cross-attention mechanism. The DA-BERT model outputs a high-dimensional audio feature vector; the audio features include frequency, pitch, speech rate, and emotional change information, which are used to express the speaker's audio characteristics. The audio features are then subjected to dimensionality reduction processing and mapped to a low-dimensional feature space to obtain the dimensionality-reduced audio features.

6. The method for speech behavior sequence recognition and human-computer dialogue action generation based on the MCE model according to claim 5, characterized in that, Step five: Extracting dialogue action features from the dialogue action data by extracting consecutive frames and performing masking and padding, specifically: The dialogue action data is continuously extracted using motion detection or keyframe extraction techniques, and the frame data related to the speech content is selected. The extracted continuous frame data is masked and padded to fill in missing or missing information, generating complete dialogue action features, and the dialogue action features are normalized.

7. The method for speech behavior sequence recognition and human-computer dialogue action generation based on the MCE model according to claim 6, characterized in that, Step six: Input the identity ID features, sentiment tag features, text features, audio features, and dialogue action features into the multi-condition encoder model, perform feature encoding, transformation, and alignment to obtain fused features, specifically as follows: The identity ID features, sentiment tag features, text features, audio features, and dialogue action features are input into the MCE model, and linear projection is applied to each to map them to the same hidden dimension. Add conditional embedding vectors corresponding to each feature to identify the source modality, and add them to the projected features to generate conditional features; The conditional features are input in parallel to the Transformer encoder unit, each unit including a self-attention layer and a feedforward network, for extracting the interaction relationship between features in a global scope; In the attention calculation of each Transformer encoder unit, a multi-conditional cross-attention mechanism is adopted to fuse the conditional features of different modalities as queries, keys, and values ​​to complete the alignment and information supplementation between modalities. The conditional output after passing through the Transformer encoder unit is concatenated or weighted and summed, and a fusion feature is generated through a linear mapping layer. The fusion feature comprehensively represents the information of each modality and has an aligned and consistent representation.

8. The method for speech behavior sequence recognition and human-computer dialogue action generation based on the MCE model according to claim 7, characterized in that, Step seven: Extracting local and global information from the fused features using a multi-condition cross-attention mechanism to generate optimized features, specifically: The fused features are mapped to query, key, and value vectors respectively, and then replicated into several subspace representations through a multi-head attention mechanism. Attention weights are calculated using the query vector and the key vector of the corresponding header, and the value vector is weighted and summed to extract local dependency information. The outputs of all attention heads are concatenated and a global fusion is formed through linear mapping. By applying feedforward networks and residual connections to global fusion, optimized features containing both local and global information are obtained.

9. The method for speech behavior sequence recognition and human-computer dialogue action generation based on the MCE model according to claim 8, characterized in that, Step eight: Based on the optimized features, generate speech actions that match the speech content, and adjust the speech actions using a reconstruction loss function and a speech action speed loss function, specifically as follows: Based on the optimized features input into the speech action generation module, speech actions that match the speech content are generated. In the generated speech actions, the difference between the generated actions and the target speech actions is calculated using the reconstruction loss function, and the actions are optimized accordingly. By using the speech action speed loss function, the deviation between the speed of the generated action and the actual speech rhythm is calculated, the speed of the action is adjusted, and the optimized speech action is output.

10. The method for speech behavior sequence recognition and human-computer dialogue action generation based on the MCE model according to claim 9, characterized in that, Also includes: A training dataset was generated by collecting a dataset containing speech videos, audio, and text, and manually annotating it. Each data sample includes speech content, audio features, and related speech action labels. The training dataset is divided into a training set, a validation set, and a test set. The speech actions are evaluated using the validation set and the test set, and the parameters of the MCE model are adjusted based on the evaluation results.

Images