Multi-modal human intention understanding system based on brain cognition inspiration

Abstract

The invention relates to a multi-modal human intention understanding system based on brain cognition inspiration, which comprises a multi-modal information acquisition unit, an intercrossing attention enhancement unit, a self-crossing attention enhancement unit, a source modal updating unit, a self-adaptive multi-modal information fusion unit and a continuous-dimension emotional intention understanding unit, and a fine-grained behavior intention understanding unit. Compared with the prior art, the method has the advantages that effective feature semantics from language, audio and video modalities are fully fused, efficient multi-modal modeling is realized by utilizing attention mechanisms in different strengthening and updating units, and human emotion and behavior intention understanding in a multi-task mode is further completed; the problems of low human intention understanding accuracy, insufficient multi-mode utilization rate, poor system robustness and the like in a real application scene are effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of multimodal intention understanding, and in particular to a multimodal human intention understanding system based on brain cognition inspiration. Background Art

[0002] Current multimodal intent understanding technology can be summarized from three dimensions: modal type, recognition target, and modeling method.

[0003] Thanks to the rapid development of multimedia, most previous methods focus on extracting clues related to intent from non-language modalities such as audio and video. Audio signals contain important parameters such as timbre characteristics and sound quality coefficients that express human intent, while video signals are mainly used to extract intent semantics related to facial action units; in addition, there is also a part of research based on audio signals, using automatic translation and transcription tools to obtain text information corresponding to the audio modality to extract the intent representation contained in human language.

[0004] From the perspective of recognition targets, some work focuses on multimodal human emotional intention recognition. As the basis of human-computer interaction, emotion plays an important role in promoting the system to understand human expressions. Current emotional intention understanding mainly focuses on the recognition of discrete emotional expressions, such as emotional states such as happiness, sadness, anger, and rage. In terms of behavioral intention understanding, multimodal behavior recognition is a technology that accurately recognizes and understands human behavior by combining multiple perceptual modalities. Traditional behavior recognition methods are mainly based on a single perceptual modality, such as using only images or videos for behavior analysis. Multimodal behavior recognition uses data from multiple perceptual modalities, such as images and voice, to obtain a more comprehensive and accurate description of behavior.

[0005] Modeling methods are a key link in multimodal human intent understanding. By extracting effective features from each perceptual modality, human behavior can be better described. For image / video modalities, commonly used feature extraction methods include convolutional neural networks and long short-term memory networks; for speech and language modalities, recurrent neural networks are commonly used feature extraction methods. The intent understanding model is a key component for identifying human intent after feature extraction. Common intent understanding models include support vector machines (SVM), random forests, and deep neural networks.

[0006] Although current multimodal human intent understanding technology has made great progress, there are still some shortcomings and challenges that need to be addressed:

[0007] Most current research focuses on the combination of two modalities and ignores the complementary advantages of the three modalities. In the dual-modal approach, when the data of one modality is noisy or occluded, the other modalities may not be able to find the corresponding remaining modal semantics to perform modal alignment and provide useful information for intent understanding. The current situation of insufficient modal utilization greatly reduces the adaptability of the intent understanding system to interference and changes, as well as the stability and accuracy of the system.

[0008] In addition, previous intention understanding research has either focused on emotion recognition in discrete dimensions or on coarse-grained behavioral intention understanding, which cannot fully combine the complementary information and semantics brought by different intention understanding tasks, resulting in system performance bottlenecks and limiting the scope of application of the system. At the same time, in single-task learning, the system can only use data from specific tasks for training, while ignoring data from other tasks. This leads to inefficient use of data, especially when data is scarce, and the available information may not be fully utilized. Single-task intention understanding also cannot share learned knowledge. This means that a new model needs to be trained from scratch on each task without benefiting from the learning gains of other tasks. This increases the training time and computing resource overhead. Most seriously, single-task intention understanding systems lack generalization capabilities, that is, systems trained by single-task learning usually perform well on specific tasks, but may lack generalization capabilities when faced with new tasks. Since single-task systems only focus on the characteristics and goals of specific tasks, they may not be well adapted to the needs of new tasks and require retraining or a lot of adjustments.

[0009] In terms of modeling methods, traditional machine learning-based methods usually require manual feature engineering, that is, extracting useful features from raw data for model learning and prediction. This requires domain expertise and experience, and is often a time-consuming and complex process. At the same time, these models have limited learning and expression capabilities and may not be able to handle complex nonlinear relationships and large-scale data well. When outliers and noise appear in the data, this may lead to a decline in model performance. In addition, most current deep learning methods usually use convolutional neural networks and recurrent neural networks to process multimodal asynchronous sequences. These models cannot learn more contextual information due to their limited receptive fields. In addition, they usually cannot achieve efficient parallel computing, which limits the speed of training and inference. In feature learning of heterogeneous sequences, traditional methods usually need to design different models for heterogeneous modalities to extract modality-specific features, which increases computational overhead and system complexity.

[0010] In summary, how to develop a new system that can simultaneously accommodate human multimodal sequences of language, audio, and video and realize the understanding of human emotions and behavioral intentions with an efficient structural framework has become an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0011] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a multimodal human intention understanding system based on brain cognition inspiration, which effectively solves the problems of low accuracy of human intention understanding in real application scenarios, insufficient multimodal utilization and poor system robustness.

[0012] The purpose of the present invention can be achieved by the following technical solutions:

[0013] The present invention provides a multimodal human intention understanding system based on brain cognition inspiration, comprising:

[0014] A multimodal information acquisition unit, used to acquire multimodal information including language modality, audio modality and video modality;

[0015] A cross-attention enhancement unit is used to perform mutual attention and cross-attention interaction operations on the multimodal information collected by the multimodal information collection unit to obtain multimodal enhancement features;

[0016] A self-cross attention enhancement unit is used to enhance the multimodal information collected by the multimodal information collection unit through self-attention and cross-attention to obtain multimodal enhanced features;

[0017] A source modality updating unit, used for filtering and updating the multimodal information output by the multimodal information acquisition unit, and for progressively strengthening and enhancing the mutual cross-attention strengthening unit and the self-cross-attention strengthening unit;

[0018] An adaptive multimodal information fusion unit, used to adaptively fuse the multimodal reinforcement features and the multimodal enhancement features using a shared mapping matrix, and output a multimodal fusion feature;

[0019] The continuous dimension emotional intention understanding unit is used to regress the multimodal fusion features into emotional scores and output the final emotional prediction value;

[0020] The fine-grained behavior intention understanding unit is used to perform multi-classification operations on multimodal fusion features and output the final behavior prediction value.

[0021] Preferably, the multimodal information acquisition unit includes independently arranged:

[0022] Language modality extraction module: Translates and transcribes the video signal and uses the pre-trained GloVe word embedding processing model to obtain a language feature sequence;

[0023] Audio modality extraction module: extracts low-order acoustic features through the COVAREP tool to obtain an audio feature sequence;

[0024] Video modality extraction module: extracts facial action units as video feature sequences through the Facet tool.

[0025] Preferably, the cross-attention reinforcement unit comprises:

[0026] Multimodal feature sequence projection module: three independent multi-layer perceptron models are used to project the language feature sequence, audio feature sequence, and video feature sequence, respectively, and then three one-dimensional convolutional neural network layers are used to align the multimodal feature sequences of different dimensions to the same dimension;

[0027] Multimodal joint mutual attention operation module: The multimodal feature sequence features after the three projection transformations after the alignment dimension are concatenated and normalized, and then the attention scores calculated by segmentation are obtained according to the length of the original multimodal feature sequence to obtain a modality-specific attention matrix, which is used to perform element-level element multiplication operations with the original multimodal features to refine the original features and obtain the first refined features;

[0028] Multimodal independent cross-attention interaction module: A single modality is projected into a specific modality feature sequence space as index access through a fully connected layer, and then two fully connected layers with similar structures but different weights are used to project the feature sequences of the other two modalities into a specific source sequence space as index keys and index values. Different modalities perform target index access and cross-modal interaction of source index keys and values ​​respectively to mine the correlation between cross-modal elements and achieve feature refinement, thus obtaining the second refined features.

[0029] Feature merging module: merges the first refined features and the second refined features based on element and level addition, and outputs multimodal enhanced features.

[0030] Preferably, the self-cross attention strengthening unit comprises:

[0031] Multimodal joint representation stage construction module: Different multimodal features are normalized through the modality-specific fully connected layer, and then the multimodal joint representation is constructed by the feature-level cascade operation. The multimodal joint representation is split into two parts, which are used for downstream self-attention enhancement and cross-attention enhancement respectively;

[0032] Self-attention enhancement module: The multimodal joint representation performs self-enhancement of the elements at each timestamp through feature joint projection and self-attention operation to obtain the first enhanced feature;

[0033] Cross-attention enhancement module: The multimodal joint representation extracts feature prototypes from three heterogeneous interaction spaces through cross-variant feature joint projection, performs cross-enhancement of cross-modal elements and generates new knowledge through mutation, and obtains the second enhanced feature.

[0034] Feature integration module: concatenates the first enhanced feature and the second enhanced feature to output multimodal enhanced features.

[0035] Preferably, the multimodal joint representation in the cross-attention enhancement module extracts feature prototypes of three heterogeneous interaction spaces through cross-variant feature joint projection, cross-enhancement of cross-modal elements and mutation induction of new knowledge, and obtains the second enhanced feature. Specifically, cross-modal element cross-enhancement and mutation induction of new knowledge are performed through cross-modal matrix multiplication and cross-softmax function between any modal feature and the joint feature. The multimodal joint representation performs element adaptation and cross-modal element enhancement from one modality to another at each timestamp through feature joint projection and cross-attention operation to obtain the second enhanced feature.

[0036] Preferably, the source modality updating unit simulates the information updating mechanism of neurons stored in the cerebral cortex to perform knowledge exchange and information filtering, specifically including:

[0037] Feature update module: uses a feedforward neural network to update multimodal heterogeneous representation features;

[0038] Information filtering module: Filter redundant information and noise information of multimodal heterogeneous representation features to obtain feature matrix, perform element-level feature filtering multiplication on the filtered feature matrix and the updated features, and output multimodal reinforcement features and multimodal enhanced features respectively, which act on the cross-attention reinforcement unit and the self-cross-attention reinforcement unit.

[0039] Preferably, the adaptive multimodal information fusion unit comprises:

[0040] Adaptive weight building module: Initialize the shared mapping matrix, and then different reinforcement and enhancement features are combined with the shared mapping matrix to produce feature-specific adaptive weight coefficients;

[0041] Dynamic weighted fusion module: Based on the adaptive weight coefficient, element-wise multiplication is used to perform weighted summation of different importance coefficients and enhanced features;

[0042] Multimodal fusion feature output module: outputs the final refined multimodal fusion features.

[0043] Preferably, the emotion intention understanding unit of the continuous dimension is specifically as follows: the multimodal fusion features are transformed into emotion score variables through emotion-specific projection transformation, and then the mean square error loss is used to calculate the sum of the squares of the differences between the emotion score variables and the emotion target values ​​to train the network model and obtain the final emotion prediction value.

[0044] Preferably, the fine-grained behavior intention understanding unit is specifically as follows: the multimodal fusion features are transformed into behavior score variables through behavior-specific projection transformation, and then the difference between the behavior score variable and the behavior target value is calculated using the cross entropy loss, and the network model is trained through the constraint of maximizing the category probability to obtain the final behavior prediction value.

[0045] Preferably, the system further comprises an intention display unit for displaying and presenting the predicted human emotions and behavioral intentions in a visual manner.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1) Different from the traditional multimodal intent understanding system, the present invention does not require additional human resource consumption and feature engineering overhead. By simulating the information processing paradigm of different neurons and synapses in the human cerebral cortex, it realizes efficient and robust multimodal feature extraction and human intent understanding in the form of combining the advantages of human perception. Specifically, inspired by the fine-grained signal fusion mechanism of multisensory neurons in the cerebral cortex, a mutual cross-attention reinforcement unit is designed in the system to realize fine-grained mutual information extraction and cross-information interaction; inspired by the coarse-grained signal fusion mechanism of multisensory neurons in the cerebral cortex, a self-cross-attention reinforcement unit is designed in the system to realize coarse-grained self-information reinforcement and cross-information iteration; inspired by the information update mechanism of storage neurons in the cerebral cortex, the source modality update unit is used to realize information filtering and knowledge exchange, and provide updated high-order information semantics for different reinforcement units to complete the further reinforcement and enhancement of features.

[0048] 2) Different from the traditional attention mechanism that can only realize element enhancement on the source modal feature sequence, the present invention involves a variety of beneficial new attention mechanisms, including mutual attention, self-attention and cross-attention. The mutual attention included in the system fully considers the different contributions of different heterogeneous modalities in the understanding of intentions in real scenes, provides more valuable weights for strongly related modalities, and suppresses the expressive power of weakly related modal features. The self-attention included in the system progressively learns the temporal contextual dependencies on the target feature sequence, laying a solid learning foundation for strengthening the effective semantics of multimodal features in the time dimension. The cross-attention included in the system effectively captures the correlation between cross-modal elements, promotes feature adaptation and enhancement of expressive power between heterogeneous modalities, and provides a new learning paradigm for generating more robust cross-modal semantics.

[0049] 3) Different from the heuristic post-fusion method in traditional feature fusion, the present invention proposes an adaptive multimodal information fusion unit. This unit simulates the information fusion strategy of neural synapses in the cerebral cortex, realizes the dynamic convergence and integration of multimodal features from different reinforcement sources, and explicitly enhances the utilization rate of multimodal features and the recognition ability of intent understanding.

[0050] 4) Different from the traditional human intention understanding system that can only adapt to specific intention understanding tasks, the present invention fully integrates the advantages of multi-task learning, realizes the simultaneous prediction and recognition of human emotional intention and behavioral expression with a unified system architecture, enhances the sharing ability and complementarity of multimodal knowledge in different tasks, and improves the generalization, reliability and accuracy of the system.

[0051] 5) The present invention overcomes the defect of insufficient emotional representation space in previous discrete emotional intention understanding, and novelly designs a model for predicting changes in continuous emotional dimensions to fully capture the evolution and expression process of emotions in emotional representation, capture more emotional semantics in limited data resources, and achieve more universal and representative emotional representation in real scenarios.

[0052] 6) The present invention overcomes the defect that the behavior representation in the previous coarse-grained behavior intention understanding has a local optimal solution, expands the traditional behavior representation space to the multi-dimensional space, realizes fine-grained behavior recognition and intention understanding in a spatial representation decoupling mode, promotes the acquisition of the global optimal solution of behavior representation in the multi-dimensional representation space, and improves the system's robustness and intention understanding ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A principle block diagram of a multimodal human intention understanding system based on brain cognition inspiration provided by the present invention;

[0054] Figure 2 The principle block diagram of the multimodal information acquisition unit provided by the present invention;

[0055] Figure 3 A block diagram of the principle of the cross-attention reinforcement unit provided by the present invention;

[0056] Figure 4 The principle block diagram of the self-cross attention reinforcement unit provided by the present invention;

[0057] Figure 5 A principle block diagram of a source modality updating unit provided by the present invention;

[0058] Figure 6 The principle block diagram of the adaptive multimodal information fusion unit provided by the present invention;

[0059] Figure 7A principle block diagram of the continuous dimension emotional intention understanding unit provided by the present invention;

[0060] Figure 8 A principle block diagram of the fine-grained behavioral intent understanding unit provided by the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0062] Example

[0063] like Figure 1 As shown, this embodiment provides a multimodal human intention understanding system based on brain cognition inspiration, including:

[0064] 1) Multimodal information acquisition unit: Automatically collects asynchronous multimodal sequences from humans in real time, including human language, audio, and video sequences, and embeds features of different modalities through preliminary feature preprocessing and modality-specific tools;

[0065] 2) Cross-attention reinforcement unit and self-cross-attention reinforcement unit: Multi-granular information interaction is carried out in two attention-related reinforcement units to mine the temporal dependencies within the modalities and capture the cross-modal element correlations between the modalities to achieve full multimodal representation reinforcement and enhancement.

[0066] 3) Source modality update unit: It uses feature updating and information filtering strategies to refine and improve multimodal features, while removing noise and redundant information, and providing high-order and rich multimodal supplementary features for the two attention enhancement units.

[0067] 4) Adaptive multimodal information fusion unit: realizes dynamic fusion and aggregation of fine-grained and coarse-grained multimodal representations.

[0068] 5) Continuous dimension emotion intention understanding unit and fine-grained behavior intention understanding unit: In the two intention understanding units, on the one hand, the system realizes continuous emotion polarity prediction from -3 to +3 to represent the complete emotion change space; on the other hand, the behavior space is mapped into a high-dimensional space through the decoupled representation learning model to represent the fine-grained human behavior categories containing 300 categories, thereby achieving full intention understanding.

[0069] 6) Intention display unit: displays the results of emotion and behavioral intention understanding.

[0070] Next, combine the attached Figures 2 to 8 Provide a detailed introduction to each unit.

[0071] Figure 2 The block diagram of the multimodal information acquisition unit is shown in Figure 1. The unit contains three independent human multimodal extraction units, namely language, audio and video modalities. In the human language modality extraction unit, the language modality is first obtained by translating and transcribing the video signal, and then the automatic transcription model is optimized and trained to reduce the transcription error rate of the text modality and improve the quality of the generated text. The high-quality text modality is converted into a 300-dimensional feature vector by the pre-trained Glove word embedding. In the human audio modality extraction unit, the system extracts the audio modality into a 74-dimensional low-order acoustic feature representation through the COVAREP tool, which includes 12Mel-frequency cepstral coefficients (MFCCs), voiced / voiced segmentation features, and glottal source parameters. In the human video modality extraction unit, the system extracts 35-dimensional facial action units through the Facet tool, which includes muscle movement representations that express human emotions and behavioral intentions in each frame. Through the above feature extraction process, the system automatically completes the feature embedding of language, audio and video.

[0072] Figure 3The schematic diagram of the cross-attention enhancement unit is shown in Figure 1. The unit includes language, audio, and video feature sequence projection, multimodal feature alignment, joint mutual attention operation, independent cross-attention interaction, feature merging, and multimodal enhancement features. Specifically, the enhancement unit simulates the fine-grained signal fusion mechanism of multisensory neurons in the cerebral cortex to achieve fine-grained information interaction and feature enhancement. The heterogeneous multimodal sequence first projects the low-order multimodal representation to the high-order feature space through three independent multi-layer perceptron models to achieve preliminary preprocessing and refinement of the features. Next, the system includes three modality-specific one-dimensional convolutional neural network modules to achieve feature transformation alignment of heterogeneous modal sequences, where the convolution kernel sizes of language, audio, and video modalities are {3,3,5}, respectively, and the aligned features are normalized to 40 dimensions. In the joint mutual attention operation, the aligned representation is subjected to feature concatenation operation on the sequence dimension to perform joint modeling of features. Subsequently, the system includes a softmax function to achieve feature weight normalization on the sequence dimension. The features on all sequences are normalized to between 0 and 1 to eliminate the heterogeneity and dimensionality of features from different modalities. Then, the system divides the obtained normalized attention scores by the length of the original different multimodal sequences to obtain a modality-specific attention score matrix. The attention matrix implements semantic optimization of the original different modal features through element-wise multiplication operations. In the independent cross-attention unit branch, the system first learns different target index accesses for the three modalities through three target fully connected layers. This process is projected into a high-dimensional target feature space and implemented using multi-head split attention to enhance the multi-dimensional expression ability of the features. In addition, in the feature transformation of each modality, two fully connected layers with the same structural pattern but completely independent parameters simultaneously generate index keys and index values ​​in the source sequence space. Based on the modality-specific index access and key pairs, the system automatically performs cross-modal attention interactions between different modalities and obtains an attention mask matrix with predictive properties in the form of matrix multiplication. This matrix is ​​further used to perform matrix multiplication with the source index value to obtain an enhanced high-level multimodal feature representation. In this process, the system captures the potential beneficial correlations between cross-modal elements, promoting feature adaptation and subsequent feature fusion between cross-modalities. In the feature merging stage, the refined features obtained from mutual attention and cross-attention interactions are used for element-by-element and level-by-level addition operations to achieve feature merging, ultimately generating multi-modal enhanced features.

[0073] Figure 4This is the principle block diagram of the self-cross-attention enhancement unit, which includes parallel multimodal joint representation, self-attention enhancement, cross-attention enhancement, feature integration, and multimodal enhancement feature stages. Specifically, the enhancement unit simulates the coarse-grained signal fusion mechanism of multisensory neurons in the cerebral cortex to achieve coarse-grained self-information refinement and cross-information enhancement. The system first realizes the standard normalization of the multimodal feature dimensions through the self-enhanced dense connection layer and the cross-enhanced dense connection layer, and then completes the construction of the multimodal joint representation through cascade operations. The multimodal joint representation is divided into two parts to achieve downstream self-attention enhancement and cross-attention enhancement. In the self-attention enhancement stage, the multimodal joint representation fully brings together the advantages of different modal representations, realizes the joint projection transformation of the feature representation in a joint decision-making manner, and maps it to three different interactive feature spaces.

[0074] Subsequently, the system obtains the self-enhanced attention importance matrix by performing matrix multiplication and softmax function operations in the first two feature spaces. The matrix is ​​multiplied by the matrix of the remaining joint feature space with the secondary transformation to fully learn the element dependencies in the temporal order within the modality, completing the efficient self-enhancement of the intra-modal features; in the cross-attention enhancement stage, the multimodal joint representation realizes the feature prototype extraction of three heterogeneous interactive spaces through the cross-variant feature joint projection. The cross-modal element cross-enhancement and the mutation of new knowledge are realized through the cross-modal matrix multiplication and cross-softmax function between any modal feature and the joint feature. In this process, the system progressively enhances the cross-modal element adaptation and effective cross-modal prototype enhancement from one modality to another modality at each sequence timestamp. Then, in the feature integration stage, the system completes the convergence and integration of the enhanced features from different attention modes through the feature splicing technology to generate multi-modal enhanced features.

[0075] Figure 5 This is the principle block diagram of the source modality update unit, which includes several stages: feature update, information filtering, multimodal reinforcement, and enhanced feature output. Specifically, the unit simulates the information update mechanism of neurons stored in the cerebral cortex to achieve knowledge exchange and information filtering. In the feature update stage of this unit, the feedforward neural network module is first used to realize the feature update and semantic refinement of multimodal heterogeneous representation, promoting the transformation and induction of low-order multimodal semantics to high-order rich semantics. Then, the system uses the information filtering module to filter the potential redundant and noisy information in the multimodal information, retaining valuable information semantics. This process is completed simultaneously through Sigmoid and Tanh nonlinear activations, and the filtered feature matrix is ​​multiplied with the updated features at the element level, and the multimodal reinforcement features and multimodal enhancement features are output respectively, which are used for the progressive reinforcement and enhancement of the output features of the cross-attention reinforcement unit and the self-cross-attention reinforcement unit.

[0076] Figure 6 This is a block diagram of the principle of the adaptive multimodal information fusion unit, which includes the stages of adaptive weight construction, dynamic weighted fusion, and multimodal fusion feature output. Specifically, the unit simulates the information fusion strategy of neural synapses in the cerebral cortex to achieve information convergence and fusion in an efficient and unified mode. In this unit, a shared mapping matrix is ​​first initialized. On this basis, different reinforcement and enhancement features are combined with the shared mapping matrix to generate feature-specific adaptive weight coefficients. Then the system uses element multiplication to achieve the weighted summation of different importance coefficients and reinforcement features to produce the final refined multimodal fusion features.

[0077] Figure 7 This is a block diagram of the principle of the emotion intention understanding unit in the continuous dimension. The unit includes multimodal fusion feature transformation, emotion score regression training, error amplitude calculation, and regression prediction output stages. The system first transforms the multimodal fusion features into emotion score variables through the emotion-specific cascade mapping projection transformation. Then the system uses the mean square error loss to calculate the sum of the squares of the difference between the emotion score variable and the emotion target value, making the model more robust to outliers, thereby training the network model based on the regression score to obtain the final emotion prediction value.

[0078] Figure 8 This is a block diagram of the principle of the fine-grained behavior intention understanding unit, which includes multimodal fusion features, multi-classification training, category probability maximization, and classification prediction output stages. The system transforms the multimodal fusion features into behavior score variables through behavior-specific projection transformations, and then uses cross-entropy loss to calculate the difference between the behavior score variable and the behavior target value. Through the category probability maximization constraint, the probability of occurrence of each fine-grained behavior category is determined, and the classification-based network model training is realized to obtain the final behavior prediction value.

[0079] After obtaining the predicted information through the emotion and behavioral intention understanding unit, the results are further displayed and presented in a visual manner in the display unit.

[0080] The multimodal human intention understanding system based on brain cognition inspiration demonstrated in this embodiment draws on the efficient multimodal information processing paradigm in the human cerebral cortex for the first time, and realizes multi-granular information interaction and cross-modal element correlation learning through two complementary multimodal feature enhancement units. At the same time, the source modality update unit included in the system provides reliable and refined high-level feature semantics for the progressive feature interaction in the feature enhancement component. The system fully gathers the perceptual advantages of different features through the adaptive multimodal information fusion unit proposed for the first time, enhances the recognition accuracy of multimodal emotion and behavioral intention understanding in a multi-task learning manner, and improves the robustness, generalization and reliability of the model. The intention understanding system proposed in the present invention can provide rich multimodal feature semantics and a complete human intention representation space, providing a reliable technical guarantee for the application of human-computer interaction and virtual reality scenarios in the real world.

[0081] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multimodal human intention understanding system based on brain cognition inspiration, characterized in that: include: A multimodal information acquisition unit, used to acquire multimodal information including language modality, audio modality and video modality; A cross-attention enhancement unit is used to perform mutual attention and cross-attention interaction operations on the multimodal information collected by the multimodal information collection unit to obtain multimodal enhancement features; A self-cross attention enhancement unit, used for performing self-attention and cross-attention enhancement on the multimodal information collected by the multimodal information collection unit to obtain multimodal enhanced features; A source modality updating unit, used for filtering and updating the multimodal information output by the multimodal information acquisition unit, and for progressively strengthening and enhancing the mutual cross-attention strengthening unit and the self-cross-attention strengthening unit; An adaptive multimodal information fusion unit, used to adaptively fuse the multimodal reinforcement features and the multimodal enhancement features using a shared mapping matrix, and output a multimodal fusion feature; The continuous dimension emotion intention understanding unit is used to regress the emotion scores of multimodal fusion features and output the final emotion prediction value; The fine-grained behavior intention understanding unit is used to perform multi-classification operations on multimodal fusion features and output the final behavior prediction value.

2. According to claim 1, a multimodal human intention understanding system based on brain cognition inspiration is characterized in that: The multimodal information acquisition unit includes independently arranged: Language modality extraction module: Translates and transcribes the video signal and uses the pre-trained GloVe word embedding processing model to obtain a language feature sequence; Audio modality extraction module: extracts low-order acoustic features through the COVAREP tool to obtain an audio feature sequence; Video modality extraction module: extracts facial action units as video feature sequences through the Facet tool.

3. According to claim 1, a multimodal human intention understanding system based on brain cognition inspiration is characterized in that: The cross-attention reinforcement unit comprises: Multimodal feature sequence projection module: three independent multi-layer perceptron models are used to project the language feature sequence, audio feature sequence, and video feature sequence, respectively, and then three one-dimensional convolutional neural network layers are used to align the multimodal feature sequences of different dimensions to the same dimension; Multimodal joint mutual attention operation module: The multimodal feature sequence features after the three projection transformations after the alignment dimension are concatenated and normalized, and then the attention scores calculated by segmentation are obtained according to the length of the original multimodal feature sequence to obtain a modality-specific attention matrix, which is used to perform element-level element multiplication operations with the original multimodal features to refine the original features and obtain the first refined features; Multimodal independent cross-attention interaction module: A single modality is projected into a specific modality feature sequence space as index access through a fully connected layer, and then two fully connected layers with similar structures but different weights are used to project the feature sequences of the other two modalities into a specific source sequence space as index keys and index values. Different modalities perform target index access and cross-modal interaction of source index keys and values ​​respectively to mine the correlation between cross-modal elements and achieve feature refinement, thus obtaining the second refined features. Feature merging module: merges the first refined features and the second refined features based on element and level addition, and outputs multimodal enhanced features.

4. According to claim 1, a multimodal human intention understanding system based on brain cognition inspiration is characterized in that: The self-cross attention reinforcement unit comprises: Multimodal joint representation stage construction module: Different multimodal features are normalized through the modality-specific fully connected layer, and then the multimodal joint representation is constructed by the feature-level cascade operation. The multimodal joint representation is split into two parts, which are used for downstream self-attention enhancement and cross-attention enhancement respectively; Self-attention enhancement module: The multimodal joint representation performs self-enhancement of the elements at each timestamp through feature joint projection and self-attention operation to obtain the first enhanced feature; Cross-attention enhancement module: The multimodal joint representation extracts feature prototypes from three heterogeneous interaction spaces through cross-variant feature joint projection, performs cross-enhancement of cross-modal elements and generates new knowledge through mutation, and obtains the second enhanced feature. Feature integration module: concatenates the first enhanced feature and the second enhanced feature to output multimodal enhanced features.

5. According to claim 4, a multimodal human intention understanding system based on brain cognition inspiration is characterized in that: The multimodal joint representation in the cross-attention enhancement module extracts feature prototypes from three heterogeneous interaction spaces through cross-variant feature joint projection, cross-enhancement of cross-modal elements and mutation induction of new knowledge, and obtains a second enhanced feature. Specifically, cross-modal element cross-enhancement and mutation induction of new knowledge are performed through cross-modal matrix multiplication and cross-softmax function between any modality feature and joint feature. The multimodal joint representation performs element adaptation and cross-modal element enhancement from one modality to another at each timestamp through feature joint projection and cross-attention operation to obtain a second enhanced feature.

6. The multimodal human intention understanding system based on brain cognition inspiration according to claim 1 is characterized in that: The source modality updating unit simulates the information updating mechanism of neurons stored in the cerebral cortex to perform knowledge exchange and information filtering, specifically including: Feature update module: uses a feedforward neural network to update multimodal heterogeneous representation features; Information filtering module: Filter redundant information and noise information of multimodal heterogeneous representation features to obtain feature matrix, perform element-level feature filtering multiplication on the filtered feature matrix and the updated features, and output multimodal reinforcement features and multimodal enhanced features respectively, which act on the cross-attention reinforcement unit and the self-cross-attention reinforcement unit.

7. The multimodal human intention understanding system based on brain cognition inspiration according to claim 1 is characterized in that: The adaptive multimodal information fusion unit comprises: Adaptive weight building module: Initialize the shared mapping matrix, and then different reinforcement and enhancement features are combined with the shared mapping matrix to produce feature-specific adaptive weight coefficients; Dynamic weighted fusion module: Based on the adaptive weight coefficient, element-wise multiplication is used to perform weighted summation of different importance coefficients and enhanced features; Multimodal fusion feature output module: outputs the final refined multimodal fusion features.

8. The multimodal human intention understanding system based on brain cognition inspiration according to claim 1 is characterized in that: The emotion intention understanding unit of the continuous dimension is specifically as follows: the multimodal fusion features are transformed into emotion score variables through emotion-specific projection transformation, and then the mean square error loss is used to calculate the sum of the squares of the differences between the emotion score variables and the emotion target values ​​to train the network model and obtain the final emotion prediction value.

9. The multimodal human intention understanding system based on brain cognition inspiration according to claim 1 is characterized in that: The fine-grained behavior intention understanding unit is specifically as follows: the multimodal fusion features are transformed into behavior score variables through behavior-specific projection transformation, and then the difference between the behavior score variable and the behavior target value is calculated using the cross entropy loss. The network model is trained through the constraint of maximizing the category probability to obtain the final behavior prediction value.

10. The multimodal human intention understanding system based on brain cognition inspiration according to claim 1, characterized in that: The system also includes an intention display unit for displaying and presenting the predicted human emotions and behavioral intentions in a visual manner.

Images