A text anchor pre-interaction multi-modal sentiment recognition method and system based on an asymmetric bottleneck architecture

Abstract

A text anchor pre-interaction multi-modal sentiment recognition method based on an asymmetric bottleneck architecture, comprising the following steps: ①. Obtain data containing text, audio and visual three modalities; ②. The data of the three modalities are input into the pre-training model of the corresponding modality of the target multi-modal sentiment recognition network for processing; the text anchor pre-interaction mechanism is adopted for cross-modal interaction, and the asymmetric semantic bottleneck module is used for cross-modal feature interaction; the audio and visual sequence features independently coded by the first k-1 encoder layers without participating in cross-modal interaction are extracted, and the audio and visual modality residual bypasses are respectively constructed; ③. Aggregate the features of each modality, and perform hierarchical dynamic weighted fusion to generate comprehensive multi-modal features, and input into a sentiment classifier to output the final sentiment prediction result. The present application can effectively decouple physiological bias, greatly improve the robustness of sentiment recognition, enhance the fine-grained sentiment cue capturing ability of the model, greatly reduce the computational complexity of cross-modal interaction, improve the system fault tolerance, prevent cascading collapse in an open scene, and ensure the high fault tolerance and running stability of the system in a real open scene.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multimodal emotion computing technology. Specifically, it relates to a text-anchored preemptive interaction (TAPI) multimodal emotion recognition method, system, and model training method based on an asymmetric bottleneck architecture. Background Technology

[0002] With the increasing aging of the world's population, AI-based emotional companionship has become a key research direction in the field of smart elderly care, and accurate multimodal emotion recognition is a prerequisite for achieving high-quality emotional companionship. However, existing multimodal emotion recognition research largely relies on general public datasets, making it difficult to directly apply the models to the elderly population. Multimodal emotion recognition for the elderly population mainly faces the following challenges: (1) Susceptible to interference from individual static physiological bias. The elderly population objectively exhibits visual “pseudo-apathy” such as facial muscle relaxation, as well as audio “pseudo-sadness” caused by vocal cord aging. Existing technologies usually perform indiscriminate mapping of multimodal signals, lacking a suppression mechanism for individual physiological background noise, which can easily lead to emotional misjudgment and affect the accuracy of emotion recognition.

[0003] (2) Subtle emotional features in audio and visual contexts are easily smoothed out in deep transmission. The emotional expressions of the elderly are often subtle and reserved. Existing models often adopt a serial processing paradigm of "independent extraction first, then interactive fusion", which lacks cross-modal semantic guidance and deep interaction in the feature extraction process. As a result, subtle emotional cues in audio and visual contexts are easily smoothed out as local noise in the forward propagation of deep networks, making it difficult to capture fine-grained emotional features.

[0004] (3) Cross-modal interaction has high computational complexity and is difficult to adapt to edge terminal deployment. The existing mainstream cross-modal interaction mechanism based on Transformer has a computational complexity that increases quadratically with the sequence length. When attempting to compensate for the loss of the above-mentioned weak features through deep modal interaction, the huge computational overhead makes it difficult to adapt to the needs of edge terminals for lightweight models and efficient inference.

[0005] With the development of companion robots, smart cockpits, remote care terminals, and human-computer interaction devices, inferring users' emotional states using multi-source signals such as text, voice, and facial video has become an important application direction for multimodal emotion computing. Compared with using only a single modality, multimodal emotion recognition can simultaneously utilize semantic content, prosodic variations, facial movements, and their temporal relationships, thus possessing richer emotional representation and judgment capabilities.

[0006] When targeting elderly users or other groups with stable individual representational differences, the recognition system is also affected by individual static baseline differences. For example, individual characteristics such as facial muscle state, speech loudness, speech rate, and fundamental frequency distribution may remain stable over multiple conversations, while the current conversation's true reflection of emotional changes is often the dynamic shift relative to that individual baseline. If the model directly fuses absolute features without individual calibration, individual differences may be mistakenly identified as emotional differences.

[0007] On the other hand, existing multimodal networks typically perform deep independent encoding of each modality first, and then concatenate, perform attention fusion, or decision fusion at the end of the network. For information such as speech prosody changes and facial micro-movements with small amplitude and short duration, the local changes may be weakened by high-level semantic representations after multiple layers of independent encoding. If full cross-modal attention is repeatedly used in the high-dimensional feature space, it will increase the computational load and memory usage on edge devices.

[0008] In existing technical literature, Chinese patent CN113312530A, entitled "A Text-Based Multimodal Sentiment Classification Method," discloses a technical solution. This method centers on the text modality, first predicting visual and acoustic features from text features, and then distinguishing shared and private information in non-text modalities based on the prediction relationships. Subsequently, a cross-modal attention mechanism is used to fuse text features with shared / private visual and acoustic features, and sentiment classification is performed based on the fusion results. Its key focus is on utilizing the prediction relationships between text and non-text elements and the shared / private semantic division to improve the interpretability of multimodal feature fusion and the performance of sentiment classification.

[0009] In existing technical literature, Chinese patent CN115063709A, entitled "A Multimodal Sentiment Analysis Method and System Based on Cross-modal Attention and Hierarchical Fusion," discloses a technical solution. This solution extracts textual, visual, and acoustic features from the video to be analyzed, performs cross-attention between the text and acoustic / visual features respectively to obtain non-textual modal representations, and then uses a gated recurrent hierarchical fusion network to perform pairwise interactions and redundant information filtering on the textual, acoustic, and visual representations, ultimately obtaining a one-dimensional representation for sentiment prediction. This solution emphasizes cross-modal attention, hierarchical fusion, and gating redundancy removal.

[0010] In summary, researching a multimodal emotion recognition method that can suppress interference from static physiological features, accurately capture fine-grained emotional cues, and significantly reduce cross-modal interaction computational overhead, thereby adapting to the lightweight deployment requirements of edge terminals, is a technical challenge that urgently needs to be solved in the field of smart elderly care. Summary of the Invention

[0011] The purpose of this invention is to overcome the shortcomings of the existing technologies and provide a text-anchored pre-interaction multimodal emotion recognition method and system based on an asymmetric bottleneck architecture. In the process of multimodal emotion recognition for target users, it can reduce the interference of the individual's long-term stable visual / acoustic baseline on the judgment of the current emotional state, and retain and enhance subtle non-textual emotional changes during deep feature extraction. It completes the interaction of text, audio and visual information without completely relying on high-dimensional full cross-modal attention, effectively decouples physiological biases, significantly improves the robustness of emotion recognition, significantly broadens the application boundaries, and enhances the model's ability to capture fine-grained emotional cues. This invention also effectively suppresses error cascading, ensuring the system's high fault tolerance and operational stability in real open scenarios.

[0012] The above-mentioned objective of the present invention is achieved through the following technical solution.

[0013] A text-anchored pre-interaction multimodal emotion recognition method based on an asymmetric bottleneck architecture is disclosed. The method is executed by an electronic device with a processor and memory, connected to a camera and a microphone. After acquiring video data from the camera, audio data from the microphone, and text data from the audio data, digital signal processing is performed, and then the emotion category and corresponding probability are output. The digital signal processing is performed according to the following steps: (1) Acquire data containing three modalities: text, audio, and visual; (2) The data of the three modalities are input into the pre-trained model of the corresponding modality of the target multimodal emotion recognition network for processing; wherein, the pre-trained model of each modality contains an embedding layer and a deep feature extraction network composed of stacked multi-layer encoders; the data of the three modalities are first converted into initial sequences through the embedding layer of the pre-trained model corresponding to each modality, and then input into the multi-layer encoder for layer-by-layer feature extraction; the multi-layer encoder feature extraction includes two stages: single-modal independent encoding and cross-modal deep interaction; the single-modal independent encoding stage includes the first k-1 layers of encoders, and the cross-modal interaction starts from the k-th layer encoder and advances to the next layer; the cross-modal interaction adopts the text-anchored preemptive interaction (TAPI) mechanism and uses the asymmetric semantic bottleneck module (ASB) to perform cross-modal feature interaction; the audio and visual sequence features that have been independently encoded by the first k-1 layers of encoders but have not participated in cross-modal interaction are extracted, and audio and visual modal residual bypasses are constructed respectively; (3) Aggregate the features of each modality and perform hierarchical dynamic weighted fusion to generate comprehensive multimodal features, and input them into the sentiment classifier to output the final sentiment prediction result.

[0014] The aforementioned text-anchored pre-interaction multimodal emotion recognition method based on an asymmetric bottleneck architecture uses the text modality t as the central anchor point, with non-text modalities m aligning and interacting with the text modality t to achieve progressive forward propagation of features extracted during interaction; wherein, , Indicates audio modality, Let represent the visual modality; where the non-textual modality m, in its pre-trained model at layer n, independently performs two cross-modal interactions with the textual modality t based on the encoder's native concatenated topology; where N represents the total number of encoder layers, and the total number of encoder layers is the same for all modes. This layer serves as the initiation layer for cross-modal interaction. After the Multi-Head Self-Attention (MHSA) sublayer and the Feed-Forward Network (FFN) sublayer, interaction is achieved through the intermodal introductory ASB module, acting as a bridge. Following the cross-modal interaction of the nth encoder layer, the text modal sequence features are output. and non-textual modal sequence features enhanced by textual modality Where L represents the length of each modality sequence and d represents the feature dimension.

[0015] The aforementioned text-anchored pre-interaction multimodal emotion recognition method based on an asymmetric bottleneck architecture, wherein the ASB module employs an asymmetric structure consisting of nonlinear feature dimensionality reduction and linear feature dimensionality enhancement, specifically includes the following components during cross-modal feature interaction: (1) Nonlinear feature dimensionality reduction: Using the down projection matrix of the corresponding modality and the nonlinear activation function, the text features and at least one non-text feature input to the current interaction node are mapped to the preset low-dimensional interaction space respectively to obtain low-dimensional text features and corresponding low-dimensional non-text features. (2) Low-dimensional cross-modal interaction: In the low-dimensional interaction space, with any of the low-dimensional non-text features as the query vector and the corresponding low-dimensional text features as the key vector and value vector, cross-attention calculation guided by text as a prior is performed to output non-text interaction features that incorporate text semantics. (3) Linear dimensionality increase and residual fusion: Using the upper projection matrix of the corresponding modality, the non-textual interaction features integrated with text semantics are linearly increased to restore the original feature dimension of the non-textual modality; the increased non-textual interaction features are superimposed with the original non-textual features before the current interaction node through residual connection, and layer normalization is performed to generate the enhanced non-textual features corresponding to the current interaction node. The enhanced non-text features, together with the text features that remain unchanged and are directly transmitted in the interaction node, constitute the output feature combination of the current interaction node, and are transmitted along the coding network to the next network sub-layer or the next adjacent coding layer.

[0016] The aforementioned text-anchored pre-interaction multimodal emotion recognition method based on an asymmetric bottleneck architecture evaluates text confidence before utilizing the ASB module for cross-modal feature interaction. Specifically, this involves: obtaining the global prior confidence score Ct of the current input text; and comparing this prior confidence score Ct with a preset safety threshold. The text modality weight coefficients are calculated through comparison using a preset mapping function. Non-textual modal weighting coefficients.

[0017] The above-mentioned text anchoring pre-interaction multimodal emotion recognition method based on asymmetric bottleneck architecture, in step (3), specifically includes the aggregation of features from various modalities: (1) Obtain the output features of each modality in the last consecutive layers of the multilayer encoder, and perform bit-by-bit addition or weighted summation on each modality to generate multi-scale aggregated sequence features of each modality, including: enhanced audio aggregated features. Enhance visual convergence features Text aggregation features ; (2) The audio and visual single-modal independent encoded sequence features transmitted from the residual bypass are respectively combined with the corresponding enhanced audio aggregation features. and enhanced visual aggregation features By performing residual fusion within each modality, full-scale audio features that combine low-level physical perception with high-order interactive semantics are obtained. With full-scale visual features ; (3) For the full-scale audio features Full-scale visual features With the text aggregation features Global pooling is performed separately to eliminate the differences in the length of each modality sequence, generating a global representation vector for the non-textual modality. and text global representation vector ;

[0018] in, This represents the global pooling function, whose purpose is to eliminate the time series dimension and output a fixed-length one-dimensional vector.

[0019] The above-mentioned text anchoring pre-interaction multimodal emotion recognition method based on asymmetric bottleneck architecture, in step (3), the hierarchical dynamic weighted fusion to generate comprehensive multimodal features specifically includes: using the text modality weight coefficients obtained from text confidence evaluation. Total weight of non-textual modalities For the text global representation vector respectively and non-text global representation vectors Dynamic weighted fusion is performed; a secondary dynamic allocation mechanism for non-text weights is introduced: based on the audio global vector. With visual global vectors Calculate the relative attention scores of the two. and To satisfy them and the total weight coefficient of the non-text modality Further decoupling yields dynamic weighting coefficients for text, audio, and visual modalities. :

[0020] Use separately The global representation vectors of the corresponding modalities are dynamically weighted and concatenated to obtain multimodal features. The calculation formula is: ,in, This indicates a feature concatenation / sponging operation, which involves concatenating multiple one-dimensional feature vectors end to end along the feature dimension to create a higher-dimensional composite feature vector.

[0021] The aforementioned text-anchored pre-interaction multimodal emotion recognition method based on an asymmetric bottleneck architecture, when applied to emotion recognition scenarios for specific target groups, further includes individual baseline calibration: acquiring non-textual modal baseline data of the target object in its calm state, including audio baseline data and visual baseline data; extracting and storing the non-textual baseline feature vectors obtained after the baseline data has undergone independent encoding stages by its respective multi-layer encoders; when extracting features from the test data, before cross-modal interaction begins after the non-textual current features of the test data have completed single-modal independent encoding, calculating the dynamic difference features between the non-textual current features and the non-textual baseline feature vectors respectively, and fusing the non-textual current features with their corresponding dynamic difference features respectively to generate non-textual calibrated features free from individual bias; and inputting the non-textual calibrated features into the ASB module to participate in cross-modal interaction.

[0022] A multi-stage training method for an emotion recognition model is disclosed, comprising the following steps: First stage, training a general multimodal emotion prediction model: using a publicly available multimodal emotion dataset, a base network including the TAPI mechanism and ASB module is trained to obtain a general multimodal emotion prediction model; Second stage, adaptive fine-tuning of the target group based on non-textual modal baseline calibration features: acquiring emotion-annotated multimodal fine-tuning data and corresponding calm-state non-textual modal baseline data for a specific target group; freezing some parameters of the pre-trained model's feature extractor, and using the pre-trained model's forward propagation to extract the non-textual baseline features of the non-textual modal baseline data after the single-modal independent encoding stage and before entering cross-modal interaction. Vector; In the forward propagation of the fine-tuning network, after the current feature of the non-text modal fine-tuning data has completed single-modal independent encoding, before starting cross-modal interaction, the dynamic difference features between the current non-text feature and the non-text baseline feature vector are calculated respectively, and the current non-text feature and its corresponding dynamic difference feature are fused to generate non-text calibrated features without individual bias; the non-text calibrated features are used as initial input and injected into a deep interleaved coding network containing the ASB module for forward propagation; based on the error between the prediction result and the true label, the parameters of the ASB module and the sentiment classifier are updated only through backpropagation, forcing the model to adaptively learn the dynamic sentiment offset features of the specific target group.

[0023] A text-anchored pre-interaction multimodal emotion recognition system based on an asymmetric bottleneck architecture comprises: a memory for storing a computer program; and a processor for executing the computer program to implement the steps in the aforementioned emotion recognition method or the steps in the aforementioned model training method. Beneficial effects

[0024] Compared with the prior art, the present invention has the following beneficial effects: ① Effectively decouples physiological biases, significantly improving the robustness of emotion recognition.

[0025] ② To address the issue of pseudo-physiological interference caused by physiological aging in the elderly population, this invention constructs a joint mechanism of forward individual baseline calibration and a two-stage model training method to dynamically decouple static physiological biases in the latent space. This design not only effectively suppresses emotional misjudgments caused by aging representations but also endows the model with generalization ability to specific groups such as those with facial paralysis, significantly broadening its application boundaries.

[0026] ③ Enhance the model's ability to capture fine-grained emotional cues. Addressing the issue of subtle emotional expression and the annihilation of weak emotional features in deep networks among the elderly, this invention innovatively constructs a "feature extraction on one side, cross-modal interaction on the other" TAPI mechanism. This mechanism uses the high signal-to-noise ratio text modality as a semantic hub to provide early semantic guidance and enhancement to the audio and visual modalities. This effectively alleviates the excessive smoothing and annihilation of weak emotional features in audio and visual media within deep networks, enhancing the model's ability to capture and express fine-grained emotional representations.

[0027] ④ Significantly reduces the computational complexity of cross-modal interaction, adapting to the lightweight requirements of edge terminals.

[0028] ⑤ To address the deployment challenges posed by limited computing power at the edge, this invention designs an ASB module, which confines computationally intensive interactive calculations to a low-dimensional bottleneck space. Compared to high-dimensional global interaction based on Transformer, this design significantly reduces floating-point operations and memory usage while preserving core semantics, achieving a balance between lightweight and efficient inference and high-precision emotion recognition.

[0029] ⑥ Enhance system fault tolerance to prevent cascading crashes in open scenarios.

[0030] ⑦ To address the anomalies caused by Automatic Speech Recognition (ASR) errors leading to the failure of text priors or the loss of modalities, this invention introduces a dynamic text confidence decay and residual bypass mechanism. When core anchor points are damaged, the model can adaptively adjust cross-modal weights and rely on bypasses to smoothly degrade, effectively suppressing error cascading and ensuring the system's high fault tolerance and operational stability in real-world open scenarios. Attached Figure Description

[0031] Figure 1 This is a flowchart of the present invention; Figure 2 This is a diagram of the overall network architecture for multimodal emotion recognition. Detailed Implementation

[0032] The present invention will be further described in detail below with reference to specific embodiments. Example 1

[0033] In this embodiment, it is assumed that the algorithm of the present invention is deployed in an intelligent companion robot serving the elderly. The robot is equipped with a lightweight edge computing chip and a local secure storage module, as well as a high-definition camera and microphone array. The recognition steps are as follows: Step S101: Obtain test data including text, audio and visual modalities, as well as baseline data of the user's calm state.

[0034] In practical applications, when elderly users engage in daily conversations with the companion robot, the robot captures real-time facial video streams via a camera as raw visual modality data (Dv), audio streams via a microphone array as raw audio modality data (Da), and converts the speech into text via a built-in ASR module as raw text modality data (Dt). These three components together constitute the raw data to be tested, Dinput. , where the subscripts t, a, and v represent text, audio, and visual modalities, respectively.

[0035] Considering the common physiological characteristics of the elderly, such as facial muscle relaxation, in order to further suppress these static background noises and achieve higher recognition accuracy, in a preferred embodiment, the system pre-collects a segment of multimodal data when the user registers for the first time or is in a quiet environment and in a neutral, calm state of mind. This data serves as the calm state baseline data Dbase for that specific individual. Specifically, the calm state baseline data can be collected, for example, by guiding the user to look at the camera in a relaxed state and read a standard, emotionally neutral prompt in a calm, everyday tone. The system simultaneously records a facial video stream and an audio stream of a preset duration, such as 3 to 5 seconds, generating corresponding visual baseline data Dbasev and audio baseline data Dbasea, respectively. Dbase is stored as the baseline data of the calm state of that specific individual.

[0036] Step S102: Input the data of each modality into the corresponding pre-trained model for processing, and perform individual baseline calibration before cross-modal interaction. The acquired trimodal data Dt, Da, and Dv are input into the pre-trained models of the corresponding modalities in the target multimodal emotion recognition network for processing. Each modality's pre-trained model includes an embedding layer and a deep feature extraction network composed of stacked multi-layer encoders. For example, the text modality uses the chinese-roberta-wwm-ext model, the audio modality uses the chinese-hubert-base model, and the visual modality uses the clip-vit-base-patch32 model.

[0037] Each modal data is first processed through its corresponding embedding layer to obtain the initial text sequence. Initial audio sequence and visual initial sequence , where L represents the length of each modality sequence and d represents the feature dimension.

[0038] Subsequently, the initial sequences are fed into the multi-layer encoders of their respective pre-trained models for forward propagation, yielding sequence features for each modality at each encoder layer. The text, audio, and visual sequence features output from the l-th layer of the encoder are represented as follows: Where N is the total number of encoder layers, .

[0039] The deep feature extraction network, composed of stacked multi-layer encoders, is generally divided into two stages: single-modal independent encoding and cross-modal deep interaction. Specifically, the single-modal independent encoding stage includes the first k-1 encoder layers, and the cross-modal interaction begins from the k-th encoder layer. Preferably, k accounts for approximately 40% to 60% of the total number of encoding layers. In the multi-layer encoder structure of the pre-trained model, lower-layer encoders are responsible for capturing shallow, fine-grained features, while higher-layer encoders focus on abstracting high-level semantics, representing global, coarse-grained features. By setting k in the middle to later stages of the encoding chain, each unimodal mode completes sufficient unimodal contextual representation before entering cross-modal interaction, maximizing the preservation and locking of shallow, fine-grained emotional cues from audio and vision.

[0040] If the system has already acquired the baseline data, then in this step, an individual baseline calibration operation is further performed to suppress physiological noise: the individual baseline calibration operation for non-text modalities occurs after the single-modal independent encoding stage, that is, after the output of the (k-1)th layer encoder and before the kth layer of the cross-modal interaction initiation layer. To avoid computational redundancy in the inference stage and improve the real-time response speed of the system, the entire calibration process is divided into two stages: offline baseline solidification and online real-time calibration. The extraction and solidification of the baseline feature vector is an offline pre-computation (or periodic update) step, which is only performed when the system is first registered and configured or when the preset update cycle is reached, and does not need to be repeatedly calculated in each daily dialogue interaction; while during real-time forward propagation, the system only needs to directly call the solidified baseline feature vector to participate in the subsequent difference and fusion operations.

[0041] During the offline benchmark solidification stage The system first extracts features from the stored calm-state baseline data Dbasea and Dbasev independently through the embedding layer of the corresponding modality and the first k-1 encoder layers. Then, by performing global pooling operations on their respective temporal dimensions, temporal length fluctuations are eliminated, transforming them into a single audio baseline feature vector of the corresponding modality feature dimension. and single visual reference feature vector And it is solidified as a priori parameter for a specific individual.

[0042] Online real-time calibration phase First, the system acquires the audio temporal feature sequence transmitted from the k-1 layer encoder. and visual temporal feature sequences And call the aforementioned cached audio and visual benchmark feature vectors. and Then, through tensor broadcasting, the features of each time step in the sequence are respectively compared with... Element-wise subtraction is performed to generate the audio difference feature sequence. Visual difference feature sequence :

[0043] in , This represents an element-wise subtraction operation based on the tensor broadcast mechanism.

[0044] Subsequently, the original audio and visual feature sequences were processed. , respectively with the corresponding dynamic difference feature sequences The fusion process yields calibrated feature sequences from both audio and visual non-textual modalities, after individual bias removal. This serves as the pure input for entering the k-th layer to perform subsequent cross-modal interactions.

[0045]

[0046] in , This indicates a feature fusion operation, such as element-wise addition.

[0047] Step S103: Evaluate the prior confidence of the text and generate text and non-text modal weights. Although text-based modalities often have a high signal-to-noise ratio, the confidence level of ASR output text is prone to degradation due to the pronunciation decline characteristics of elderly users and the interference of ambient noise at home. To prevent text errors from causing cascading error amplification in subsequent networks, the system evaluates the text confidence level before performing cross-modal interactions, thereby improving the system's robustness.

[0048] Specifically, the system first obtains the global prior confidence score Ct of the current ASR text, where The "text prior confidence" mentioned in this invention specifically refers to the initial recognition probability directly assigned by the underlying speech engine before the text modality participates in cross-modal interaction. It can be represented by sentence-level scores or generated based on word-level confidence weighting, and is used as prior knowledge to guide downstream network routing.

[0049] Subsequently, the system compares the prior confidence score Ct of the text with a preset safety threshold. The comparison is performed, and the text modality weight coefficients are calculated using a preset mapping function. In one embodiment, its mathematical expression is:

[0050] in, The preset attenuation control adjustment factor, Based on the above calculations The total weight of the non-text modality is obtained. .

[0051] when When the "strong text anchoring" strategy is triggered, the formula calculates the following: The system assigns a high interaction weight coefficient to the text modality, while the corresponding total weight for the non-text modality is... This ensures that downstream cross-modal interactions are text-centric, leveraging its high-order semantics to precisely guide and refine audio and visual features.

[0052] when When this happens, the "non-text adaptive takeover" strategy is triggered: Based on the above formula, a proportional decay occurs. The system dynamically reduces the weight coefficient of the text modality. Simultaneously increase the total weight of non-textual modalities. When facing extremely low confidence scenarios, i.e. When, by formula Natural deduction ,at this time The text modality performs complete backoff, causing subsequent networks to rely entirely on pure non-text residual bypass for sentiment prediction.

[0053] It should be noted that the preset safety threshold mentioned in this invention... The determination mechanism includes, but is not limited to, two modes: static prior setting and dynamic adaptive adjustment. The specific configuration can be flexibly adjusted according to the computing power and needs of the actual deployment environment.

[0054] Static prior setting of safety threshold As in one embodiment, It can be used as a hyperparameter of the network model. Its value can be based on grid search on an offline validation set, or on the statistical distribution of ASR word error rates in historical real-world application scenarios such as specific home-based elderly care environments. These values ​​can be pre-defined empirical reference values ​​by domain experts, for example, setting... .

[0055] Dynamically adaptive adjustment of safety threshold In another, more preferred embodiment, to adapt to the complex and varied background noise in the environment, A floating threshold is used, dynamically generated by the system based on real-time acoustic environment characteristics. Specifically, the system calculates the signal-to-noise ratio (SNR) of the audio capture frame corresponding to the current text in real time, or retrieves the elderly user's historical average pronunciation clarity profile; when the real-time SNR is lower than the preset environmental standard, or when the user is in a physiological state of unclear pronunciation, the system automatically increases it. The numerical value makes the network's acceptance criteria for text modalities more stringent, thus triggering the "non-text adaptive takeover" strategy earlier and more sensitively. It should be noted that the preset environmental standard refers to the signal-to-noise ratio warning line set based on engineering experience, such as 5dB or 10dB.

[0056] This step introduces an adaptive soft routing mechanism into the multimodal emotion recognition network, decoupling the system from its absolute dependence on a single text modality and greatly improving the robustness of emotion recognition.

[0057] Step S104: Perform pre-modal interaction based on text anchors, and simultaneously construct audio and visual residual bypasses. (1) Performing pre-modal interaction based on text anchors In this embodiment, as described in step S102, cross-modal interaction begins with the first... Layer encoder. For any encoder layer participating in cross-modal interaction, denoted as layer n, Where N is the total number of encoder layers, and the total number of encoder layers is the same for all modalities. Define a non-text modal indicator variable. .

[0058] With text modality t as the central anchor point, each non-text modality m, at its nth layer, independently performs two cross-modal interactions with text modality t based on the encoder's native concatenated topology. Specifically, after the Multi-Head Self-Attention (MHSA) sublayer and the Feed-Forward Network (FFN) sublayer, the interaction is achieved by introducing an ASB module as a bridge between the modalities. At the current nth layer, the complete multimodal feature transfer process is as follows: First, the system will analyze the non-textual features from level n-1. Text features Intermediate features are output through their respective modal MHSA sub-layers and via the encoder's native residual connections and layer normalization. and Using this as input, the first ASB module is invoked to perform the initial interaction at this level, resulting in non-text modal features enhanced with text modality. The input is fed into the FFN sublayer of modality m. (Text modality) After passing through ASB, it remains unchanged and directly enters the FFN sub-layer of the text modality. It should be noted that when... That is, during the first cross-modal interaction, the input is an independent single-modal feature after individual baseline calibration. .

[0059] Subsequently, the system will use the aforementioned initial alignment features Corresponding text features Continuing with nonlinear mapping through their respective FFN sublayers, and similarly after native residual connections and normalization, intermediate features are obtained. Using this as input, the second ASB module is invoked to perform a secondary interaction, resulting in enhanced non-textual modal features. The final output of this layer is non-text features. and text features This is passed down to level n+1. .

[0060] In the two interactions described above, the internal computational logic executed by calling the ASB module is completely consistent, including three steps: "non-linear dimensionality reduction, low-dimensional interaction, and linear dimensionality increase." The unified mathematical computation process is as follows: Suppose the non-text feature sequence input to this ASB module is The text feature sequence is The output is the non-textual features after text enhancement. Where L represents the sequence length and d represents the feature dimension. To reduce the computational overhead during interaction, a low-dimensional interaction dimension is preset for the non-textual modality m. ,in, , It means much smaller than.

[0061] It should be noted that the local variables here... Corresponding to the interactive features passed in from the main process, such as those during the initial interaction. and during secondary interaction Furthermore, after the module completes its calculations, its final output local features will be available. Assign the corresponding values ​​to the update features in the main process, for example... .

[0062] First, nonlinear dimensionality reduction. This is applied to non-textual feature sequences. and text feature sequences Perform a downprojection operation to map it to dimension 1. Low-dimensional space:

[0063] in, The down projection matrix representing the non-textual modality; This represents the downprojection matrix used to adapt modality m when mapping text modality to a lower-dimensional space. These represent the corresponding bias terms; both the projection matrix and the bias terms mentioned above are learnable parameters that can be jointly trained and optimized end-to-end with the network.

[0064] This represents a nonlinear activation function, such as GELU, ReLU, or Swish, used to inject nonlinear expressive power into a low-dimensional space.

[0065] These are low-dimensional non-text features and low-dimensional text features, respectively, used for subsequent cross-modal attention calculations.

[0066] Subsequently, cross-modal interaction occurs in a low-dimensional space, utilizing low-dimensional non-textual features. As the query vector, using low-dimensional text features As a key-value vector, perform cross-attention computation guided by text priors, and output non-textual interaction features incorporating text semantics. .

[0067]

[0068] in, Representing the low-dimensional text feature matrix transpose, This represents the normalized exponential function, which is used to transform the scaled dot product result into attention probability distribution weights that sum to 1 in the sequence dimension. This enables dynamic weighting and adaptive focusing of text modality features that match query conditions based on non-text modalities.

[0069] Finally, lightweight dimensionality upscaling and residual fusion are performed. This utilizes the upprojection matrix of the corresponding mode. Perform linear dimensionality upscaling to include cross-modal interaction features. Restore the original feature dimensions to the corresponding modality; combine with the text gating weights obtained in step S103. The upgraded interaction features are combined with the input features. Perform residual fusion and layer normalization processing to output the final non-textual modal features processed by the ASB module. .

[0070]

[0071] in, The layer normalization function is used to stabilize the distribution scale of the feature sequence after residual fusion, prevent gradient vanishing or gradient explosion during deep network training, and thus ensure the stability of multimodal feature transfer and model convergence.

[0072] It should be noted that when the text is unreliable, that is... When the above formula degenerates into At this point, the ASB module automatically shields the interference of text features, allowing non-text modal features to retain their original semantics to the greatest extent, thereby ensuring the noise resistance and security of deep network flow.

[0073] (2) Constructing audio and visual residual bypasses While performing cross-modal interaction in the main branch, the system directly extracts the corresponding single-modal sequence features from the audio and visual feature nodes after individual calibration in step S102. The audio residual bypass and the visual residual bypass are constructed. The bypass bypasses the subsequent cross-modal interaction link and directly feeds forward the single-modal independent representation after baseline calibration to the end feature integration stage for subsequent integration with the deep coding features of the corresponding modality.

[0074] Step S105: Aggregate the features of each modality and perform hierarchical dynamic weighted fusion to generate comprehensive multimodal features.

[0075] First, the output features of each modality in the last consecutive layers of the stacked encoder network are obtained, and bit-by-bit addition or weighted summation is performed on each modality to generate multi-scale aggregated sequence features for each modality, including: enhanced audio aggregated features. Enhance visual convergence features Text aggregation features ; Subsequently, the audio and visual calibrated single-modal independent encoded sequence features from the residual bypass in step S104 are combined. , respectively with the corresponding enhanced audio aggregation features and enhanced visual aggregation features By performing residual fusion within each modality, full-scale audio features that combine low-level physical perception with high-order interactive semantics are obtained. With full-scale visual features For example, its full-scale features The calculation process can be expressed as: ,in .

[0076] Next, the full-scale audio features Full-scale visual features With the text aggregation features Global pooling is performed separately to eliminate the differences in the length of each modality sequence, generating a global representation vector for the non-textual modality. and text global representation vector ;

[0077] in, This represents the global pooling function, whose purpose is to eliminate the time series dimension and output a fixed-length one-dimensional vector.

[0078] Finally, the text gating coefficients obtained from the text confidence assessment in S103 are used. Non-text combined weighting coefficient The text global representation vector and the non-text global representation vector are dynamically weighted and fused respectively. To further address the dynamic noise differences between audio and visual modalities, this invention introduces a secondary dynamic allocation mechanism for non-text weights: based on the audio global vector... With visual global vectors Calculate the relative attention scores of the two. To satisfy them and the non-text comprehensive weight coefficient Further decoupling yields dynamic weighting coefficients for text, audio, and visual modalities. :

[0079] Use separately The global representation vectors of the corresponding modalities are dynamically weighted and concatenated to obtain multimodal features. The calculation formula is: ,in, This indicates a feature concatenation / sponging operation, which involves concatenating multiple one-dimensional feature vectors end to end along the feature dimension to create a higher-dimensional composite feature vector.

[0080] It should be noted that this step innovatively employs a modality-specific feature construction and hierarchical dynamic fusion mechanism for feature fusion. Its technical advantages are as follows: At the modality-specific feature construction level, based on modal heterogeneity, residual bypasses are specially designed for low-semantic-density continuous audio and visual signals to compensate for physical perception details, fusing full-scale features; for high-semantic-density discrete text signals, only multi-scale features are aggregated to avoid shallow morpheme interference and ensure high-order semantic purity. At the hierarchical dynamic weighted fusion level, a two-level adaptive modality collaboration and noise reduction mechanism is constructed: the first-level weights adjust the fusion ratio of text and non-text based on text confidence to prevent single-modality dominance; the second-level weights are based on the relative attention scores of audio and video to achieve feature competition and weight decoupling between modalities. In short, this mechanism takes into account both optimal information complementarity under normal conditions and smooth degradation and anti-interference capabilities in extreme scenarios, such as modality impairment or failure, which can greatly improve the accuracy and robustness of multimodal emotion recognition.

[0081] Step S106: Perform sentiment classification prediction based on comprehensive multimodal features and generate corresponding sentiment interaction instructions.

[0082] The integrated multimodal feature vector obtained in step S105 is input into the fully connected classifier at the end. The Softmax function is used for feature dimensionality reduction and probability mapping. The probability distribution of the user in each preset emotion category is calculated and output, and the category with the highest probability value is selected as the final emotion prediction result. The preset emotion categories include, for example, happiness, neutrality, sadness, anger, and surprise. Furthermore, based on the final emotion prediction result, the companion robot generates corresponding anthropomorphic voice or action control commands, thereby completing a high-precision, intelligent emotional interaction response. Example 2

[0083] In this embodiment, it is assumed that the present invention is used in a psychiatric outpatient clinic. The workstation is equipped with a camera, microphone, local computing unit, and encrypted storage module, and is operated by a psychiatrist or a qualified clinical psychologist. Exemplarily, the workstation uses Python to establish a unified program for data acquisition, preprocessing, scale scoring, model training, and inference, scheduling each functional module according to steps S201 to S206, thereby providing a reproducible software interface and operating environment. The system is used to simultaneously analyze the patient's verbal content, vocal behavior, and facial behavior during the interview process, outputting an auxiliary classification of the severity of depressive symptoms, an evidence summary, and safety risk warnings; the final conclusion is made by the clinician based on medical history, mental examination, functional impairment, physical and drug factors, and a history of mania or hypomania.

[0084] This embodiment uses the same framework as Embodiment 1, including trimodal acquisition, individual baseline calibration, text confidence assessment, TAPI pre-modal interaction, ASB asymmetric semantic bottleneck, audio-video residual bypass, hierarchical dynamic weighted fusion, and classifier output. The difference lies in that the interview questions, response scores, clinical labels, risk gating, and output content are all configured for auxiliary assessment of the severity of depression.

[0085] Step S201: Obtain interview trimodal data, response codes, and individual baselines. Before the interview begins, after obtaining the patient's informed consent and completing permission verification, the camera and microphone are activated. For example, the camera captures frontal or slightly side-view facial video at no less than 25 frames per second, and the microphone captures audio at 16 kHz, 16-bit mono. The system timestamps audio frames, video frames, and doctor's questions. The acquisition program calls FFmpeg to synchronously read the camera and microphone streams, unifying the video to a fixed frame rate, converting the audio to 16 kHz mono, and slicing it according to the start and end times of the questions and answers. This is to ensure strict alignment of text, audio, and visual segments and reduce sampling format differences. The patient maintains a distance of approximately 0.8 to 1.2 meters from the camera. If severe facial obstruction, multiple people appearing in the frame, audio clipping, or persistent loud noise is detected, the equipment needs to be readjusted.

[0086] To obtain an individual patient baseline, the doctor first asks the patient to give a 20-30 second factual description without emotional evaluation, such as describing the route to the hospital that day, the room's furnishings, or recent daily routine. The system simultaneously generates audio baseline data (Dbasea) and visual baseline data (Dbasev). If a valid record of the patient's previous stable or remission period exists, the most recent valid record is used as the historical baseline, and the current factual description is used for correction; if no historical record exists, the current segment is used as a temporary baseline. This process reduces the influence of the patient's inherent facial features, usual speaking speed, regional accent, habitually low facial expressions, or eye contact habits on the assessment results. The individual baseline, original audio / video, and doctor's revisions are stored and encrypted using SQLite and SQLCipher with de-identified numbers, and combined with role-based authorization and audit logs, longitudinal comparisons during follow-up visits are achieved.

[0087] Subsequently, the doctor used a customized DepressionScorer interview scoring module to ask nine main questions sequentially over the past two weeks. This module displays the questions, records frequency codes from 0 to 3, functional impairment codes, and doctor's notes on the interface, and automatically calculates the total score and reference level for the nine items, creating a traceable, structured text tagging system for the questions and answer criteria. For each question, the patient is first allowed to describe their symptoms, and then the doctor confirms the frequency of symptom occurrence. The standardized frequency codes are: 0 for none; 1 for several days; 2 for more than half the days; and 3 for almost every day. The system records the patient's original answers, the codes confirmed by the doctor, and the start and end times of the question-and-answer session.

[0088] After completing the nine main questions, the doctor further inquired about the degree of difficulty the above symptoms caused to work, study, housework, social interactions, and intimate relationships, recording it as: 0 no difficulty, 1 some difficulty, 2 very difficult, 3 extremely difficult. This functional impairment code is not included in the total score of the nine items, but serves as an important basis for handling classification conflicts and doctor review. The nine codes and functional impairment codes confirmed by the doctor can be appended to the corresponding interview text using structured tags [Q1=…][Q9=…][FUNC=…], so that it remains part of the text modality input model without adding a fourth modality. The test text data Dt consists of doctor's questions, patient verbatim transcription, question numbers, and structured codes; the test audio data Da consists of speech segments from each question-and-answer interval; the test visual data Dv consists of facial video frames time-aligned with each speech segment. Together, these three constitute Dinput={Dt, Da,Dv}.

[0089] Step S202: Perform preprocessing, pre-trained encoding, and individual baseline calibration on the trimodal data. The system first divides the continuous interview into nine segments based on a unified timestamp: main question segments, functional impairment segments, and safety follow-up segments. On the audio side, FFmpeg performs format conversion and time slicing, and torchaudio or librosa is called to perform resampling, amplitude normalization, silence and pause statistics, and extraction of auxiliary acoustic indicators such as pitch and energy, thereby improving audio input consistency and providing a basis for audio quality assessment. On the visual side, OpenCV is used to perform camera reading, single-face detection, facial region cropping, key point alignment, and sharpness, occlusion, brightness, and multi-person on-camera quality detection to form a stable visual input and visual quality mask; further, OpenFace's FeatureExtraction function is called to export time-series features such as facial motion units, head posture, gaze direction, and detection confidence scores frame by frame. On the text side, FunASR's Chinese speech recognition, speech activity detection, and punctuation recovery modules are used to transcribe the patient's speech into text and output word timestamps and recognition confidence scores; doctors can manually correct these in the interface. Both automatically recognized text and doctor-revised text retain version numbers to trace the source of text confidence scores. The visual or acoustic features extracted by the aforementioned software are used only as behavioral descriptions and are not directly interpreted as depression levels based on a single action unit, gaze direction, tone of voice, or pause.

[0090] After preprocessing, the text modality `chinese-roberta-wwm-ext`, the audio modality `chinese-hubert-base`, and the visual modality `clip-vit-base-patch32` are loaded into the PyTorch and Hugging Face Transformers framework to generate initial text, audio, and visual sequences, respectively. PyTorch is then used to implement TAPI, ASB, residual bypass, dynamic fusion, and a classifier, thus completing end-to-end training, fine-tuning, and inference of the core multimodal network. Similar to Example 1, all three pre-trained models include embedding layers and a deep feature extraction network consisting of N layers of encoders. The first k-1 layers perform single-modal independent encoding, and layers k through N perform TAPI cross-modal deep interaction, where 2 ≤ k ≤ N-1.

[0091] Individual baseline calibration occurs after the (k-1)th layer output and before the kth layer cross-modal interaction. The system calculates the audio baseline vector Ba and the visual baseline vector Bv offline. During online inference, the current audio and visual sequences are subtracted element-wise from Ba and Bv respectively using tensor broadcasting. The subtraction results are then fused with the current sequence to form calibrated audio and visual features. For this application scenario, this processing focuses on suppressing static biases inherent to the patient, such as low volume, dialectal acoustic differences, facial structure, habitually expressionless features, and eye contact habits, allowing the model to primarily focus on changes relative to the patient's own baseline.

[0092] Step S203: Evaluate the prior confidence of the text and generate text and non-text modal weights.

[0093] The system integrates the ASR confidence score, recognition loss ratio, doctor correction status, and audio quality score of each response segment output by FunASR to form a segment-level text prior confidence score Ct,i, which is further aggregated into the global text prior confidence score Ct for this interview. If the doctor has confirmed the transcript verbatim, the system increases the confidence score of the confirmed interval; if the patient's volume is too low, dialect or articulation affects the error rate, or multiple people are speaking in the segment, the system decreases the confidence score of the corresponding interval.

[0094] The system compares Ct with a preset safety threshold τ and generates a text modality weight α and a total non-text modality weight 1-α according to the mapping function described in Example 1. For example, τ can be set to 0.75 to 0.85 on the validation set and dynamically adjusted based on the real-time signal-to-noise ratio, whether the doctor has completed text confirmation, and the patient's historical pronunciation clarity. When Ct is high, text content and structured code serve as strong anchors; when Ct is low, the system reduces the intensity of text guidance and increases the contribution of audio and visual residual bypasses, but must not override doctor-confirmed answers related to death or self-injury simply because the patient's voice or expression appears normal.

[0095] Step S204: Execute TAPI pre-interaction with the question-level text as the anchor point, and construct audio and visual residual bypasses.

[0096] The system uses the text sequence corresponding to each main question as a semantic anchor and selects audio and visual sequences within the same interval using question-and-answer timestamps. In the deep interleaved coding network implemented in PyTorch, the ASB module is called after the MHSA and FFN sub-layers in each encoder layer from layer k to layer N, respectively, so that the audio and visual features receive semantic guidance from the corresponding question text in a low-dimensional interaction space. The ASB module still sequentially performs nonlinear dimensionality reduction, low-dimensional cross-attention with non-text features as queries and text features as keys and values, linear dimensionality increase, and residual normalization, and its mathematical calculation process is the same as in Example 1; thereby restricting the core interaction to a low-dimensional bottleneck space, reducing the amount of computation and supporting real-time inference on local workstations.

[0097] In this scenario, text anchors include the patient's descriptions of changes in interest, depression, hopelessness, self-blame, sleep, energy, attention, psychomotor activity, and self-harming thoughts; audio sequences represent behavioral cues such as speech rate, response latency, pause ratio, and changes in pitch and loudness; visual sequences represent behavioral cues such as facial motor units, head posture, gaze direction, blinking, and facial movement amplitude. These audio and visual cues are only used for joint modeling with the patient's baseline and language content, and a single facial expression, head tilt, reduced eye contact, crying, or a low voice is not directly equated to a specific level of depression.

[0098] Simultaneously, the system extracts residual bypasses from the calibrated audio and visual features at layer k-1, allowing them to bypass subsequent cross-modal interactions and be directly sent to the final integration stage. This setup can preserve original behavioral evidence and achieve smooth degradation in cases of ASR errors, short-term facial occlusion, prolonged patient silence, or inconsistencies between text and non-textual representations.

[0099] Step S205: Aggregate problem-level and session-level features and perform hierarchical dynamic weighted fusion. The system acquires the outputs of text, audio, and visual modalities in the last consecutive multi-layer encoder. First, it performs bit-by-bit addition or learnable weighted summation within each modality to form a multi-scale aggregation sequence. Then, it performs residual fusion of audio and visual residual bypasses with the corresponding enhanced aggregation features to form full-scale audio features and full-scale visual features, and performs global pooling on each.

[0100] For the nine main questions, the system first forms a question-level comprehensive representation zi, which is then aggregated into a conversation-level representation by a hierarchical dynamic weighting module built with PyTorch based on the quality of the fragment data, the confidence of the question text, and the learnable question attention coefficients. The first-level weights are still controlled by α to determine the overall ratio of text to non-text; the second-level weights are assigned 1-α based on the relative attention scores of audio and visual data. For blurry, occluded, or non-effective speech fragments, the system lowers the corresponding modality quality mask and records the reason for the absence, rather than using zero values ​​to impersonate normal behavior. Functional impairment codes and nine frequency codes participate in the fusion along with the text representation.

[0101] Step S206: Output the auxiliary grading of the severity of depressive symptoms, confidence level, and independent safety risk warning.

[0102] The conversation-level integrated multimodal features are input into a fully connected classifier, which is then processed by Softmax to output a probability distribution of five preset assistance levels. The sum of the nine frequency codes confirmed by the doctor is denoted as Sq, with a value ranging from 0 to 27. The DepressionScorer interview scoring module automatically calculates Sq and forms a reference level.

[0103] The system simultaneously compares the scale reference level Gq with the multimodal classifier level Gm. If the classifier's maximum probability is not lower than the preset confidence threshold η, and Gq and Gm are the same or only one level apart, the system outputs Gm, Sq, functional impairment code, primary problem-level evidence, and data quality description. If the two differ by more than two levels, the classification confidence is lower than η, there are insufficient effective modalities, or the doctor's observation is significantly inconsistent with the system's results, the system outputs "Requires manual review," without forcibly providing a definitive level. The level confirmed or corrected by the doctor is used as the medical record result and can be used as a subsequent fine-tuning label under anonymization and authorization conditions. Training data can be imported into LabelStudio, where question-and-answer segments, transcripts, audio, video, and timestamps are used as annotation items. Two qualified clinicians annotate the problem code, severity, risk markers, and data availability, respectively. Disagreements are resolved by a third doctor, thus forming a high-quality clinical multimodal dataset that can be used for supervised training and validation.

[0104] Independent security gating The custom SafetyGate module continuously monitors item 9 and death, self-harm, or suicide-related trigger information in the free text. Whenever item 9's code is greater than 0, or any free narrative or physician record contains information related to death wishes, self-harm, suicide methods, intentions, plans, preparatory behaviors, or past attempts, the module immediately displays follow-up questions and locks the status to "Pending Physician Confirmation," without waiting for Sq or Gm. Its purpose is to treat self-harm or suicide risk independently of the total depression score, preventing a low total score from masking immediate danger. The interface prompts the physician to directly ask: Do you wish to die or never wake up again? Have you had thoughts of ending your life? Have you considered specific methods? Do you have an intention to carry it out? Have you formed a plan or made preparations? Have you recently or in the past attempted, interrupted, or terminated such an attempt? The system records yes / no, most recent occurrence time, frequency, controllability, plan, intention, available means, and past behavior. Example 3

[0105] This implementation method follows the three-modal acquisition, individual baseline calibration, text confidence assessment, TAPI pre-modal interaction, ASB asymmetric semantic bottleneck, audio and video residual bypass, hierarchical dynamic weighted fusion, and classifier output framework of Examples 1 and 2. The difference lies in that the question design, response scoring, supervision labeling, review gating, and output content are all configured for the process of criminal police interrogating criminal suspects in accordance with the law, and are used to make auxiliary judgments on specific answer fragments as "the answer is true", "partial concealment", or "lying".

[0106] In this embodiment, it is assumed that the system of the present invention is deployed in the interrogation room of a criminal police team, and is equipped with synchronous audio and video recording equipment, a local computing unit, a case evidence indexing interface, and an encrypted storage module in accordance with the law. The system uses Python to program steps S301 to S306, uniformly calling the acquisition, transcription, quality control, structured scoring, multimodal model, and auditing modules. The system only performs consistency assessment on clearly marked factual propositions or answer fragments, and does not directly determine whether the criminal suspect is guilty, nor does it regard silence, tension, crying, avoiding eye contact, dialect, disability, disease, trauma reaction, or a certain facial movement as evidence of lying; the final judgment is made by the investigators based on objective evidence.

[0107] Step S301: Obtain interrogation trimodal data, question units, response codes, evidence reliability, and individual baselines.

[0108] During interrogation, the camera records the upper body and face of the suspect from the front or slightly side at a rate of no less than 25 frames per second, while the microphone records audio at 16 kHz, 16-bit mono, or higher. The system simultaneously records a unified timestamp for the interrogator, question number, audio / video frames, and evidence citation number. The acquisition program uses FFmpeg to unify the frame rate and audio format and slices the audio by question-and-answer intervals to create time-aligned text, audio, and visual data. If multiple people speak simultaneously, there is severe obstruction, recording clipping, strong noise, or dropped frames, the system records the quality defect and prompts for adjustment, without interpreting the quality defect itself as abnormal behavior.

[0109] Before delving into the core issues of the case, the interrogator first poses several verifiable, neutral factual questions unrelated to the case's substance, such as basic name information, how the suspect arrived on the day of the incident, room furnishings, or recent daily activities, asking the suspect to describe these for 30 to 60 seconds. Based on this, the system generates audio baseline data Dbasea and visual baseline data Dbasev. If a previously valid neutral question-and-answer record of the same person is legally stored, the most recent record can be used as the historical baseline and updated with the current neutral statement. This baseline is only used to suppress static biases such as an individual's inherent speech rate, accent, pitch, facial structure, gaze habits, and gesture amplitude; it is not used to set so-called "guilty responses" or leading control questions.

[0110] Subsequently, the interrogation system established question units through a customized InterrogationManager interrogation question management module. This module displays the question topic, records the original answer, follow-up questions, time range, whether the answer is complete, evidence citation number, and manual notes, and associates each question with its audio and video clips to ensure the traceability of questions, answers, evidence, and model outputs. The wording of questions can be adjusted according to the facts of the case, but it is advisable to first allow free narration, then verify the facts, and finally clarify any contradictions; it is forbidden to repeatedly pressure, hint at answers, or force the interrogated person to maintain a specific expression or gaze in order to obtain a model score.

[0111] The reliability of evidence, ri, can range from 0 to 1: lower values ​​are used for materials that are unverified or of unknown origin; higher values ​​are used for materials whose origin, acquisition process, timestamp, and content have all been verified and are independently corroborated. The importance of fact, wi, indicates the degree of importance of the issue to the factual proposition being evaluated. The system can calculate a structured inconsistency reference score, Sc = Σ(wi·ri·ci) / Σ(wi·ri). Units that are unanswered, legally refused to answer, lack a valid record, or have insufficient evidence coverage are assigned a separate status marker and are not directly scored as ci=3. For example, provided the effective evidence coverage reaches a preset threshold, a "true answer" rule reference is formed when Sc is less than 0.75 and there are no major issues with ci ≥ 2; a "partial concealment" rule reference is formed when Sc is between 0.75 and less than 1.75, or when there is a verified ci=2 issue; and a "lying" rule reference is formed when Sc is not less than 1.75, or when any core issue has ci=3 and ri is not less than 0.8 and is independently corroborated. These thresholds can be adjusted using the training and validation sets.

[0112] The text data Dt to be tested consists of interrogation questions, verbatim transcription of the suspect, speaker identifiers, question numbers, answer versions, ci, ri, wi, and evidence citation numbers; the audio data Da to be tested consists of speech segments corresponding to the question-and-answer intervals; the visual data Dv to be tested consists of time-aligned video frames of the face and upper body. Together, these three constitute Dinput={Dt, Da,Dv}. Structured code participates in the model as text markers, without adding a fourth modality. Original audio and video, individual baselines, evidence citations, manual revisions, and model versions are encrypted and stored using SQLite and SQLCipher by case number, with role permissions, access logs, and integrity checks set to ensure the confidentiality and traceability of case data.

[0113] Step S302: Perform preprocessing, pre-trained encoding, and individual baseline calibration on the trimodal data.

[0114] The system segments continuous interrogation records into neutral baseline segments, nine types of question segments, and contradiction clarification segments based on a unified timestamp. FFmpeg is used to perform audio and video demultiplexing, format unification, and time slicing; on the audio side, torchaudio or librosa are called for resampling, amplitude normalization, speech activity detection, clipping, and noise detection, and auxiliary acoustic sequences such as speech rate, response latency, pause ratio, pitch, energy, and their relative baseline changes are extracted, with the aim of forming a stable input and quantifying audio quality.

[0115] The visual side utilizes OpenCV to perform single-face detection, facial and upper-body region cropping, keypoint alignment, blurring, occlusion adjustment, brightness adjustment, out-of-frame detection, and multi-person presence detection, generating a visual quality mask. It also uses OpenFace's FeatureExtraction function to output facial motion units, head pose, gaze direction, and detection confidence frame by frame, forming a behavioral time series that can be synchronized with the question text and audio. These tools only generate describable visual features and do not directly determine the truthfulness of the answer based on a single motion unit or gaze direction.

[0116] On the text side, FunASR is invoked to perform Chinese speech recognition, speech activity detection, punctuation recovery, speaker segmentation, and word timestamp output. Interrogators correct proper nouns, dialect words, homophones, and speaker attribution within the interface. The system simultaneously retains both automatically transcribed and manually revised versions. After preprocessing, chinese-roberta-wwm-ext, chinese-hubert-base, and clip-vit-base-patch32 are loaded into the PyTorch and Hugging Face Transformers frameworks as pre-trained models for text, audio, and vision, respectively, generating a trimodal initial sequence that enters an N-layer encoder. The first k-1 layers perform single-modal independent encoding, while layers k through N perform TAPI cross-modal deep interaction, where 2 ≤ k ≤ N-1.

[0117] Individual baseline calibration occurs after the (k-1)th layer output and before the kth layer cross-modal interaction. The system calculates the audio baseline vector Ba and the visual baseline vector Bv offline or at the beginning of the interrogation. During online inference, the current audio and visual sequences are element-wise differencing with Ba and Bv respectively and fused with the current sequence to form calibrated features free from individual bias. This processing aims to reduce individual differences such as accent, articulation disorders, inherent low volume, habitual pauses, facial structure, blink frequency, gaze habits, and body movement amplitude. It does not eliminate the influence of fatigue, medication, disease, trauma, or environment; therefore, these factors still need to be individually recorded by the interrogator and trigger manual review.

[0118] Step S303: Evaluate the prior confidence of the text and generate text and non-text modal weights.

[0119] The system integrates the ASR confidence level, word / word missing ratio, multi-person overlapping speech ratio, speaker separation reliability, interrogator correction status, and audio quality score output by FunASR to form the prior confidence level Ct,i for the question segment, and aggregates it to obtain the prior confidence level Ct for the conversation-level text. The confidence level is increased for intervals that have been verified sentence-by-sentence by the interrogator and are consistent with the original recording; the confidence level is decreased for intervals with noise, dialect, articulation disorders, homonyms, or question-answer misalignment, while retaining the cause code. The reliability of the evidence consistency code ci is determined by the evidence verification status and does not automatically increase with the increase in ASR confidence.

[0120] The system compares Ct with a preset safety threshold τ and generates a text modality weight α and a total non-text modality weight 1-α according to the mapping function in Example 1. When Ct is high, the question text, the corrected answer, and the structured evidence code serve as strong anchors; when Ct is low, the text guidance strength is reduced and the relative contribution of audio and visual residual bypasses is increased. If the core answer is not transcribed completely, the question and answer are not aligned, or there are insufficient effective modalities, the system must enter the "data quality insufficient / manual review required" state and must not output "lying" based solely on audio or visual behavior.

[0121] Step S304: Execute TAPI pre-interaction with the question-level text as the anchor point, and construct audio and visual residual bypasses.

[0122] The system uses the text sequence of each problem unit as a semantic anchor and selects audio and visual sequences from the same interval according to a unified timestamp. In the PyTorch implementation of the encoders at layers k to N, the ASB module is called after the MHSA and FFN sub-layers, respectively. The ASB module performs nonlinear dimensionality reduction on the text and corresponding non-text features, performs cross-attention in the low-dimensional space using audio or visual features as queries and text features as keys and values, and then performs linear dimensionality increase, residual stacking, and layer normalization; its mathematical calculation process is the same as in Example 1. In this way, the temporal variations in audio and visual data are modeled under the semantic constraints of a specific problem, and cross-modal computation is restricted to a low-dimensional bottleneck space.

[0123] In this scenario, text features represent factual propositions such as people, time, location, behavior, objects, information sources, denial, correction, and explanation; audio features represent speech rate, response latency, pauses, self-correction, pitch, and energy changes relative to the baseline; visual features represent facial action units, head posture, gaze direction, blinking, and movement amplitude changes relative to the baseline. These non-text features serve only as question-related behavioral context and are not directly interpreted as deception by the rule layer. The system simultaneously extracts residual bypasses from the (k-1)th layer calibrated audio and visual features, allowing them to bypass subsequent interactions and be sent to the final integration stage. This preserves underlying information and achieves smooth degradation in the event of ASR errors, occlusion, silence, or modal conflicts.

[0124] Step S305: Aggregate problem-level and session-level features and perform hierarchical dynamic weighted fusion.

[0125] The system acquires the outputs of text, audio, and visual modalities in the last consecutive multi-layer encoder, performs bit-by-bit addition or learnable weighted summation to form multi-scale aggregated sequences for each modality; then, it performs intra-modal residual fusion of audio and visual residual bypasses with corresponding enhanced aggregated features to form full-scale audio features and full-scale visual features, and generates fixed-length vectors through global pooling.

[0126] For each question unit i, the system forms a question-level comprehensive representation zi. A hierarchical dynamic weighting module implemented in PyTorch controls the first-level weights of text and non-text using α, and then further distributes 1-α based on the relative attention scores of audio and visual input. The question-level representation is aggregated into a conversation-level comprehensive representation zsession based on text confidence, audio / video quality, evidence reliability ri, factual importance wi, and learnable question attention coefficients. For segments with no valid speech, severe facial occlusion, no response, or unverified evidence, the system reduces the corresponding quality or masks the evidence and records the reason, avoiding the use of zero vectors to impersonate normal behavior or automatically treating no response as false. ci, ri, wi, and evidence numbers participate in the fusion as structured text markers, enabling the model output to be traceable back to specific questions and evidence.

[0127] Step S306: Output the auxiliary judgment of whether the answer is true, partially concealed, or a lie, the confidence level, and the result of manual review.

[0128] Multimodal features, either question-level or conversation-level, are input into a fully connected classifier. After passing through Softmax, the classifier outputs probability distributions for three target categories: PT represents "true answer," PH represents "partial concealment," and PF represents "lying." "True answer" indicates that the core fact to be evaluated is substantially consistent with the currently verified materials; "partial concealment" indicates that the answer contains truthful parts, but there are significant omissions, limited admissions, or selective disclosures verified to have occurred; and "lying" indicates that a clear and important factual statement is substantially refuted by highly reliable evidence. Each of the three categories addresses a specific question or factual proposition and is not extended to judgments about the suspect's character, overall credibility, or guilt or innocence.

[0129] The system compares the structured reference category Gs, formed based on Sc and major issue triggering rules, with the multimodal classifier category Gm. If the classifier's maximum probability is not lower than the preset threshold η, the effective evidence coverage reaches the threshold, the key modality quality is qualified, and Gs and Gm are consistent, the system outputs the corresponding auxiliary category and probability. If Gs and Gm are inconsistent, the maximum probability is lower than η, the core evidence has not been verified, there are dialect or communication barriers, disease or drug effects, prolonged fatigue, strong trauma response, insufficient data quality, or a significant conflict between human observation and model results, the custom ReviewGate manual review module outputs "Requires manual review" or "Insufficient evidence," without forcibly selecting one of the three categories. Even if "lying" is output, the specific proposition being evaluated, the corresponding ci, the evidence number, the evidence reliability, and the model uncertainty must be listed; audio and video behavior cannot be used as the sole basis.

[0130] Data used for training or fine-tuning can be imported into Label Studio to create a multimodal annotation project. This project links questions, answers, timestamps, audio, video, evidence verification status, and final fact-checking results. At least two authorized personnel, who are not directly involved in the same round of annotation, independently label the data as "true," "partially concealed," or "lying," and also indicate data availability. Disagreements are resolved by a third person based on objective evidence. Training labels must not be based solely on the interrogator's subjective impressions, facial expressions, tone of voice, silence, or unverified statements to reduce labeling bias.

[0131] The final report includes at least: case and session authorization numbers, interrogation time, model and rule versions, question and answer versions, ci, ri, wi, sc, PT / PH / PF probabilities, question-level and session-level auxiliary categories, evidence citation numbers, text / audio / visual quality, summary of relative baseline changes, review gating results, triggering reasons, and operator confirmation and revision records. The report is stored encrypted using SQLite and SQLCipher, and access, export, and modification logs are recorded. During deployment, the trained PyTorch model is exported to ONNX format, and ONNX Runtime is used to perform offline CPU or GPU inference on a dedicated workstation in the criminal investigation team. Simultaneously, model file verification values ​​and versions are recorded to reduce inference latency, avoid unnecessary external transmission of case audio and video, and support result reproduction. This completes a consistency auxiliary analysis for the authenticity of interrogation answers.

Claims

1. A text-anchored pre-interaction multimodal emotion recognition method based on an asymmetric bottleneck architecture, characterized in that, The method is executed by an electronic device with a processor and memory, which is connected to a camera and a microphone. After acquiring video data from the camera, acquiring audio data from the microphone, and obtaining text data from the audio data, digital signal processing is performed, and then the emotion category and corresponding probability are output. The digital signal processing is performed according to the following steps: (1) Acquire data containing three modalities: text, audio, and visual; (2) The data of the three modalities are input into the pre-trained model of the target multimodal emotion recognition network corresponding to the modality for processing; wherein, the pre-trained model of each modality includes an embedding layer and a deep feature extraction network composed of stacked multi-layer encoders; the data of the three modalities are first converted into initial sequences through the embedding layer of the pre-trained model corresponding to each modality, and then input into the multi-layer encoder for layer-by-layer feature extraction; the feature extraction of the multi-layer encoder includes two stages: single-modal independent encoding and cross-modal deep interaction; the single-modal independent encoding stage includes the first k-1 Layer encoder layer, cross-modal interaction from the first layer k Starting with the layer encoder, cross-modal interaction is advanced layer by layer. Cross-modal interaction employs a text-anchored preemptive interaction (TAPI) mechanism, utilizing an asymmetric semantic bottleneck (ASB) module for cross-modal feature interaction; pre-processing is then extracted. k-1 The layer encoder independently encodes audio and visual sequence features that do not participate in cross-modal interaction, and constructs audio and visual modal residual bypasses respectively; (3) Aggregate the features of each modality and perform hierarchical dynamic weighted fusion to generate comprehensive multimodal features, and input them into the sentiment classifier to output the final sentiment prediction result.

2. The method according to claim 1, characterized in that, The TAPI mechanism uses text modality t as the central anchor point, and non-text modality m aligns and interacts with text modality t to achieve progressive forward propagation of features extracted while interacting; wherein, , Indicates audio modality, Let represent the visual modality; where the non-textual modality m, in its pre-trained model at layer n, independently performs two cross-modal interactions with the textual modality t based on the encoder's native concatenated topology; where N represents the total number of encoder layers, and the total number of encoder layers is the same for all modes. This layer serves as the initiation layer for cross-modal interaction. After the Multi-Head Self-Attention (MHSA) sublayer and the Feed-Forward Network (FFN) sublayer, interaction is achieved through the intermodal introductory ASB module, acting as a bridge. Following the cross-modal interaction of the nth encoder layer, the text modal sequence features are output. and non-textual modal sequence features enhanced by textual modality Where L represents the length of each modality sequence and d represents the feature dimension.

3. The method according to claim 1, characterized in that, The ASB module employs an asymmetric structure consisting of nonlinear feature dimensionality reduction and linear feature dimensionality enhancement. Specifically, when performing cross-modal feature interaction, it includes: (1) Nonlinear feature dimensionality reduction: Using the down projection matrix of the corresponding modality and the nonlinear activation function, the text features and at least one non-text feature input to the current interaction node are mapped to the preset low-dimensional interaction space respectively to obtain low-dimensional text features and corresponding low-dimensional non-text features. (2) Low-dimensional cross-modal interaction: In the low-dimensional interaction space, with any of the low-dimensional non-text features as the query vector and the corresponding low-dimensional text features as the key vector and value vector, cross-attention calculation guided by text as a prior is performed to output non-text interaction features that incorporate text semantics. (3) Linear dimensionality increase and residual fusion: Using the upper projection matrix of the corresponding modality, the non-textual interaction features integrated with text semantics are linearly increased to restore the original feature dimension of the non-textual modality; the increased non-textual interaction features are superimposed with the original non-textual features before the current interaction node through residual connection, and layer normalization is performed to generate the enhanced non-textual features corresponding to the current interaction node. The enhanced non-text features, together with the text features that remain unchanged and are directly transmitted in the interaction node, constitute the output feature combination of the current interaction node, and are transmitted along the coding network to the next network sub-layer or the next adjacent coding layer.

4. The method according to claim 1, characterized in that, Before using the ASB module for cross-modal feature interaction, the text confidence is evaluated. Specifically, this involves: obtaining the global prior confidence score Ct of the current input text; and comparing the prior confidence score Ct with a preset safety threshold. The text modality weight coefficients are calculated through comparison using a preset mapping function. Non-textual modal weighting coefficients.

5. The method according to claim 1, characterized in that, The aggregation of modal features in step (3) specifically includes: (1) Obtain the output features of each modality in the last consecutive layers of the multilayer encoder, and perform bit-by-bit addition or weighted summation on each modality to generate multi-scale aggregated sequence features of each modality, including: enhanced audio aggregated features. Enhance visual convergence features Text aggregation features ; (2) The audio and visual single-modal independent encoded sequence features transmitted from the residual bypass are respectively combined with the corresponding enhanced audio aggregation features. and enhanced visual aggregation features By performing residual fusion within each modality, full-scale audio features that combine low-level physical perception with high-order interactive semantics are obtained. With full-scale visual features ; (3) For the full-scale audio features Full-scale visual features With the text aggregation features Global pooling is performed separately to eliminate the differences in the length of each modality sequence, generating a global representation vector for the non-textual modality. and text global representation vector ; in, This represents the global pooling function, whose purpose is to eliminate the time series dimension and output a fixed-length one-dimensional vector.

6. The method according to claim 1, characterized in that, Step (3) involves performing hierarchical dynamic weighted fusion to generate comprehensive multimodal features, specifically including: using text modality weight coefficients obtained from text confidence evaluation. Total weight of non-textual modalities For the text global representation vector respectively and non-text global representation vectors Perform dynamic weighted fusion; Among them, a secondary dynamic allocation mechanism for non-text weights is introduced: based on the audio global vector. With visual global vectors Calculate the relative attention scores of the two. and To satisfy them and the total weight coefficient of the non-text modality Further decoupling yields dynamic weighting coefficients for text, audio, and visual modalities. : Use separately The global representation vectors of the corresponding modalities are dynamically weighted and concatenated to obtain multimodal features. The calculation formula is: ,in, This indicates a feature concatenation / sponging operation, which involves concatenating multiple one-dimensional feature vectors end to end along the feature dimension to create a higher-dimensional composite feature vector.

7. The method according to claim 1, characterized in that, When applied to emotion recognition scenarios targeting specific groups, it also includes individual baseline calibration: acquiring non-textual modal baseline data of the target object in a calm state, including audio baseline data and visual baseline data; Extract and store the non-text baseline feature vectors obtained from the baseline data after the independent encoding stages of their respective multi-layer encoders; When extracting features from the test data, after the non-text current features of the test data have completed single-modal independent encoding, before starting cross-modal interaction, the dynamic difference features between the non-text current features and the non-text baseline feature vector are calculated respectively, and the non-text current features and their corresponding dynamic difference features are fused to generate non-text calibrated features after removing individual bias. The non-text calibrated features are input into the ASB module to participate in cross-modal interaction.

8. A multi-stage training method for an emotion recognition model, used to implement multi-stage training as described in any one of claims 1 to 7, characterized in that, The process is as follows: First, training a general multimodal sentiment prediction model: Using a publicly available multimodal sentiment dataset, train a basic network containing the TAPI mechanism and ASB module to obtain a general multimodal sentiment prediction model. Second, adaptive fine-tuning of the target group based on non-textual modal baseline calibration features: Obtain sentiment-annotated multimodal fine-tuning data and corresponding calm-state non-textual modal baseline data for a specific target group; freeze some parameters of the pre-trained model's feature extractor; use the pre-trained model's forward propagation to extract the non-textual baseline feature vector of the non-textual modal baseline data after the single-modal independent encoding stage and before cross-modal interaction; during the forward propagation of the fine-tuning network, after the current feature of the non-textual modal fine-tuning data completes single-modal independent encoding and before cross-modal interaction begins, calculate the dynamic difference features between the current non-textual feature and the non-textual baseline feature vector, and fuse the current non-textual feature with its corresponding dynamic difference feature to generate non-textual calibration features free from individual bias. The non-text-calibrated features are used as initial inputs and injected into a deep interleaved coding network containing the ASB module for forward propagation. Based on the error between the prediction results and the true labels, the parameters of the ASB module and the sentiment classifier are updated only through backpropagation, forcing the model to adaptively learn the dynamic sentiment shift features of the specific target group.

9. A text-anchored pre-interaction multimodal emotion recognition system based on an asymmetric bottleneck architecture, characterized in that, include: Memory, used to store computer programs; A processor, when executing the computer program, implements the steps of the emotion recognition method as described in any one of claims 1 to 7, or implements the steps of the model training method as described in claim 8.

Images