A multi-modal deep fusion human-computer interaction method and system of an intelligent cockpit

Abstract

The application relates to the technical field of intelligent cockpits, and discloses a multi-modal deep fusion man-machine interaction method and system for an intelligent cockpit, which comprises the following steps: unifying time and confidence, supporting modal loss and noise robustness; learning time distortion and window attention, soft alignment to construct a fusion segment; extracting modal representation, fusing and optimizing a self-supervised target; estimating domain context and performing conditional normalization and enhancement; decomposing domain invariance / domain specificity and performing distribution and adversarial alignment; domain perception sampling and weighted comparison optimization. The application will realize the core transition of environment non-sensing self-adaptation, scene seamless generalization and man-machine unhindered cooperation, can accurately understand the complex intention of a driver or passenger in an extreme environment such as strong light, heavy rain and wind noise, can automatically maintain interaction consistency when switching between day and night urban and rural scenes, can predict potential risks through low-latency fusion of multi-modal signals, and finally constructs an active safety cockpit ecology.

Description

Technical Field

[0001] This invention relates to the field of intelligent cockpits, and more specifically, to a multimodal deep fusion human-computer interaction method and system for intelligent cockpits. Background Technology

[0002] The intelligent cockpit is a next-generation driving and riding space centered on the vehicle domain controller, integrating AI perception, multimodal interaction, and cloud services. It monitors the status of drivers and passengers in real time through in-cabin devices and combines vehicle environmental data to provide proactive services such as voice assistants, driver monitoring, AR-HUD navigation, and intelligent seat adjustment.

[0003] In the use of intelligent cockpits, there are challenges in robust intent understanding and low-latency response under noise, modal loss and asynchronous data flow. Moreover, changes in scenarios such as day and night, urban and rural highways, and sunny, rainy and foggy conditions can cause degradation of the shared feature space, leading to feature space distortion and performance degradation of cross-domain scenario models. Summary of the Invention

[0004] This invention provides a multimodal deep fusion human-computer interaction method and system for intelligent cockpits, which solves the technical problems in related technologies, such as noise interference, modality loss and asynchronous data flow causing delays in intent recognition and increased error rates, and feature space distortion caused by day / night / urban / rural / weather changes, resulting in performance degradation of cross-domain scene models.

[0005] This invention provides a multimodal deep fusion human-computer interaction method for intelligent cockpits, comprising the following steps: S100, Acquisition and Event Encoding: Unified Time and Confidence, Supports Modal Missing and Noise Robustness; S200, Asynchronous Spatiotemporal Alignment: Learning Temporal Warp and Window Attention, Soft Alignment Constructs Fusion Fragments; S300, Characterization and Initial Fusion: Extract modal representations, fuse and optimize self-supervised objectives; S400, Domain Context Estimation and Conditional Normalization: Estimate the domain context and perform conditional normalization and enhancement; S500, characterization of decomposition and domain alignment: decomposition domain invariant / domain specific and performing distribution and adversarial alignment; S600, biased contrastive learning: domain-aware sampling and weighted contrastive optimization; S700, Semantic Anchoring and Cyclic Consistency: Constructing cross-modal anchors and optimizing consistency and mutual information proxy; S800, Reliability-Weighted Fusion and Security Decision-Making: Domain Adaptive Belief Fusion and Imposition of Security Constraints; S900, online personalized updates and cross-domain distillation: federated aggregation and cloud distillation, delivering lightweight, adaptable incremental updates.

[0006] Furthermore, the steps for data acquisition and event coding are as follows: S110, Confidence normalization and masking: Read the original confidence vector modally, perform Softmax normalization according to the time dimension to obtain the normalized confidence; then compare it with the threshold to generate a binary mask; S120, Timestamp Unification and Lightweight Encoding: First, call the cross-modal time synchronization function to complete time alignment, then concatenate the original features, original confidence and context or device metadata and send them to the lightweight encoder to output lightweight features and cache them; S130, Robust Encoding of Missing Modal: If the valid event mask is valid, the encoded feature vector is directly retained; otherwise, the interpolation function is triggered, and the interpolation value is calculated using the same modal neighborhood or cross-modal aligned features. The robust feature representation is then output and sent downstream.

[0007] Furthermore, the steps for asynchronous spatiotemporal alignment are as follows: S210, Time Warp Function Learning: Discretize a unified clock with 10ms intervals, optimize the piecewise linear warp function using dynamic programming or gradient descent, initialize with hardware timestamp linear mapping, and cache the optimal time warp function after 50 iterations to converge. S220, Window soft alignment: A sliding window is established for each uniform time step, and the numerator and denominator are calculated only within the window to avoid global traversal; the result is cached as a sparse weight matrix; S230, Fragment Construction: Sparse matrix multiplication is used on the GPU to achieve weighted summation, the output frame rate is uniform, forward padding is used for missing weighted frames to ensure temporal continuity, and the result is cached as an aligned fragment representation.

[0008] Furthermore, the steps for characterization and initial fusion are as follows: S310, Representation Extraction: The 128-dimensional aligned segment is fed into the corresponding modal encoder, and a 256-dimensional representation is output and cached. S320, Initial Fusion: First, the representation vector is weighted and summed according to the fusion weight vector to obtain a temporary vector, and then the final shared latent space representation is obtained by shared 1×1 convolution. S330, self-supervised objective: where the comparison term is: the final shared latent space representation of 256 dimensions for each modality pair is used as a positive sample, and the same mode / different mode at other times is used as a negative sample, with a temperature coefficient of 0.1, and the InfoNCE loss is calculated; The reconstruction term is as follows: the encoded features are randomly occluded by the mask, the occluded features are concatenated with the modal representation and sent to the decoder to predict the masked part, and the L1 error is used as the measure. The weighted sum of the two results in the monitoring loss, which is used for end-to-end backpropagation.

[0009] Furthermore, the steps for domain context estimation and conditional normalization are as follows: S410, Domain Context Estimation: The pre-trained domain discriminator is used to classify the fused shared latent space representation into domains, and the posterior probability of belonging to each domain at the current time is output, providing domain context for subsequent conditional normalization and enhancement. S420, Conditional Normalization: Dynamically generate scaling and offset factors based on the domain context, and perform channel-by-channel conditional normalization of the input features to alleviate distribution offset between different domains; S430, Lightweight Style Enhancement: Apply random style perturbations to visual or sensor features during the training phase to improve the model's robustness to domain style changes such as brightness, contrast, noise, and fog. The perturbation amplitude is sampled from a uniform distribution, ensuring that the enhancement is lightweight and can be completed online in real time.

[0010] Furthermore, the steps for characterization decomposition and domain alignment are as follows: S510, Representation Decomposition: The fused shared latent space representation is split into domain-invariant components and domain-specific components for subsequent alignment and dedomaining. S520, Distribution Alignment: Within each training step, batch samples are divided according to domain labels, the mean and covariance are calculated online, and the distribution alignment loss is obtained by substituting them into the formula. After multiplying by the weights, the loss is added to the total loss. S530, Adversarial Dedomaining: The domain-invariant components are fed into the domain discriminator through a gradient inversion layer to obtain the domain prediction. The cross-entropy is then calculated with the true label to obtain the adversarial loss. This loss is multiplied by the weights and added to the total loss to achieve adversarial learning of the domain-invariant representation.

[0011] Furthermore, the steps for bias-free contrastive learning are as follows: S610, de-biased sampling: Positive samples are semantically consistent but have different domains; Negative samples are semantically different and are excluded from the easily confused set; S620, Domain Weighted Comparison: Increases the weight of low-frequency domain samples to reduce bias.

[0012] Furthermore, the steps for semantic anchoring and circular consistency are as follows: S710, Anchor Construction: ASR is called on the audio stream to obtain text embeddings; Scene Graph is called on the video stream to obtain scene graph embeddings; text embeddings, scene graph embeddings, and external common sense are concatenated into a 384-dimensional vector; and fed into the semantic fusion network to obtain 256-dimensional semantic anchors and cache them. S720, Semantic Consistency Loss: Calculate the cosine similarity between the domain-invariant representation and the semantic anchor point of the positive sample, randomly select semantically mismatched negative sample anchor points within the same batch, calculate the hinge loss, and then incorporate the hinge loss into the total loss. S730, Cyclic Consistency Loss: Audio features are fed into an audio modal encoder to obtain a 256-dimensional audio representation, which is then processed by a text-image generator to generate 128-dimensional pseudo-visual features. The pseudo-visual features are fed into a visual modal encoder to obtain a 256-dimensional cyclic representation. The original visual features are directly fed into a visual modal encoder to obtain a 256-dimensional true representation. The squared L2 distance between the two is calculated to obtain the cyclic consistency loss, which is then added to the total loss. S740, Mutual Information Proxy: The domain-invariant representation is fed into the domain discriminator via GRL and cross-entropy is calculated with the domain label to obtain the domain loss term; the same domain-invariant representation is directly fed into the semantic classifier and cross-entropy is calculated with the semantic label to obtain the semantic loss term; the mutual information loss is calculated and added to the total loss.

[0013] Furthermore, the steps for reliability-weighted fusion and security decision-making are as follows: S810, Domain Reliability Estimation: Read the pre-stored domain risk table to obtain the risk score, look up the table by modality to obtain the bias term and slope coefficient, substitute them into the Sigmoid function to calculate the reliability confidence, and cache the results; S820, Reliability-Weighted Fusion: The initial fusion weights are multiplied by the reliability confidence for each modality to obtain new reliability-weighted weights, and then L1 normalization is performed to ensure that the sum is 1; the modal representations are weighted and summed according to the new weights, and then a 256-dimensional fusion result is output through a shared 1×1 convolution. S830, Safety Constraint Decision: The policy network first outputs the probability of all actions, calculates the action-state risk estimation function output one by one, removes actions that exceed the risk limit, and selects the action with the highest probability from the remaining actions as the final action to be executed; if there is no action that meets the requirements, the safety barrier is triggered and an alarm flag is recorded. S840, Evidence Recording: The cosine similarity of domain-invariant representations and semantic anchors, modal reliability, action and risk values ​​are written to a local circular log by timestamp. Each record is appended with a Unix timestamp and frame ID. The log is periodically compressed and uploaded to the cloud for incident backtracking, model auditing and visualization interpretation.

[0014] This invention also proposes a multimodal deep fusion human-computer interaction system for an intelligent cockpit, comprising: Acquisition and event coding module: Synchronizes timestamps of multimodal data and labels confidence levels, realizes automatic filling of missing modalities and noise filtering, and builds a robust input pipeline; Asynchronous spatiotemporal alignment module: learns dynamic time warp functions to align cross-modal sequences and generates soft-aligned fused fragments through a window attention mechanism; The representation and initial fusion module extracts deep features from each modality and constructs a joint embedding space, optimizing the self-supervised reconstruction loss to achieve initial feature fusion; Domain context estimation and conditional normalization module: Estimates domain statistical properties, performs domain conditional normalization, and generates enhanced samples; Representation decomposition and domain alignment module: Decouple features as domain-invariant / specific components, use adversarial learning to align cross-domain distributions, and minimize domain differences; The bias-reduction contrastive learning module dynamically constructs positive and negative sample pairs based on domain attributes and optimizes the weighted contrastive loss function to eliminate domain bias. Semantic Anchoring and Cyclic Consistency Module: Constructs cross-modal semantic anchors to establish alignment benchmarks and maximizes modal mutual information through cycle consistency loss; Reliability-weighted fusion and safety decision module: Calculates modal-level confidence weights for adaptive fusion and applies safety constraint boundaries to prevent high-risk predictions; Online individualized update and cross-domain distillation module: Federated aggregation client model updates global knowledge base, cloud distillation of lightweight incremental models and distribution to local devices.

[0015] The beneficial effects of this invention are as follows: The intelligent cockpit multimodal system of this invention will achieve a core leap in environmental self-adaptation, seamless scene generalization, and unimpeded human-machine collaboration. Even in extreme environments such as strong light, heavy rain, and wind noise, it can still accurately understand the complex intentions of drivers and passengers, automatically maintain interaction consistency when switching between day and night and urban and rural scenes, and predict potential risks through low-latency fusion of multimodal signals, ultimately building an active safety cockpit ecosystem. Attached Figure Description

[0016] Figure 1 This is a flowchart of a multimodal deep fusion human-computer interaction method for an intelligent cockpit according to the present invention; Figure 2 This is a structural block diagram of a multimodal deep fusion human-computer interaction system for an intelligent cockpit according to the present invention.

[0017] In the diagram: 101. Acquisition and Event Encoding Module; 102. Asynchronous Spatiotemporal Alignment Module; 103. Representation and Initial Fusion Module; 104. Domain Context Estimation and Conditional Normalization Module; 105. Representation Decomposition and Domain Alignment Module; 106. Bias-Free Contrastive Learning Module; 107. Semantic Anchoring and Cyclic Consistency Module; 108. Reliability Weighted Fusion and Security Decision Module; 109. Online Individualized Update and Cross-Domain Distillation Module. Detailed Implementation

[0018] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0019] like Figure 1 As shown, a multimodal deep fusion human-computer interaction method for an intelligent cockpit includes the following steps: S100, Acquisition and Event Encoding: Unified Time and Confidence, Supports Modal Missing and Noise Robustness; In one embodiment of the present invention, the following steps are specifically included: S110, Confidence normalization and masking: Read the original confidence vector modally, perform Softmax normalization according to the time dimension to obtain the normalized confidence; then compare it with the threshold to generate a binary mask; ; in, To normalize the confidence level, For modality At any moment The original confidence level, This is the valid event mask (1 indicates valid, 0 indicates invalid). The preset threshold (value range 0–1, typical 0.05) is used. For event time indexing, Modal identifiers (e.g., visual v, audio a, radar r). This refers to all time indices traversed during summation; S120, Timestamp Unification and Lightweight Encoding: First, call the cross-modal time synchronization function to complete time alignment, then concatenate the original features, original confidence and context or device metadata and send them to the lightweight encoder to output lightweight features and cache them; ; in, For cross-modal time synchronization functions (calibrated by hardware timestamps or NTP / PTP), the local timestamps of each mode are... Mapping to a unified clock , For modality At any moment The original features (such as image CNN mesh, audio MFCC, radar point cloud). , For modality Lightweight encoders (1×1 convolution or 2-layer MLP, parameter count <50k). For context or device metadata (such as sensor ID, sampling rate, environment label). For the encoded feature vector ( (Unified to 128 dimensions) ; S130, Robust Encoding of Missing Modal: If the valid event mask is valid, the encoded feature vector is directly retained; otherwise, the interpolation function is triggered, and the interpolation value is calculated using the same modal neighborhood or cross-modal aligned features. The robust feature representation is then output and sent downstream. ; in, Interpolation functions designed based on neighboring events or cross-modal information (such as third-order temporal neighborhood averaging or cross-modal attention). Representing modes exist Neighborhood ( , The set of available coded features within a frame. This is the final robust feature representation; S200, Asynchronous Spatiotemporal Alignment: Learning Temporal Warp and Window Attention, Soft Alignment Constructs Fusion Fragments; In one embodiment of the present invention, the following steps are specifically included: S210, Time Warp Function Learning: Discretize a unified clock with 10ms intervals, optimize the piecewise linear warp function using dynamic programming or gradient descent, initialize with hardware timestamp linear mapping, and cache the optimal time warp function after 50 iterations to converge. ; in, For modality The optimal time warp function, To make the mode Local timestamp Mapping to a unified clock The twist function, Alignment weights (combined from confidence score and neighborhood density). To unify clock indexes, For modality Local event time index, The specific calculation is as follows: , The normalized confidence level obtained for S110, for The ratio of the number of valid events in the neighborhood to the maximum allowed density (values ​​range from 0 to 1). S220, Window Soft Alignment: Create a sliding window at each uniform time point, and only align the window with the desired alignment. Calculate the numerator and denominator to avoid global traversal; cache the result as a sparse weight matrix. ; in, For modality At the time of unity For local events Soft alignment weights, For modality The time bandwidth parameters (typical values: visual 40ms, audio 30ms, radar 50ms). To sum all local time indices, The output is the optimized twist function; S230, Fragment Construction: Sparse matrix multiplication is used on the GPU to achieve weighted summation, the output frame rate is uniform, forward padding is used for missing weighted frames to ensure temporal continuity, and the result is cached as an aligned fragment representation; ; in, For modality At the same time The aligned fragment representation (128-dimensional). For modality Local time Robust feature representation (128-dimensional). The soft alignment weights output by S220; S300, Characterization and Initial Fusion: Extract modal representations, fuse and optimize self-supervised objectives; In one embodiment of the present invention, the following steps are specifically included: S310, Representation Extraction: The 128-dimensional aligned segment is fed into the corresponding modal encoder, and a 256-dimensional representation is output and cached. ; in, For modality The encoder (composed of two layers of 512-dimensional MLP, ReLU, and Layer Norm, with a parameter count of <0.5M). For modality At the time of unity The aligned fragment representation, , For modality At any moment The representation vector, ; S320, Initial Fusion: First, the representation vector is weighted and summed according to the fusion weight vector to obtain a temporary vector, and then the final shared latent space representation is obtained by a shared 1×1 convolution (outputting 256 dimensions); ; in, The fusion operator (using weighted summation + 1×1 convolution for dimensionality reduction) is used. Indicates all modes at time... The set of representations ( ), For a moment The fusion weight vector (dimensions consistent with the number of modalities, elements) and ), , This represents the shared potential space after integration. ; S330, Self-Supervised Objective: Comparison items: for each pair 256-dimensional sampling and As positive samples, other time-series same-mode / different-mode samples are negative samples, temperature coefficient Calculate the InfoNCE loss; Reconstruction item: by mask Random occlusion , after occlusion features and splicing and feeding Predict the masked portion, and measure it using L1 error; The weighted sum of the two items yields , used for end-to-end backpropagation; ; in, A binary mask of the same dimension as the encoded features (mask ratio 15%). , For modality At any moment The encoded features, , For modality The decoder (2-layer 512-dimensional MLP, ReLU, output 128-dimensional). Modal characterization output by S310 , It is an L1 norm. This represents a pairing of any two different modalities. To monitor losses; S400, Domain Context Estimation and Conditional Normalization: Estimate the domain context and perform conditional normalization and enhancement; In one embodiment of the present invention, the following steps are specifically included: S410, Domain Context Estimation: A pre-trained domain discriminator is used to classify the fused shared latent space representation into domains, outputting the posterior probability of belonging to each domain at the current time step, providing domain context for subsequent conditional normalization and enhancement; Formula: ; in, For domain tags (total) Categories, such as city / highway / sunny / rainy / foggy). , To share the feature extractor, Mapped to the discriminator input dimension, For the pre-trained domain discriminant network, output Unnormalized logits; obtained after Softmax A probability vector of dimension , whose ? Vij Finally, the domain ID or complete probability vector corresponding to the highest probability is taken as the domain context. .

[0020] S420, Conditional Normalization: Dynamically generates scaling and offset factors based on the domain context, performing channel-by-channel conditional normalization on the input features to mitigate distribution shifts between different domains; Formula: ; in, Features to be normalized ( aisle), , , The mean and standard deviation of the current mini-batch or instances. , ; Lightweight context networks (such as 2-layer MLPs) based on domain context generate, ; Represents channel-by-channel product; Output Dimensions and If the features are the same, they can be directly replaced and fed into the downstream network; S430, Lightweight Style Enhancement: Applyes random style perturbations to visual or sensor features during training to improve the model's robustness to style variations in domains such as brightness, contrast, noise, and fog; perturbation amplitude... Sampling from a uniform distribution ensures lightweight enhancements that can be performed online in real time. Formula: ; ; in, This is a four-dimensional enhancement parameter vector, corresponding to brightness respectively. Contrast Noise intensity atomization level ; For the range of values ​​for each parameter (e.g.) , , , ); in For differentiable images / feature-level enhancement operators, the following steps are performed in sequence: 1. Brightness adjustment: 2. Contrast Adjustment: , 3. Additive Gaussian noise: (This refers to the channel mean.) 4. Atomization simulation: , Global atmospheric light estimation; output enhanced Used for subsequent training; it can be turned off during the inference phase or used in a downgraded manner as needed.

[0021] S500, characterization of decomposition and domain alignment: decomposition domain invariant / domain specific and performing distribution and adversarial alignment; In one embodiment of the present invention, the following steps are specifically included: S510, Representation Decomposition: The fused shared latent space representation is split into domain-invariant components and domain-specific components for subsequent alignment and dedomaining. ; in, It is a domain-invariant projector, consisting of two layers of 256→256 dimensional MLP and ReLU, outputting domain-invariant components. ; For domain-specific projectors, the structure and Same but with independent parameters, specific components in the output domain Both lines are connected to Layer Norm and then fed downstream.

[0022] S520, Distribution Alignment (MMD): In each training step, the batch samples are divided according to the domain label, the mean and covariance are calculated online, and the distribution alignment loss is obtained by substituting them into the formula. After multiplying by the weights, the loss is added to the total loss. ; ; ; in, The mean of the source domain. The mean of the target domain; This is the sample covariance matrix for the corresponding domain; It is the L2 norm. It is the Frobenius norm; These represent the mini-batch sample index sets of the source and target domains, respectively. For distribution alignment loss, Alignment weights for the distribution.

[0023] S530, Adversarial Removal (GRL): The domain-invariant components are fed into the domain discriminator via GRL to obtain the domain prediction. The cross-entropy is then calculated with the true label to obtain the adversarial loss. This loss is multiplied by the weights and added to the total loss to achieve adversarial learning of the domain-invariant representation. ; in, This is a gradient inversion layer; forward identity and backward multiplication are performed. ( ); The domain discriminator consists of two layers of 256→128→K-dimensional MLP and ReLU, and outputs K-dimensional logits (K is the number of domain categories). For a moment Real domain tags, ; Cross-entropy loss; expectation Approximate average on mini-batch To counter the weight, To combat the losses.

[0024] S600, biased contrastive learning: domain-aware sampling and weighted contrastive optimization; In one embodiment of the present invention, the following steps are specifically included: S610, de-biased sampling: Positive samples Semantically consistent but with different domains; negative samples Semantic differences lead to the exclusion of easily confused sets. ; in, For sample index; Points and samples The set of "easily confused" samples with similar semantic labels but different domain labels is obtained by filtering with a pre-trained semantic similarity threshold to ensure that the semantic differences of negative samples are significant. S620, Domain Weighted Comparison: Increases the weight of low-frequency domain samples to reduce bias; ; in, For the inverse frequency weight of the domain, For temperature, For similarity, For the sample The fusion representation (i.e., the output of S320) ); For the sample Positive sample representation, domain label and Different but with the same semantic tags; negative samples The representation, semantic tags and Different and ; The domain-weighted InfoNCE contrastive loss is applied to all samples within the current mini-batch. Accumulation; It is the natural logarithm; It is an exponential function; For the sample The domain label takes values ​​ranging from 0 to K-1, where K is the number of domain categories; S700, Semantic Anchoring and Cyclic Consistency: Constructing cross-modal anchors and optimizing consistency and mutual information proxy; In one embodiment of the present invention, the following steps are specifically included: S710, Anchor Construction: ASR is called on the audio stream to obtain text embeddings; Scene Graph is called on the video stream to obtain scene graph embeddings; text embeddings, scene graph embeddings, and external common sense are concatenated into a 384-dimensional vector; and fed into the semantic fusion network to obtain 256-dimensional semantic anchors and cache them. ; in, For a moment Cross-modal semantic anchor vectors, This is a semantic fusion network, consisting of two layers of 256-dimensional MLP and ReLU. The input is concatenated and the output is a unified semantic space. For the automatic speech recognition module, audio features are converted into text embeddings, outputting a 128-dimensional vector. For audio features, Visual scene graph extractor extracts image features. Parsed as object-relational embedding. For image features, Provide prior semantic constraints for the external common sense ontology, and then concatenate it with ASR and Scene Graph before feeding it in. ; S720, Semantic Consistency Loss: Calculation and The cosine similarity is used to randomly select semantically mismatched negative sample anchors within the same batch, and the hinge loss is calculated to obtain... Later included in the total loss; ; in, As weight, This is a hyperparameter for margins; The cosine similarity is given, with an output range of [-1, 1]. For field-invariant representation, For positive sample semantic anchors, As negative sample anchors, random sampling is performed from different semantic fragments within the mini-batch. For hinge damage; S730, Cyclic Consistency Loss: Audio features are fed into an audio modal encoder to obtain a 256-dimensional audio representation, which is then processed by a text-image generator to generate 128-dimensional pseudo-visual features. The pseudo-visual features are fed into a visual modal encoder to obtain a 256-dimensional cyclic representation. The original visual features are directly fed into a visual modal encoder to obtain a 256-dimensional true representation. The squared L2 distance between the two is calculated to obtain the cyclic consistency loss, which is then added to the total loss. ; in, For cycle-consistent loss, It is an audio modal encoder that outputs a 256-dimensional representation. This text-to-image generator transforms 256-dimensional audio representations into pseudo-visual features. It is a visual modal encoder that outputs a 256-dimensional representation. Original visual features; S740, Mutual Information Agent: will Feeded via GRL Calculate the cross-entropy with the domain label to obtain the domain loss term; then combine the same... Send directly The cross-entropy is calculated with the semantic labels to obtain the semantic loss term; the calculation yields... It also incorporates the total loss to achieve the goal of "suppressing domain information and enhancing semantic information"; ; in, For semantic terms; This is a gradient reversal layer; the forward gradient is identical, and the backward gradient is multiplied by -1. For the domain discriminator, a 2-layer MLP and ReLU with dimensions 256→128→K is used to output K-dimensional logits, where K is the number of domain categories. For real domain labels, integers 0…K-1, For semantic classifiers, a 2-layer MLP with 256→128→C-dimensional ReLU is used, outputting C-dimensional logits, where C is the number of semantic categories. For true semantic labels, integers 0…C-1, For semantic term coefficients, This results in mutual information loss.

[0025] S800, Reliability-Weighted Fusion and Security Decision-Making: Domain Adaptive Belief Fusion and Imposition of Security Constraints; In one embodiment of the present invention, the following steps are specifically included: S810, Domain Reliability Estimation: Retrieved by reading the pre-stored domain risk table. Obtained by looking up the table according to the modality Substitute into Sigmoid to calculate Result caching; ; in, For modality In the domain The reliability confidence is as follows; For the Sigmoid function; For modality The bias term, typically 0.5; For modality The slope coefficient, typically 2.0; For modality In the domain Risk score below, 0 indicates no risk, and 1 indicates the highest risk; S820, Reliability-Weighted Fusion: The initial fusion weights are multiplied by the reliability confidence values ​​for each mode to obtain a new reliability-weighted weight, then L1 normalization is performed to ensure the sum is 1; the new weights are then applied to... After weighted summation, a 256-dimensional output is obtained through a shared 1×1 convolution. ,cache; ; in, For a moment Modal The initial fusion weights, For modality In the domain The reliability confidence is as follows; The new weights are based on reliability. This is the weight vector; For weighted summation + 1×1 convolution fusion operator, The result of the fusion; S830, Safety Constraint Decision: The policy network first outputs the probability of all actions, and then calculates the probability of each action one by one. Remove more than After the action, select the action with the highest probability from the remaining actions as... If the action is not met, the safety barrier will be triggered (e.g., slow braking to a stop), and a warning sign will be recorded. ; in, For policy distribution, For state, For a set of actions, This represents the upper limit of risk. For the final action to be performed; For discrete action space (such as the set of acceleration, deceleration, and steering angles); The action probability distribution output by the policy network; For a moment The extended state vector (including vehicle speed, obstacle distance, etc.); Output the action-state risk estimation function; To set a preset risk limit; S840, Evidence Record: [The following appears to be a separate, unrelated sentence:] Write local circular logs according to timestamps, with each record appended with a Unix timestamp and frame ID. The logs are periodically compressed and uploaded to the cloud for incident backtracking, model auditing, and visualization explanation. in, Cosine similarity between the domain-invariant representation and the semantic anchor; For modal reliability; Actions output by the S830; The risk value for the corresponding action; S900, online personalized updates and cross-domain distillation: federated aggregation and cloud distillation, delivering lightweight, adaptable incremental updates; In one embodiment of the present invention, the following steps are specifically included: S910, Fed Aggregation (Fed Avg): Collects local parameters uploaded by each client in the cloud. and corresponding sample size A new round of global parameters is obtained by weighting the samples according to their proportions. And cache it for the next round of distribution; ; in, For the first Round global model parameters; For the first Round Local parameters for each client; For the first Local sample size for each client; Total number of samples from all clients; S920, Cloud Distillation: The federated global model is used as the teacher, and the small models distributed in the previous round are used as the students; forward inference is performed on the public unlabeled data to calculate the output distribution of the two at the temperature, calculate the KL divergence loss, and then the student model is updated in reverse to achieve knowledge transfer. ; in, Distribution by teacher / student For temperature; For the teacher model at temperature The distribution of soft labels below; For the student model's predicted distribution at the same temperature, The distillation temperature is used to determine the softening probability. KL divergence measures the difference in distribution. For KL divergence loss, This is unlabeled data; S930, Lightweight Adaptation and Distribution: Perform channel pruning and quantization on the distilled student model to obtain lightweight adapter parameters; re-statistically analyze semantic co-occurrence using the current round of federated data, update the edge weights and node confidences of the common sense ontology, and generate incremental updates; package the lightweight adapter parameters and incremental updates into an OTA patch and distribute it to each terminal to complete the closed loop.

[0026] ; in, This is the increment of the compressed adapter parameters. For structured pruning or low-rank decomposition, the compression rate is ≥80%. For incremental updates of the common sense ontology, To fine-tune the weights or relationships of ontology nodes based on the semantic statistics of the new samples in this round.

[0027] like Figure 2 As shown, based on a multimodal deep fusion human-computer interaction method for an intelligent cockpit, a multimodal deep fusion human-computer interaction system for an intelligent cockpit is also provided, including the following modules: Acquisition and event coding module 101: Synchronizes the timestamps of multimodal data and labels the confidence level, realizes automatic filling of missing modes and noise filtering, and builds a robust input pipeline; Asynchronous Spatiotemporal Alignment Module 102: Learns dynamic time warp functions to align cross-modal sequences and generates soft-aligned fused fragments through a window attention mechanism; Representation and Initial Fusion Module 103: Extracts deep features of each modality and constructs a joint embedding space, optimizes the self-supervised reconstruction loss to achieve initial feature fusion; Domain Context Estimation and Conditional Normalization Module 104: Estimates domain statistical properties (such as illumination / device differences), performs domain conditional normalization, and generates enhanced samples; Representation decomposition and domain alignment module 105: Decouple features as domain-invariant / specific components, use adversarial learning to align cross-domain distributions, and minimize domain differences; Debiased contrastive learning module 106: Dynamically constructs positive and negative sample pairs based on domain attributes and optimizes the weighted contrastive loss function to eliminate domain bias; Semantic Anchoring and Cyclic Consistency Module 107: Constructs cross-modal semantic anchors to establish alignment benchmarks and maximizes modal mutual information through cyclic consistency loss; Reliability weighted fusion and safety decision module 108: Calculates modal-level confidence weights for adaptive fusion and applies safety constraint boundaries to prevent high-risk predictions; Online Individualized Update and Cross-Domain Distillation Module 109: Federated Aggregation Client Model Updates Global Knowledge Base, Cloud Distillation Deploys Lightweight Incremental Models to Local Devices.

[0028] Based on the above system and method, the following example is given for a multimodal driver state monitoring system; Scenario Description: During vehicle operation, the system monitors the driver's status in real time using cameras (vision), steering wheel sensors (tactile), microphones (auditory), and bio-radar (physiological signals). When the system detects fatigued driving or distracted behavior, it automatically triggers tiered warnings (seat vibration, voice reminder, emergency braking).

[0029] It should be noted that the cockpit domain controller is equipped with heterogeneous computing units (SoC / NPU / GPU) and integrates a multimodal sensing array: Acoustic system: 8-microphone array (supporting beamforming and noise suppression); Vision system: 200° FOV fisheye camera, infrared binocular camera (nighttime pupil tracking), ToF type sensor (gesture depth perception); Biosensing: Steering wheel capacitive film (grip force detection), seat piezoelectric sensor (sitting posture analysis), millimeter-wave radar (vital signs monitoring); Interactive devices: haptic feedback steering wheel, eye-tracking module (50Hz sampling), facial key point camera; Hardware collaboration features: Synchronize multi-source data streams through a unified time service (μs-level accuracy), enable causal triggering between sensors through an event bus (such as activating heart rate radar when yawning is detected), and dedicate an NPU accelerator to hierarchical fusion computing; Here is an example of module collaboration: Data Acquisition and Event Encoding: Synchronize 4-channel sensor timestamps (0.1ms accuracy) and label the confidence level of low-light frames from nighttime cameras; fill in historical heart rate data when bio-radar fails; filter steering wheel noise caused by engine vibration; Asynchronous spatiotemporal alignment: Aligning slow physiological signals (1Hz heart rate) with fast visual data (30fps eye movement) to capture cross-modal correlation fragments such as "yawning, slight steering wheel tremor, and decreased heart rate"; Characterization and initial fusion: Visual features such as eyelid closure degree, steering wheel grip force fluctuation, and acoustic features such as yawn frequency were extracted. Preliminary fusion revealed that the simultaneous occurrence of eyelid closure and sudden drop in grip force is a high-risk fatigue feature. Domain context estimation: Identify current domain features: nighttime highway sections (low light, high frequency vibration), enhance eye features in low light, and simulate seat pressure distribution for drivers of different body types; Characterization decomposition and alignment: Separate general fatigue characteristics (blink frequency) from vehicle-specific characteristics (sport seat pressure threshold) to ensure that fatigue assessment criteria are consistent for sedans and SUVs; Bias-free contrastive learning: Constructing positive samples: standard fatigue states of different drivers; Eliminating sample bias: avoiding misjudging driving while wearing sunglasses as drowsy; Semantic anchoring: Establish cross-modal anchor points: Define "severe fatigue" as closing your eyes for more than 2 seconds, while your heart rate drops by more than 15% of the normal driving heart rate value, ensuring that the visual judgment of "yawning" is synchronized with the acoustic detection of yawning sound; Safety decisions: Increase the weight of visual modalities when driving at night (because steering wheel vibrations are easily disturbed by road conditions), and trigger strict safety boundaries on highways: activate a level one alarm even at the slightest fatigue; Online updates: Aggregate driving modes of 1,000 vehicles to the cloud and deliver personalized incremental models to specific drivers (such as identifying fatigue characteristics of people wearing glasses).

[0030] The desired effect is as follows: Precise monitoring: Improved fatigue detection accuracy and reduced false alarm rate (especially addressing misjudgments caused by changes in sunglasses / lighting conditions); Cross-domain adaptation: The deployment time of new models is shortened, and the detection rate remains above 80% under extreme environments (strong light / heavy rain); Safety Enhancement: Improved early warning response speed (critical accident avoidance window), tiered early warning reduces false alarm rate and increases driver acceptance; Personalized service: After learning individual biometric characteristics, fatigue recognition accuracy is improved, and the local incremental model has a low memory usage ratio; Typical applications of this system include real-time safety monitoring of car-sharing platforms or intelligent cockpit interaction systems for high-end passenger vehicles.

[0031] The embodiments of the present invention have been described above, but the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention, all of which are within the protection scope of the present invention.

Claims

1. A multimodal deep fusion human-computer interaction method for an intelligent cockpit, characterized in that, Includes the following steps: S100, Acquisition and Event Encoding: Unified Time and Confidence, Supports Modal Missing and Noise Robustness; Synchronize the timestamps of 4 sensors to mark the confidence level of low-light frames from the night camera; fill in historical heart rate data when the bio-radar fails; filter steering wheel noise caused by engine vibration; S200, Asynchronous Spatiotemporal Alignment: Learning Temporal Warp and Window Attention, Soft Alignment Constructs Fusion Fragments; Align slow physiological signals with fast visual data to capture cross-modal correlation fragments such as "yawning, slight steering wheel tremor, and decreased heart rate"; S300, Characterization and Initial Fusion: Extract modal representations, fuse and optimize self-supervised objectives; By extracting visual features such as eyelid closure degree, steering wheel grip force fluctuations, and acoustic features such as yawn frequency, preliminary fusion revealed that the simultaneous occurrence of eyelid closure and a sudden drop in grip force is a high-risk fatigue characteristic. S400, Domain Context Estimation and Conditional Normalization: Estimate the domain context and perform conditional normalization and enhancement; Identify current domain features, low light, high frequency vibration nighttime highway sections, enhance eye features under low light, and simulate seat pressure distribution for drivers of different body types; S500, characterization of decomposition and domain alignment: decomposition domain invariant / domain specific and performing distribution and adversarial alignment; By separating general fatigue characteristics from vehicle-specific characteristics, the fatigue assessment standards for sedans and SUVs can be made consistent. S600, biased contrastive learning: domain-aware sampling and weighted contrastive optimization; Construct positive samples representing the standard fatigue state of different drivers; eliminate sample bias to avoid misjudging drivers wearing sunglasses as drowsy; S700, Semantic Anchoring and Cyclic Consistency: Constructing cross-modal anchors and optimizing consistency and mutual information proxy; The steps for semantic anchoring and loop consistency are as follows: S710, Anchor Construction: ASR is called on the audio stream to obtain text embeddings; Scene Graph is called on the video stream to obtain scene graph embeddings; text embeddings, scene graph embeddings, and external common sense are concatenated into a 384-dimensional vector; and fed into the semantic fusion network to obtain 256-dimensional semantic anchors and cache them. S720, Semantic Consistency Loss: Calculate the cosine similarity between the domain-invariant representation and the semantic anchor point of the positive sample. Randomly select semantically mismatched negative sample anchor points within the same batch, calculate the hinge loss, and then incorporate the hinge loss into the total loss. S730, Cyclic Consistency Loss: Audio features are fed into an audio modal encoder to obtain a 256-dimensional audio representation, which is then processed by a text-image generator to generate 128-dimensional pseudo-visual features. The pseudo-visual features are fed into a visual modal encoder to obtain a 256-dimensional cyclic representation. The original visual features are directly fed into a visual modal encoder to obtain a 256-dimensional true representation. The squared L2 distance between the two is calculated to obtain the cyclic consistency loss, which is then added to the total loss. S740, Mutual Information Proxy: The domain-invariant representation is fed into the domain discriminator via GRL and cross-entropy is calculated with the domain label to obtain the domain loss term; the same domain-invariant representation is directly fed into the semantic classifier and cross-entropy is calculated with the semantic label to obtain the semantic loss term; the mutual information loss is calculated and added to the total loss; S800, Reliability-Weighted Fusion and Security Decision-Making: Domain Adaptive Belief Fusion and Imposition of Security Constraints; The steps of reliability-weighted fusion and security decision-making are as follows: S810, Domain Reliability Estimation: Read the pre-stored domain risk table to obtain the risk score, look up the table by modality to obtain the bias term and slope coefficient, substitute them into the Sigmoid function to calculate the reliability confidence, and cache the results; S820, Reliability-Weighted Fusion: The initial fusion weights are multiplied by the reliability confidence for each modality to obtain new reliability-weighted weights, and then L1 normalization is performed to ensure that the sum is 1; the modal representations are weighted and summed according to the new weights, and then a 256-dimensional fusion result is output through a shared 1×1 convolution. S830, Safety Constraint Decision: The policy network first outputs the probability of all actions, calculates the action-state risk estimation function output one by one, removes actions that exceed the risk limit, and selects the action with the highest probability from the remaining actions as the final action to be executed; if there is no action that meets the requirements, the safety barrier is triggered and an alarm flag is recorded. S840, Evidence Recording: The cosine similarity of domain-invariant representations and semantic anchors, modal reliability, action and risk values ​​are written to a local circular log by timestamps. Each record is appended with a Unix timestamp and frame ID. The log is periodically compressed and uploaded to the cloud for incident backtracking, model auditing and visualization interpretation. S900, online personalized updates and cross-domain distillation: federated aggregation and cloud distillation, delivering lightweight, adaptable incremental updates.

2. The multimodal deep fusion human-computer interaction method for an intelligent cockpit according to claim 1, characterized in that, The steps of data acquisition and event coding are as follows: S110, Confidence normalization and masking: Read the original confidence vector modally, perform Softmax normalization according to the time dimension, and obtain the normalized confidence; Then, it is compared with a threshold to generate a binary mask; S120, Timestamp Unification and Lightweight Encoding: First, call the cross-modal time synchronization function to complete time alignment, then concatenate the original features, original confidence and context or device metadata and send them to the lightweight encoder to output lightweight features and cache them; S130, Robust Encoding of Missing Modal: If the valid event mask is valid, the encoded feature vector is directly retained; otherwise, the interpolation function is triggered, and the interpolation value is calculated using the same modal neighborhood or cross-modal aligned features. The robust feature representation is then output and sent downstream.

3. The multimodal deep fusion human-computer interaction method for an intelligent cockpit according to claim 1, characterized in that, The steps for asynchronous spacetime alignment are as follows: S210, Time Warp Function Learning: Discretize a unified clock with 10ms intervals, optimize the piecewise linear warp function using dynamic programming or gradient descent, initialize with hardware timestamp linear mapping, and cache the optimal time warp function after 50 iterations to converge. S220, Window soft alignment: A sliding window is established for each uniform time step, and the numerator and denominator are calculated only within the window to avoid global traversal; the result is cached as a sparse weight matrix; S230, Fragment Construction: Sparse matrix multiplication is used on the GPU to achieve weighted summation, the output frame rate is uniform, forward padding is used for missing weighted frames to ensure temporal continuity, and the result is cached as an aligned fragment representation.

4. The multimodal deep fusion human-computer interaction method for an intelligent cockpit according to claim 1, characterized in that, The steps for characterization and initial fusion are as follows: S310, Representation Extraction: The 128-dimensional aligned segment is fed into the corresponding modal encoder, and a 256-dimensional representation is output and cached. S320, Initial Fusion: First, the representation vector is weighted and summed according to the fusion weight vector to obtain a temporary vector, and then the final shared latent space representation is obtained by shared 1×1 convolution. S330, self-supervised objective: where the comparison term is: the final shared latent space representation of 256 dimensions for each modality pair is used as a positive sample, and the same mode / different mode at other times is used as a negative sample, with a temperature coefficient of 0.1, and the InfoNCE loss is calculated; The reconstruction term is as follows: the encoded features are randomly occluded by the mask, the occluded features are concatenated with the modal representation and sent to the decoder to predict the masked part, and the L1 error is used as the measure. The weighted sum of the two results in the monitoring loss, which is used for end-to-end backpropagation.

5. The multimodal deep fusion human-computer interaction method for an intelligent cockpit according to claim 1, characterized in that, The steps of domain context estimation and conditional normalization are as follows: S410, Domain Context Estimation: The pre-trained domain discriminator is used to classify the fused shared latent space representation into domains, and the posterior probability of belonging to each domain at the current time is output, providing domain context for subsequent conditional normalization and enhancement. S420, Conditional Normalization: Dynamically generate scaling and offset factors based on the domain context, and perform channel-by-channel conditional normalization of the input features to alleviate distribution offset between different domains; S430, Lightweight Style Enhancement: Apply random style perturbations to visual or sensor features during the training phase to improve the model's robustness to domain style changes such as brightness, contrast, noise, and fog. The perturbation amplitude is sampled from a uniform distribution, ensuring that the enhancement is lightweight and can be completed online in real time.

6. The multimodal deep fusion human-computer interaction method for an intelligent cockpit according to claim 1, characterized in that, The steps for characterization decomposition and domain alignment are as follows: S510, Representation Decomposition: The fused shared latent space representation is split into domain-invariant components and domain-specific components for subsequent alignment and dedomaining. S520, Distribution Alignment: Within each training step, batch samples are divided according to domain labels, the mean and covariance are calculated online, and the distribution alignment loss is obtained by substituting them into the formula. After multiplying by the weights, the loss is added to the total loss. S530, Adversarial Dedomaining: The domain-invariant components are fed into the domain discriminator through a gradient inversion layer to obtain the domain prediction. The cross-entropy is then calculated with the true label to obtain the adversarial loss. This loss is multiplied by the weights and added to the total loss to achieve adversarial learning of the domain-invariant representation.

7. The multimodal deep fusion human-computer interaction method for an intelligent cockpit according to claim 1, characterized in that, The steps for bias-free contrastive learning are as follows: S610, de-biased sampling: Positive samples are semantically consistent but have different domains; Negative samples are semantically different and are excluded from the easily confused set; S620, Domain Weighted Comparison: Increases the weight of low-frequency domain samples to reduce bias.

8. A multimodal deep fusion human-computer interaction system for an intelligent cockpit, characterized in that, The steps for performing the multimodal deep fusion human-computer interaction method for a smart cockpit as described in any one of claims 1-7 include: Acquisition and event coding module: Synchronizes timestamps of multimodal data and labels confidence levels, realizes automatic filling of missing modalities and noise filtering, and builds a robust input pipeline; Asynchronous spatiotemporal alignment module: learns dynamic time warp functions to align cross-modal sequences and generates soft-aligned fused fragments through a window attention mechanism; The representation and initial fusion module extracts deep features from each modality and constructs a joint embedding space, optimizing the self-supervised reconstruction loss to achieve initial feature fusion; Domain context estimation and conditional normalization module: Estimates domain statistical properties, performs domain conditional normalization, and generates enhanced samples; Representation decomposition and domain alignment module: Decouple features as domain-invariant / specific components, use adversarial learning to align cross-domain distributions, and minimize domain differences; The bias-reduction contrastive learning module dynamically constructs positive and negative sample pairs based on domain attributes and optimizes the weighted contrastive loss function to eliminate domain bias. Semantic Anchoring and Cyclic Consistency Module: Constructs cross-modal semantic anchors to establish alignment benchmarks and maximizes modal mutual information through cycle consistency loss; Reliability-weighted fusion and safety decision module: Calculates modal-level confidence weights for adaptive fusion and applies safety constraint boundaries to prevent high-risk predictions; Online individualized update and cross-domain distillation module: Federated aggregation client model updates global knowledge base, cloud distillation of lightweight incremental models and distribution to local devices.

Images