Hierarchical cross-domain information injection method and system for assisted driving system

Abstract

The present application relates to computer vision and artificial intelligence technology, aiming at providing a hierarchical cross-domain information injection method and system for an assisted driving system. The method comprises: extracting initial feature sequences of a driver domain and an environment domain respectively, and obtaining information encoded feature sequences after injecting learnable embedding vectors; respectively introducing learnable query vectors after respectively performing splicing processing, refining information through a multi-head cross attention mechanism, and forming fused backbone representation sequences; performing interactive information coupling processing by using a dominant-complementary asymmetric injection mechanism, and obtaining corresponding feature flow input to a decoupled multi-task learning network downstream to perform specific driving task prediction. The present application can process more complex multi-modal data and has good plug-and-play scalability; the dominant-complementary mechanism can effectively protect the core semantics of the driver domain and the environment domain, and can realize higher accuracy of driver and environment domain tasks compared with the prior art.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, specifically to a hierarchical cross-domain information injection method and system for driver assistance systems, which is a novel multimodal information fusion technology. Background Technology

[0002] Currently, Advanced Driver Assistance Systems (ADAS) are evolving from partial automation (L2) to conditional automation (L3) and higher levels. The core characteristic of this evolution is that the system's perception boundaries are no longer limited to the pure monitoring of the external road environment (such as vehicles, lanes, and traffic signals), but extend to a deep understanding of the driver's internal state (such as emotions, behavior, and attention). To achieve truly safe and intelligent assisted driving, the system must possess the ability to perform collaborative reasoning between in-cabin and out-of-cabin information.

[0003] In real-world driving scenarios, there is a close causal coupling between the driver's internal state and the external environment at the information level. For example, external traffic congestion (environmental information) may cause the driver to become anxious or engage in distracting behaviors such as drinking water (driver state information), and these distracting behaviors may become the direct cause of traffic accidents. Therefore, analyzing information from either domain in isolation is one-sided. The system must be able to understand complex cross-domain causal chains such as "because of traffic congestion, the driver is impatiently drinking water" in order to make more accurate risk predictions and decision support. However, existing research paradigms often treat these two task spaces separately, with one part focusing on the perception of the external traffic scene and the other on the monitoring of the driver's state. This separation ignores the crucial intrinsic connection between the two, thus limiting the system's intelligence and safety.

[0004] To address this challenge, existing technologies have primarily explored the path of multimodal information fusion, but they generally suffer from the following limitations, which constitute key bottlenecks restricting the improvement of current ADAS perception performance. Early multimodal fusion methods typically involved directly concatenating feature vectors from different modalities or fusing them through simple linear transformations, weighted summations, etc. Further, the cross-attention mechanism based on the Transformer architecture was introduced into the field of assisted driving to fuse in-vehicle and external features; this is known as a flattened fusion strategy. This strategy works well when dealing with homogeneous modalities with relatively aligned feature spaces and similar semantic levels (such as fusing images from cameras at different angles). However, in cross-domain scenarios, such as fusing micro-expression features expressing driver emotions with traffic light features representing traffic rules, a significant "semantic gap" exists between the two. The former represents sophisticated, dynamic biometric features, while the latter represents concise, explicit symbolic information. Forcibly aligning two features that differ significantly in semantic dimension and data distribution not only introduces a large amount of noise but also dilutes the most valuable core semantic information in each modality, severely weakening the model's comprehensive judgment ability. Representative works include Cross ViT, which uses a two-branch multi-scale fusion, ViLBERT, which uses a two-stream hierarchical attention mechanism, and LXMERT, which uses alternating self-attention and cross-attention layers. In the field of assisted driving, recent methods apply cross-attention to fuse in-vehicle and out-of-vehicle features, while separating task-shared and private modalities to improve learning. These methods all achieve dynamic information interaction between modalities through attention mechanisms. However, although the above methods improve the fusion effect to some extent, their fusion process still has a fundamental flaw: it still adopts a flattened fusion strategy, lacks an effective semantic abstraction level, and does not effectively fuse in-vehicle driver and out-of-vehicle environment information, limiting the upper limit of ADAS.

[0005] Therefore, this invention proposes a new information injection scheme to solve the above problems. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a hierarchical cross-domain information injection method and system for driver assistance systems.

[0007] To solve the technical problem, the solution of the present invention is:

[0008] A hierarchical cross-domain information injection method for driver assistance systems is provided, including:

[0009] Based on the different information sources, the data streams are divided into driver domain and environment domain, and the initial feature sequences corresponding to each data stream are extracted respectively;

[0010] Two learnable embedding vectors are injected into each initial feature sequence to obtain an information-encoded feature sequence. The modality-type embedding vector and positional encoding are used to distinguish the different sources of features and their inherent spatiotemporal order.

[0011] The encoded feature sequences of the driver domain and the environment domain are concatenated to obtain their respective unified feature sequences. Then, learnable query vectors are introduced into each sequence, and information is extracted from the unified feature sequences through a multi-head cross-attention mechanism to form a fused backbone representation sequence.

[0012] A dominant-complementary asymmetric injection mechanism is adopted to perform interactive information coupling processing on the backbone representation sequences of the driver domain and the environment domain, resulting in two feature streams: driver domain features and environment domain features. These features are then input into the downstream decoupled multi-task learning network for specific driving task prediction.

[0013] As a preferred embodiment of the present invention, the data stream is input from multiple source devices through one or more shared backbone networks. The multiple source devices include: multiple cameras for acquiring video of the vehicle's external environment, sensors for acquiring video of the vehicle's external environment, multiple cameras for acquiring video of the vehicle's internal environment, and at least one camera for acquiring the driver's face, body, and posture. The sensors are at least one of lidar, millimeter-wave radar, or ultrasonic radar.

[0014] As a preferred embodiment of the present invention, the feature sequence of the information encoding is specifically calculated using the following formula:

[0015]

[0016] in: It is a feature sequence that has been encoded with information; This sequence is the initial characteristic sequence, representing the set of matrices consisting of s rows and d columns of real elements. represents the set of real numbers, s represents the sequence length, and d represents the feature dimension; It is a modality embedding, used to give unique labels to features from different sources; It is a positional encoding used to inject the order information of the absolute or relative temporal or spatial location of each feature in a sequence.

[0017] As a preferred embodiment of the present invention, when processing the feature sequence of the environmental domain, the feature sequence of the vehicle's external environment is first spliced ​​and information extracted to obtain a fused representation sequence; then a learnable query vector is introduced again, and the representation sequence is processed with the feature sequence of the vehicle's internal environment through a multi-head cross-attention mechanism to obtain the backbone representation sequence of the environmental domain, and the backbone representation sequence is used for information coupling processing of the dominant-complementary asymmetric injection mechanism.

[0018] As a preferred embodiment of the present invention, the operation of the multi-head cross-attention mechanism specifically includes:

[0019] When processing the feature sequences of the environmental domain, we first consider the feature sequences of the vehicle's external environment. After splicing and information extraction, the fused representation sequence is obtained. The calculation process is shown in the following formula:

[0020]

[0021] In the formula, The formula for calculating the attention mechanism is given; where is the learnable query vector. As the query matrix Q, the key matrix K, and the value matrix V, all use feature sequences. ;

[0022] Then, the representation sequence is processed again through a multi-head cross-attention mechanism. Characteristic sequences of the vehicle's interior environment By splicing and extracting information, the backbone representation sequence of the environmental domain is obtained. ;

[0023] When processing the feature sequences of the driving domain:

[0024] Referring to the processing method of vehicle external environment feature sequences, the driver-related feature sequences are spliced ​​together to obtain features. Then use a learnable query vector that is only used in the driver domain. After aggregation, the backbone representation of the driver domain is directly generated. .

[0025] As a preferred embodiment of the present invention, the dominant-complementary asymmetric injection mechanism refers to using one of the backbone representation sequences of the driver domain and the environment domain as the dominant query to focus on the complementary representation in the other backbone representation sequence; then selectively absorbing information according to its own semantic needs, and completing the fusion through residual connection, so that the absorbed information is only used as an incremental adjustment to the dominant representation.

[0026] As a preferred embodiment of the present invention, an interactive information coupling processing is achieved by employing a dominant-complementary asymmetric injection mechanism, enabling environmental representation to perceive the driver's current state and enabling the driver's representation to perceive changes in the external environment; wherein...

[0027] Enabling environmental representation to perceive the driver's current state is achieved through the following method: using a backbone representation sequence of the environmental domain. As the primary query, focus on the backbone representation sequence of the driver domain. And through residual connections, an environmental representation sequence with driver state and context awareness is generated. The specific calculation process is as follows:

[0028] In the formula, The computational formula representing the attention mechanism; where the backbone representation sequence of the environment domain is used. As a query matrix Key matrix Sum matrix All use the backbone representation sequence of the driver domain ;

[0029] Enabling driver representations to perceive changes in the external environment is achieved through the following method: using the backbone representation sequence of the driver domain. As the primary query, focus on the backbone representation sequence of the environmental domain. A context-aware driver representation is generated through residual connections. The specific calculation process is as follows:

[0030]

[0031] In the formula, The computational formula representing the attention mechanism; where the backbone representation sequence of the driver domain is used. As a query matrix Key matrix Sum matrix All use the backbone representation sequence of the environment domain. .

[0032] As a preferred embodiment of the present invention, the decoupled multi-task learning network specifically refers to: using a hybrid expert model MOE composed of a gating network and an MLP network as the multi-task learning network, wherein the gating network adopts the TOP-1 strategy; and decoupling means dividing the data into a driver domain and an environment domain, using two hybrid expert models to separate and decouple the parameters of these two domains respectively, and then outputting the task category.

[0033] The present invention further provides a hierarchical cross-domain information injection system for driver assistance systems, comprising:

[0034] The feature extraction and encoding module is used to divide the data stream into driver domain and environment domain according to the information source, and extract the initial feature sequence corresponding to each data stream respectively; inject two learnable embedding vectors into each initial feature sequence to obtain the information-encoded feature sequence, and use modality type embedding vectors and position encoding to distinguish the different sources of features and the inherent spatiotemporal order;

[0035] The domain aggregation and extraction module is used to concatenate the encoded feature sequences separately to obtain unified feature sequences for the driver domain and the environment domain. Then, learnable query vectors are introduced into them respectively. The unified feature sequences are extracted through a multi-head cross-attention mechanism to form a fused backbone representation sequence.

[0036] The cross-domain asymmetric injection module is used to perform interactive information coupling processing on the backbone representation sequences of the driver domain and the environment domain using a dominant-complementary asymmetric injection mechanism to obtain driver domain features and environment domain features. These two feature streams are then output to the downstream decoupled multi-task learning network to achieve driving task prediction.

[0037] This invention also provides an assisted driving task prediction method based on hierarchical cross-domain information injection, which inputs the two feature streams of driver domain features and environment domain features obtained by the aforementioned method into a downstream decoupled multi-task learning network, and then performs specific driving task prediction.

[0038] Compared with the prior art, the beneficial effects of the present invention are:

[0039] 1. The hierarchical, asymmetric injection architecture proposed in this invention can handle more complex multimodal data (such as V2X vehicle-road cooperative information that may be integrated in the future); based on the flexible framework, the hierarchical injection system (HIM model) has good plug-and-play scalability.

[0040] 2. Traditional technologies typically employ a flattened fusion strategy (such as splicing or weighted fusion), which often leads to information dilution due to ignoring differences and forcibly merging elements. In contrast, this invention adopts a dominant-complementary mechanism that respects information differences; by clearly defining the primary and secondary interactions, it can effectively protect the core semantics of the driver domain and the environment domain.

[0041] 3. By establishing deep causal relationships between in-vehicle and out-of-vehicle information, this invention can achieve higher accuracy in driver and environmental domain tasks compared to existing technologies. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the workflow of the layered injection model in this invention. Detailed Implementation

[0043] Multimodal autonomous driving systems refer to acquiring various types of information about the surrounding environment through multiple sensors and then fusing this information to comprehensively perceive road conditions. The core of this approach is the "information source." It primarily relies on four core sensors, each responsible for collecting environmental data from different dimensions: the visual modality of cameras, the point cloud modality of lidar, the radar modality of millimeter-wave radar, and the short-range modality of ultrasonic radar. Information from a single sensor has significant limitations; only by fusing multimodal data can autonomous driving perception become more reliable and safer, compensating for the deficiencies of single sensors, improving perception accuracy, and increasing redundancy and safety.

[0044] In multimodal fusion strategies, the ideal fusion model should be "understand first, then fuse": that is, the system should first be able to abstract multi-source information from various domains into high-level, concise semantic concepts (for example, effectively aggregating information such as the driver's face, posture, and gaze into the high-level concept of "fatigue driving"), and then establish connections and perform reasoning between these highly abstract semantic concepts (such as "fatigue driving" and "sharp bend ahead"). This invention overcomes the shortcomings of existing "flattened" fusion strategies in fusing multimodal information inside and outside the vehicle, which result in semantic dilution and a lack of semantic abstraction levels. It proposes a novel Hierarchical Infusion Module (HIM) to achieve effective fusion of in-vehicle driver state and external environmental information for decoupled multi-task learning in ADAS. Its core is to use the HIM model to efficiently couple multimodal information from the driver domain and the environment domain.

[0045] It should be noted that this invention relates to the application of computer technology in the fields of vision and artificial intelligence, and autonomous driving. The implementation of this invention involves the application of multiple software functional modules. The applicant believes that, after carefully reading the application documents and accurately understanding the implementation principles and objectives of this invention, and in conjunction with existing known technologies, those skilled in the art can fully utilize their software programming skills to implement this invention. The aforementioned software functional modules include, but are not limited to: feature extraction and encoding modules, intra-domain aggregation and refinement modules, cross-domain asymmetric injection modules, etc. All modules mentioned in this application fall within this scope, and will not be listed individually here.

[0046] Part One: Implementation Scheme of the Invention

[0047] 1. Initial Feature Extraction and Information Encoding

[0048] First, initial feature sequences are extracted from input data streams from multiple sources (such as camera video, sensor data, etc.) through one or more shared backbone networks. This sequence represents a set consisting of matrices with s rows and d columns of real elements. represents the set of real numbers; s represents the sequence length; and d represents the feature dimension.

[0049] The multi-source device in this invention includes at least: multiple cameras for acquiring video of the vehicle's external environment, sensors for acquiring video of the vehicle's external environment, multiple cameras for acquiring video of the vehicle's internal environment, and at least one camera for acquiring the driver's face, body, and posture; the sensors are at least one of lidar, millimeter-wave radar, or ultrasonic radar. For example, four cameras are synchronized via a precise timing protocol, with two side cameras mounted in front of each side mirror of the vehicle to generate left and right views capturing the traffic environment. Simultaneously, a front-view camera is mounted in the center of the dashboard to observe the scene ahead. An interior camera non-invasively records the driver's natural reactions from the side to clearly show the interaction of the face, body, and hands with the steering wheel. Because different driving reactions have different durations, the raw video data from the four views is first synchronously processed into 3-second short video clips using a Movimetry Library. The driver's facial bounding box is captured using a face detector MTCNN. Simultaneously, driver-centric information, including body bounding boxes, 2D skeleton pose (26 key points), and gestures (42 key points), is obtained using a pose estimator AlphaPose. Movimetry Library is a Python library specifically designed for multi-view video processing and biomechanical analysis. Its core functionality is to automatically synchronize, calibrate, and crop raw video data from multiple independent devices.

[0050] To enable the HIM model to distinguish features from different data streams and understand their inherent spatiotemporal order, information encoding of the initial feature sequence is necessary. This step is accomplished by injecting two learnable embedding vectors into the initial feature sequence, as shown in the following calculation:

[0051]

[0052] in: These are the encoded input features fed into the HIM model. Modality-type embedding (MTB) assigns unique "identity tags" to features (such as facial images and external scene images) from different cameras or sensors, enabling the model to learn specific patterns for each modality. Positional encoding refers to the process of injecting absolute or relative positional information (such as temporal or spatial order) into each feature in a sequence to compensate for the insensitivity of subsequent attention mechanisms to order.

[0053] 2. Intra-domain feature aggregation and macro-level characterization extraction

[0054] Depending on the source of information, this application distinguishes the sources of multi-source input data streams into driver domain and environment domain; the former includes camera input data for acquiring the real-time status of the driver; the latter includes input data from cameras, lidar, millimeter-wave radar, and ultrasonic radar for acquiring the environmental status of the vehicle.

[0055] Before injecting cross-domain information, the HIM model first performs feature aggregation independently within each domain (driver domain and environment domain), aiming to condense multi-source, fragmented low-level features into a macro-level representation that is information-compact and semantically focused. Based on this operation, this step follows the design principle of "understanding first".

[0056] Taking the environmental domain as an example, the aggregation process is as follows. First, the feature sequences from multiple external cameras (such as forward-looking, left-looking, and right-looking) and radar are concatenated according to feature dimensions to form a unified feature sequence. .

[0057] Next, a learnable query vector is introduced. The feature sequence is processed through a multi-head cross-attention mechanism. Information is extracted from the key and value matrices used in the cross-attention operation to generate a fused representation of the external environment. .

[0058] In this invention, the cross-attention operation is defined as , where Q, K, and V are the query matrix, key matrix, and value matrix, respectively.

[0059] The calculation process is shown in the formula below:

[0060]

[0061] In the formula, the learnable query vector As the query matrix Q, the key matrix K, and the value matrix V, all use feature sequences. ;

[0062] To further enrich the context of environmental characterization, the present invention further proposes to incorporate the above-mentioned... With view feature sequence from in-vehicle camera A fusion process is performed to ultimately generate the backbone representation of the environment domain. .

[0063] The driver domain employs a similar operational process, but only performs a single fusion process. Specifically, driver-related feature sequences (such as face, body, and posture) are concatenated to obtain a unified feature sequence. Then use another separate (driver domain-only) learnable query vector After aggregation, a backbone representation of the driver domain is generated. .

[0064] Through this step, the system obtains two highly condensed macroscopic abstract backbone representations, one for the environment domain and the other for the driver domain. and This laid the foundation for subsequent cross-domain injection.

[0065] 3. Cross-domain asymmetric information injection

[0066] This step is the core of the HIM model, aiming to achieve controlled and efficient information coupling between two domains. This step employs a dominant-complementary asymmetric injection mechanism. In this mechanism, the representation of one domain acts as the "dominant query," actively seeking out the "complementary" representation (as keys and values) of the other domain and selectively absorbing information according to its own semantic needs. Fusion is accomplished through residual connections, ensuring that complementary information is only used for incremental fine-tuning of the dominant representation. This approach, while introducing cross-domain context, preserves the core semantic structure of the dominant domain to the greatest extent possible, effectively avoiding semantic dilution.

[0067] The specific implementation process is as follows:

[0068] To enable the environmental representation to perceive the driver's current state, the following operation was adopted: using the environmental domain backbone representation. As the primary query, focus on the core representation of the driver domain. Furthermore, a context-rich environmental representation with driver state awareness is generated through residual connections. .

[0069] The calculation is as follows:

[0070]

[0071] In the formula, The computational formula representing the attention mechanism; where the backbone representation sequence of the environment domain is used. As a query matrix Key matrix Sum matrix All use the backbone representation sequence of the driver domain ;

[0072] Similarly, to enable the driver representation to perceive changes in the external environment, the following operation was adopted: using the driver domain backbone representation As the primary query, focus on the core representations of the environmental domain. Generate context-aware driver representations .

[0073] The calculation is as follows:

[0074]

[0075] In the formula, The computational formula representing the attention mechanism; where the backbone representation sequence of the driver domain is used. As a query matrix Key matrix Sum matrix All use the backbone representation sequence of the environment domain. .

[0076] 4. Model output feature stream

[0077] After the above three steps, the Hierarchical Injection Model (HIM) finally outputs two deeply coupled, semantically clear, and context-aware feature streams: (Driver domain characteristics) and (Environmental domain features). These two feature streams will then be fed into a downstream decoupled multi-task learning network for specific driving task prediction.

[0078] In current publicly available research, the trend in multi-task learning tends to use hybrid expert models (MOEs). These models consist of a gating network and an MLP network, with the gating network employing a TOP-1 strategy. The decoupled approach described in this invention involves dividing multimodal data into a driver domain and an environment domain; therefore, two independent expert networks are used to separate and decouple the parameters of these two domains, and then the task category is output. In ADAS systems, multimodal data is processed through the HIM to output driver and environment domain features, which are then directly input into the aforementioned decoupled multi-task learning network to output the corresponding task results. The specific structure and usage of the decoupled multi-task learning network can adopt schemes described in publicly available literature. Since this is not part of the core innovation of this invention, it will not be elaborated further.

[0079] 5. Layered cross-domain information injection system

[0080] Based on the above description, this invention proposes a hierarchical cross-domain information injection system for assisted driving (hereinafter referred to as the Hierarchical Injection Model HIM). The workflow of this model is as follows: Figure 1 As shown in the figure, the feature extractors all have the same structure. The embedder is used for initial feature extraction, MTE stands for Modal Encoder, and PE stands for Positional Encoding. Specifically, it includes:

[0081] The feature extraction and encoding module is used to divide the data stream into driver domain and environment domain according to the information source, and extract the initial feature sequence corresponding to each data stream. Two learnable embedding vectors are injected into each initial feature sequence to obtain the information-encoded feature sequence. The modality type embedding vector and position encoding are used to distinguish the different sources of features and the inherent spatiotemporal order.

[0082] For example Figure 1 In the input data, x1 to x4 represent face, side profile, gaze, and gesture (multimodal information related to the driver), respectively, while x5 represents multi-view environmental monitoring information (related to the vehicle). When each data stream passes through the feature extractor, it is first transformed into an initial vector by the embedder, and then modal embedding (MTE) and position encoding (PE) are superimposed. The final output is a standard feature sequence with a dimension of s×d, ensuring that data from different sensors are comparable in the same dimensional space.

[0083] The domain aggregation and refinement module is used to concatenate the encoded feature sequences separately to obtain unified feature sequences for the driver domain and the environment domain. Then, learnable query vectors are introduced into each sequence. The unified feature sequences are refined through a multi-head cross-attention mechanism to form a fused backbone representation sequence.

[0084] For example Figure 1 In the process, the four s×d feature sequences of the driver domain are merged into a unified 4s×d sequence through a concatenation operation (marked as C in the figure), and then refined with the query vector Q through a cross-attention mechanism. At the same time, the environmental domain features are initially adaptively aggregated through the query vector generated by the learnable matrix, thereby completing the refinement from "original features" to "key representations" within each domain and reducing redundant information.

[0085] The cross-domain asymmetric injection module is used to perform interactive information coupling processing on the backbone representation sequences of the driver domain and the environment domain using a dominant-complementary asymmetric injection mechanism to obtain driver domain features and environment domain features. These two feature streams are then output to the downstream decoupled multi-task learning network to achieve driving task prediction.

[0086] For example Figure 1In the interaction process shown in the dashed box on the right, the refined driver features are injected into the driver domain processing unit as the query vector Q, while the environment domain features provide environmental context support as K and V. Similarly, the environment domain processing unit also receives feature injections from the driver domain. This asymmetric coupling method ultimately produces semantically highly condensed driver domain features h. drv and environmental domain characteristics h env This provides accurate data support for subsequent decision-making and prediction.

[0087] In ADAS systems incorporating decoupled multi-task learning networks, the hierarchical injection model (HIM) of this invention is positioned after the multimodal data has passed through the feature extractor. For example, in the scheme presented in the AIDE dataset paper, the hierarchical injection model (HIM) of this invention directly replaces the multimodal fusion module.

[0088] 6. Assisted Driving Task Prediction Methods

[0089] Based on the detailed description of the aforementioned hierarchical cross-domain information injection system, the core operation steps of the hierarchical cross-domain information injection method for driver assistance systems described in this invention can be summarized as follows:

[0090] (S1) Based on the different information sources, the data stream is divided into the driver domain and the environment domain, and the initial feature sequence corresponding to each data stream is extracted respectively;

[0091] (S2) Inject two learnable embedding vectors into each initial feature sequence to obtain information-encoded feature sequences. Use modality-type embedding vectors and positional encoding to distinguish the different sources and inherent spatiotemporal order of features.

[0092] (S3) The encoded feature sequences of the driver domain and the environment domain are concatenated to obtain their respective unified feature sequences; then, learnable query vectors are introduced respectively, and information is extracted from the unified feature sequences through a multi-head cross-attention mechanism to form a fused backbone representation sequence.

[0093] When processing the feature sequence of the environmental domain, the feature sequence of the vehicle's external environment is first spliced ​​and information extracted to obtain the fused representation sequence. Then, a learnable query vector is introduced again, and the representation sequence is processed with the feature sequence of the vehicle's internal environment through a multi-head cross-attention mechanism to obtain the backbone representation sequence of the environmental domain.

[0094] (S4) By adopting a dominant-complementary asymmetric injection mechanism, the backbone representation sequences of the driver domain and the environment domain are interactively coupled to enable the environment representation to perceive the current state of the driver and the driver representation to perceive changes in the external environment. The two feature streams, driver domain features and environment domain features, are obtained and used as inputs to the downstream decoupled multi-task learning network for specific driving task prediction.

[0095] The specific structure and usage of decoupled multi-task learning networks, as well as their application in the driving task prediction process, can be found in publicly available literature (such as the MOE scheme described in Switch Transformers). Since this is not part of the core innovation of this invention, it will not be elaborated further.

[0096] Part Two: Specific Verification Experiments

[0097] 1. Dataset and evaluation metrics

[0098] This application utilizes the open-source AIDE dataset to conduct extensive experiments to evaluate the effectiveness of the proposed holistic network (HD-Nexus) consisting of HIM and a decoupled multi-task learning network in multi-task learning.

[0099] The AIDE dataset, created by the Institute of Engineering and Technology at Fudan University, contains 2,898 samples covering multi-view, multi-modal, and multi-task driving data, including driver facial, body, posture, and gesture information, as well as the vehicle's external environment. Captured through four different camera views, the dataset supports four assisted driving perception tasks based on Driver Emotion Recognition (DER), Driver Behavior Recognition (DBR), Traffic Context Recognition (TCR), and Vehicle Condition Recognition (VCR). The dataset contains 2,898 data samples, totaling 521.64K frames. Each sample consists of a 3-second video clip from four views, with multi-view, multi-modal, and multi-task annotations. Each sample includes video streams from the front, left, right, and in-cabin cameras, as well as driver-specific modalities such as facial / body images and joint-based posture data. All video-based modalities are processed into 16-frame sequences as input to the model. The dataset is divided into a 65% training set, a 15% validation set, and a 20% test set.

[0100] To ensure a fair and comprehensive comparison, the experimental procedure followed the evaluation protocol established in the AIDE dataset, using mean accuracy (mAcc) as the primary metric for evaluating the overall performance of all four tasks. Due to the long-tailed label distribution present in natural driving scenarios, accuracy may not reflect the performance of a few important categories; therefore, the mean weighted F1 score was also reported as a supplementary metric.

[0101] 2. Experimental Setup

[0102] The validation experiments used the proposed HD-Nexus and all baseline models (referring to the choice of feature extractors, including the models in the effect table) implemented in PyTorch and trained on four NVIDIA 4090 GPUs. To ensure consistency between different models and ablation studies, the final output feature dimension of all models was uniformly set to 128.

[0103] The AdamW optimizer was used with an initial learning rate of 10. -4 The weight decays to 5×10 -5 The λ parameter was set to 0.2. All models were trained on the AIDE dataset for 30 epochs, with an effective batch size of 32.

[0104] 3. Effect Comparison

[0105] The training results of each model are shown in Table 1:

[0106] Table 1 Training Results of Each Model

[0107]

[0108] The methods listed in Table 1 include classic baseline methods as well as recently published methods. AIDE is a publicly available multimodal dataset in the field of assisted driving perception (i.e., the AIDE dataset mentioned above), and it serves as the core training / testing benchmark for methods such as MMTL-UniAD. MMTL-UniAD is a unified multimodal and multi-task learning framework for assisted driving perception (A Unified Framework for Multimodal and Multi-Task Learning in Assistive Driving Perception, 2025 arXiv paper, open-source project).

[0109] The models used in the testing process are categorized into 2D, 2D plus temporal, and 3D models according to their paradigms. The method of this invention also belongs to the 3D paradigm. The backbone model needs to process five modalities, including face, body, environment, gesture, and posture. Evaluations, metrics, or results for each model are obtained through training, including: DER (Driver Emotion Recognition), DBR (Driver Behavior Recognition), TCR (Traffic Condition Recognition), VCR (Vehicle Condition Recognition), mAcc (mean Accuracy), and mF1 (mean F1-Score).

[0110] The data in the table shows that the method of this invention is used to fuse the samples. (Driver domain characteristics) and (Environmental domain features) are input into expert networks in different domains for multi-task prediction, achieving significant technical results. For example, the mAcc in the AIDE dataset reaches 85.18%, a significant improvement compared to other baseline models, and also 2.88% higher than the latest state-of-the-art model MMTL-UniAD. Simultaneously, the mF1 score reaches 80.03%, surpassing all baseline models; this metric demonstrates that the model of this invention still exhibits strong system robustness even under the imbalanced AIDE data condition. Typically, in a complete model (referring to the overall ADAS model incorporating decoupled multi-task learning networks), removing the HIM model causes the average accuracy (mAcc) to drop sharply from 85.18% to 79.87%. This significant performance gap directly quantifies the contribution of this invention, proving that the hierarchical information coupling method adopted in this invention can effectively solve the semantic entanglement and information dilution problems caused by "flattened" fusion in existing technologies.

[0111] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A hierarchical cross-domain information injection method for driver assistance systems, characterized in that, include: Based on the different information sources, the data streams are divided into driver domain and environment domain, and the initial feature sequences corresponding to each data stream are extracted respectively; Two learnable embedding vectors are injected into each initial feature sequence to obtain an information-encoded feature sequence. The modality-type embedding vector and positional encoding are used to distinguish the different sources of features and their inherent spatiotemporal order. The encoded feature sequences of the driver domain and the environment domain are concatenated to obtain their respective unified feature sequences. Then, learnable query vectors are introduced into each sequence, and information is extracted from the unified feature sequences through a multi-head cross-attention mechanism to form a fused backbone representation sequence. A dominant-complementary asymmetric injection mechanism is employed to interactively couple the backbone representation sequences of the driver and environment domains, enabling the environment representation to perceive the driver's current state and the driver representation to perceive changes in the external environment. This results in two feature streams: driver domain features and environment domain features, which are then input into a downstream decoupled multi-task learning network for specific driving task prediction. Enabling environmental representation to perceive the driver's current state is achieved through the following method: using a backbone representation sequence of the environmental domain. As the primary query, focus on the backbone representation sequence of the driver domain. And through residual connections, an environmental representation sequence with driver state and context awareness is generated. ; Enabling driver representations to perceive changes in the external environment is achieved through the following method: using the backbone representation sequence of the driver domain. As the primary query, focus on the backbone representation sequence of the environmental domain. A context-aware driver representation is generated through residual connections. .

2. The method according to claim 1, characterized in that, The data stream is input from multiple source devices via one or more shared backbone networks. The multiple source devices include: multiple cameras for acquiring video of the vehicle's external environment, sensors for acquiring video of the vehicle's external environment, multiple cameras for acquiring video of the vehicle's internal environment, and at least one camera for acquiring the driver's face, body, and posture. The sensors are at least one of lidar, millimeter-wave radar, or ultrasonic radar.

3. The method according to claim 1, characterized in that, The feature sequence of the information encoding is specifically calculated using the following formula: ； in: It is a feature sequence that has been encoded with information; This sequence is the initial characteristic sequence, representing the set of matrices consisting of s rows and d columns of real elements. represents the set of real numbers, s represents the sequence length, and d represents the feature dimension; It is a modality embedding, used to give unique labels to features from different sources; It is a positional encoding used to inject the order information of the absolute or relative temporal or spatial location of each feature in a sequence.

4. The method according to claim 1, characterized in that, When processing the feature sequence of the environmental domain, the feature sequence of the vehicle's external environment is first spliced ​​and information extracted to obtain the fused representation sequence. Then, a learnable query vector is introduced again, and the representation sequence is processed with the feature sequence of the vehicle's internal environment through a multi-head cross-attention mechanism to obtain the backbone representation sequence of the environmental domain. The backbone representation sequence is then used for information coupling processing of the dominant-complementary asymmetric injection mechanism.

5. The method according to claim 1, characterized in that, The operation of the multi-head cross-attention mechanism specifically includes: When processing the feature sequences of the environmental domain, we first consider the feature sequences of the vehicle's external environment. After splicing and information extraction, the fused representation sequence is obtained. The calculation process is shown in the following formula: ； In the formula, The formula for calculating the attention mechanism is given; where is the learnable query vector. As the query matrix Q, the key matrix K, and the value matrix V, all use feature sequences. ; Then, the representation sequence is processed again through a multi-head cross-attention mechanism. Characteristic sequences of the vehicle's interior environment By splicing and extracting information, the backbone representation sequence of the environmental domain is obtained. ; When processing the feature sequences of the driving domain: Referring to the processing method of vehicle external environment feature sequences, the driver-related feature sequences are spliced ​​together to obtain features. Then use a learnable query vector that is only used in the driver domain. After aggregation, the backbone representation of the driver domain is directly generated. .

6. The method according to claim 1, characterized in that, The specific calculation process for enabling environmental representation to perceive the driver's current state is as follows: ； In the formula, The computational formula representing the attention mechanism; where the backbone representation sequence of the environment domain is used. As a query matrix Key matrix Sum matrix All use the backbone representation sequence of the driver domain ; The specific calculation process for enabling the driver's character to perceive changes in the external environment is as follows: ； In the formula, The computational formula representing the attention mechanism; where the backbone representation sequence of the driver domain is used. As a query matrix Key matrix Sum matrix All use the backbone representation sequence of the environment domain. .

7. The method according to claim 1, characterized in that, The decoupled multi-task learning network specifically refers to using a hybrid expert model (MOE) composed of a gating network and an MLP network as the multi-task learning network, where the gating network adopts the TOP-1 strategy; and decoupling means dividing the data into a driver domain and an environment domain, using two hybrid expert models to separate and decouple the parameters of these two domains respectively, and then outputting the task category.

8. A hierarchical cross-domain information injection system for assisted driving systems, characterized in that, include: The feature extraction and encoding module is used to divide the data stream into the driver domain and the environment domain according to the information source, and extract the initial feature sequence corresponding to each data stream respectively; Two learnable embedding vectors are injected into each initial feature sequence to obtain an information-encoded feature sequence. The modality-type embedding vector and positional encoding are used to distinguish the different sources of features and their inherent spatiotemporal order. The domain aggregation and extraction module is used to concatenate the encoded feature sequences separately to obtain unified feature sequences for the driver domain and the environment domain. Then, learnable query vectors are introduced into them respectively. The unified feature sequences are extracted through a multi-head cross-attention mechanism to form a fused backbone representation sequence. The cross-domain asymmetric injection module employs a dominant-complementary asymmetric injection mechanism to interactively couple the backbone representation sequences of the driver and environment domains. This allows the environment representation to perceive the driver's current state and the driver representation to perceive changes in the external environment. Ultimately, two feature streams—driver domain features and environment domain features—are obtained and input into the downstream decoupled multi-task learning network for specific driving task prediction. Enabling environmental representation to perceive the driver's current state is achieved through the following method: using a backbone representation sequence of the environmental domain. As the primary query, focus on the backbone representation sequence of the driver domain. And through residual connections, an environmental representation sequence with driver state and context awareness is generated. ; Enabling driver representations to perceive changes in the external environment is achieved through the following method: using the backbone representation sequence of the driver domain. As the primary query, focus on the backbone representation sequence of the environmental domain. A context-aware driver representation is generated through residual connections. .

9. A method for predicting assisted driving tasks based on hierarchical cross-domain information injection, characterized in that, The method described in claim 1 takes the driver domain features and the environment domain features obtained by the method, inputs them into the downstream decoupled multi-task learning network, and then performs specific driving task prediction.

Images