Method for comprehensive detection of driving state of operating vehicle driver based on multi-modal fusion

Abstract

The application discloses a driving state comprehensive detection method for operating vehicles based on multi-modal fusion, and particularly relates to the technical field of network traffic security detection, and the method comprises the following steps: collecting network traffic, extracting three kinds of modal data of encrypted handshake metadata, flow statistical features and message length time sequence, extracting deep features from three sub-networks at the edge after preprocessing; outputting three kinds of probabilities of safety, maliciousness and doubt through a pre-classifier, quickly judging based on the marginal difference of the probabilities, and uploading the cloud after applying differential privacy disturbance to the doubtful fuzzy samples; the cloud generates a single-step fusion vector through a cross-modal attention mechanism and a gate fusion, and obtains a joint representation through recurrent neural network time series modeling. The application has edge real-time performance and cloud deep analysis capability, effectively protects user privacy, and can simultaneously identify abnormal behaviors under known applications and unknown zero-day attacks.

Description

Technical Field

[0001] This invention relates to the field of network traffic security detection technology, and more specifically, to a comprehensive detection method for the driver status of operating vehicles based on multimodal fusion. Background Technology

[0002] Currently, encrypted traffic detection mainly relies on single-modal analysis or simple multimodal concatenation. While methods based on handshake metadata can identify TLS / QUIC protocol characteristics, they struggle to capture the temporal dynamic patterns of traffic. Methods based on statistical features are effective but susceptible to network fluctuations. Deep learning methods based on payload length sequences can model behavioral patterns but frequently misclassify due to neglecting the details of encryption negotiation. Existing publicly available solutions often simply concatenate these features before feeding them into a classifier, ignoring deep interactions between modalities. When encountering new encryption protocols or malicious traffic mimicking normal behavior, detection performance degrades drastically. Furthermore, edge computing power limits the ability to run complex fusion models, while directly uploading raw traffic or features risks user privacy leaks, creating a structural contradiction between fine-grained discrimination and privacy protection.

[0003] Another technological gap lies in the limitations of anomaly detection paradigms. Current systems largely rely on supervised training using known attack samples, offering virtually no awareness of unknown threats such as zero-day exploits and covert tunnels. Anomaly detection strategies based on fixed thresholds lack adaptability to different network environments and application types, making it difficult to balance false positives and false negatives. Meanwhile, although the cloud possesses powerful computing resources and a global perspective, the lack of targeted in-depth analysis mechanisms for suspicious edge samples results in a significant amount of computation being wasted on clearly identified traffic, rendering edge-cloud collaboration merely a formality and failing to achieve the expected efficiency of "rapid edge screening and refined cloud analysis."

[0004] To address the aforementioned shortcomings, this invention proposes a progressive multimodal fusion detection framework. A lightweight three-part network is deployed at the edge to achieve rapid pre-screening of traffic, with differential privacy protection applied only to suspicious samples before uploading to the cloud. The cloud leverages a cross-modal attention mechanism to uncover deep correlations between encrypted handshakes, statistical features, and time-series data, and introduces an anomaly detection branch based on prototype contrastive learning to achieve unified identification of anomalous behavior and unknown zero-day attacks under known application types. Simultaneously, a dynamically updatable prototype library and adaptive threshold mechanism enable the detection system to continuously evolve, adapting to changes in encryption protocols and attack methods. Summary of the Invention

[0005] To overcome the above-mentioned deficiencies of the prior art, embodiments of the present invention provide a comprehensive detection method for the state of drivers of operating vehicles based on multimodal fusion.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The comprehensive detection method for the driver status of commercial vehicles based on multimodal fusion includes the following steps: Collect network traffic data and simultaneously extract raw data in three modes from each flow. The three modes include encrypted handshake metadata, flow statistical features, and packet length time sequence. The raw data of the three modalities are preprocessed to obtain a modal representation with a unified dimension. The three preprocessed modal representations are input into three sub-networks inside the edge gateway, and the corresponding deep feature vectors are extracted respectively. The three deep feature vectors are concatenated and fed into the pre-classifier, which outputs the safe probability, malicious probability, and suspicious probability. The system makes a quick prediction based on the difference between the probability of safety and the probability of malice. If the difference is greater than the set marginal threshold, it is directly judged as either clearly safe or clearly malicious and the corresponding action is taken. Otherwise, it enters the fixed probability threshold judgment process. If the final determination is that the problem is questionable and ambiguous, differential privacy perturbation is applied to the three deep feature vectors before uploading them to the cloud. The cloud receives the noisy feature vector, generates a single-step fusion vector through cross-modal attention mechanism and gated fusion, and uses a recurrent neural network to perform temporal modeling on the fusion vector of the continuous sliding window to obtain a joint multimodal representation vector. Based on the joint multimodal representation vector, two branches, application identification and anomaly detection, are executed in parallel. The risk score is generated by fusing the outputs of the two branches. Risk levels are determined by risk scores, and corresponding control strategies are implemented for each level.

[0007] Specifically, the encrypted handshake metadata is obtained by parsing the ClientHello and ServerHello messages of the TLS handshake protocol, including a list of cipher suites, SNI domain name, certificate chain hash value, and signature algorithm field. For QUIC traffic, the initial version number and connection parameters are extracted. If a flow does not have a complete handshake process, the corresponding bits are filled with zeros. The flow statistics include flow duration, total number of bidirectional packets, total number of bytes, uplink / downlink packet ratio, and packet arrival interval statistics; the message length time sequence is taken as the payload length of the first few data packets of the flow, and zeros are added if it is insufficient.

[0008] Specifically, the preprocessing includes: Z-score normalization is performed on the original vectors of each modality; principal component analysis is used to reduce the dimensionality of the handshake metadata and flow statistics features to 16 dimensions; sinusoidal position coding is added to the message length time sequence to preserve the order information of the packets, resulting in a sequence carrying position information.

[0009] Specifically, the three sub-networks include: The three sub-networks consist of a three-layer multilayer perceptron for processing handshake metadata, a one-dimensional convolutional network for processing streaming statistical features, and a long short-term memory network for processing message length time-series sequences. Each sub-network outputs a 16-dimensional deep feature vector. The pre-classifier is a three-classifier consisting of a fully connected layer and a Softmax layer.

[0010] Specifically, the fixed probability threshold decision process is as follows: If the security probability is greater than the first set threshold, it is determined to be clearly secure; if the malicious probability is greater than the second set threshold, it is determined to be clearly malicious; if neither of the above two conditions is met, it is classified as questionable and ambiguous. The differential privacy perturbation uses a Gaussian mechanism, adding normally distributed noise to the three feature vectors according to the preset privacy budget, sensitivity, and noise standard deviation.

[0011] Specifically, the cross-modal attention mechanism is as follows: For each modality's feature vector, the query, key, and value vectors are obtained through linear projection. The attention score between any two modalities is calculated and normalized to obtain the weight. The value vectors of each modality are weighted and summed to obtain the interaction enhancement representation. The gated fusion is achieved by using a learnable gated vector to perform a weighted summation of the interaction enhancement representations of the three modalities, resulting in a single-step fusion vector; The temporal modeling employs a gated recurrent unit network with residual connections, which sequentially inputs the single-step fusion vectors of three consecutive sliding windows and outputs a joint multimodal representation vector.

[0012] Specifically, the application identifies the probability distribution of the application category to which the branch output traffic belongs; The anomaly detection branch is based on prototype contrastive learning. It calculates the cosine similarity between the joint multimodal representation vector and multiple pre-stored normal behavior prototype vectors, takes the maximum similarity, and if the maximum similarity is lower than the adaptive threshold, it is determined to be an anomaly, and an anomaly confidence score is calculated. The risk score is calculated by weighting the anomaly confidence level and the maximum probability value of the application identification branch.

[0013] Specifically, the risk levels are divided into three levels: low risk, medium risk, and high risk, which correspond to allowing normal communication and recording application tags, triggering secondary verification or limiting bandwidth, and automatically sending blocking policies to edge nodes, respectively.

[0014] Specifically, this also includes a phased training strategy: The first phase uses labeled traffic data to perform supervised training on the edge encoder and pre-classifier; The second stage freezes the edge encoder parameters, uses questionable and ambiguous samples received from the cloud and their real annotations to train the attention fusion network and application recognition branch, and updates the prototype library and adaptive threshold. The prototype library is updated regularly. During the update, joint representations are collected from samples that have recently been judged to be low-risk and have high application recognition confidence. New prototypes are obtained by re-clustering and cosine similarity matching is performed between the new prototype and the old prototype. If the similarity between the new prototype and any old prototype is lower than a set threshold, it can only be activated after manual sampling confirmation.

[0015] The technical effects and advantages of this invention are as follows: First, the edge employs a three-tiered rapid screening mechanism of "marginal prediction + dual thresholds," uploading only suspicious samples with ambiguous probabilities to the cloud. This ensures that the vast majority of clear traffic is processed locally, effectively reducing edge-cloud communication overhead and cloud computing load. Simultaneously, a Gaussian difference privacy perturbation is applied to the uploaded intermediate features, mathematically guaranteeing that even if attackers intercept the transmission data, they cannot deduce the original traffic content, thus establishing a robust privacy defense before achieving refined analysis. This three-step design enables the lightweight edge network to handle real-time detection while preserving necessary feature information for in-depth cloud analysis, avoiding the dilemma of traditional solutions that either infringe on privacy by uploading all data or suffer from missed detections due to insufficient local capabilities.

[0016] Second, the cloud-based system automatically captures the nonlinear interactions between encrypted handshakes, statistical attributes, and length time series through a cross-modal attention mechanism, making the fused representation more discriminative of spoofed traffic. Combined with an anomaly detection branch built using prototype contrastive learning, it can identify unknown threats deviating from normal behavior patterns without malicious samples, effectively filling the blind spot of supervised models in detecting zero-day attacks. Furthermore, the prototype library and judgment thresholds are updated periodically with new samples, and new prototypes must undergo similarity verification before activation, ensuring the system's ability to follow network evolution while preventing anomalous data from contaminating the reference benchmark. Finally, the risk scores output by the dual branches can be mapped to a three-level control strategy, making the entire framework superior to existing single-modal or simple splicing schemes in terms of accuracy, robustness, and maintainability. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figure 1 As shown, the steps of the comprehensive detection method for the driver status of operating vehicles based on multimodal fusion are as follows: Step 1: Deploy bypass packet capture probes at internet communication egress points or edge routing nodes to acquire bidirectional traffic in a non-intrusive manner. For each IP flow with a unique five-tuple, define a flow timeout of 15 seconds to segment the complete flow accordingly. For each stream, raw data in three modalities is extracted synchronously: The first modal is encrypted handshake metadata, which parses 48 discrete and continuous fields from the ClientHello and ServerHello messages of TLS 1.2 / 1.3, including the cipher suite list, SNI domain name, certificate chain SHA-256 hash, signature algorithm, etc. The initial version number and connection parameters are extracted from QUIC traffic, and if a stream does not have a complete handshake process, the corresponding bits are filled with zeros; The second modal is stream statistical features, which calculates 30 statistical quantities such as stream duration, total number of packets in both directions, total number of bytes, uplink-to-downlink packet ratio, mean, variance, minimum and maximum packet arrival interval, etc., to form a 30-dimensional feature vector; The third modal is message length temporal behavior, which extracts the TCP / UDP payload length (ciphertext length) of the first 40 packets of a stream, and pads the tail with zeros if it is less than 40, forming a fixed 40-dimensional sequence. Next, multimodal spatiotemporal alignment is performed: A 5-second sliding time window is used to extract the modalities of each stream within the window. If a stream spans multiple time windows, it is divided into multiple independent samples according to the window. Within each window, the three original modal vectors of all samples are Z-score normalized. The mean and standard deviation are calculated from the pre-collected offline dataset and embedded into the probe. The normalized message length sequence is denoted as... To eliminate redundancy and unify dimensionality, Principal Component Analysis (PCA) was used to reduce the dimensionality of the handshake metadata to 16 dimensions, and the flow statistical features were also reduced to 16 dimensions. For the message length temporal modality, sinusoidal position coding was added to inject packet sequence information. Specifically, for the ... Locations ( Generate a 40-dimensional position encoding vector. , its first dimensional components are , No. dimensional components are , From 0 to 19, we get Position encoding matrix Pick The diagonal elements form the position enhancement vector. Element-wise and normalized sequence Add them together to get a sequence carrying location information. This sequence, serving as the final representation of the message length temporal modality, retains a 40-dimensional temporal structure for further encoding by subsequent networks. At this point, the handshake metadata and flow statistical features of each flow sample have been represented as a 16-dimensional lightweight embedding vector, while the message length modality is a 40-dimensional position-enhanced sequence. .

[0020] Step 2: A feature encoder consisting of three lightweight sub-networks is deployed within the edge gateway. The handshake metadata embedding vector is fed into a three-layer multilayer perceptron (MLP) with 64, 32, and 16 neurons per layer, using ReLU activation, outputting 16-dimensional deep features. The stream statistics embedding vector is fed into a one-dimensional convolutional network (1D-CNN), containing two convolutional layers with kernel size 3 and 32 and 16 channels respectively, followed by global average pooling, outputting 16-dimensional features. The location-enhanced message length sequence... The data is fed into a single-layer LSTM with 32 hidden units. The hidden state at the last time step is then mapped to 16 dimensions via a fully connected layer to obtain the deep temporal features of the message length. After obtaining three 16-dimensional modality feature vectors, they are concatenated into a 48-dimensional vector and fed into the gate control pre-classifier. This classifier consists of a fully connected layer... Composed of Softmax, it outputs three types of probabilities: safe probability. Probability of malice Probability of doubt During the training phase, a large number of labeled encrypted traffic samples are used to optimize the entire encoder and classifier with pre-classified cross-entropy loss.

[0021] During inference, the fast prediction module based on the marginal difference of probability is activated first. Definition And set marginal thresholds .when In this case, the category with the higher probability is directly selected as the final judgment: if If the flow is marked as "explicitly safe", it is allowed and logged; if If it is detected as "clearly malicious," it will be immediately blocked and an alert will be issued. Only if this boundary condition is not met (i.e., Only traffic exceeding a certain threshold will proceed to the subsequent fixed probability threshold decision process: if If deemed "explicitly secure," communication is permitted but only log summaries are reported; if If a request is deemed "clearly malicious," the edge device will directly block the request and issue an alert; all other cases will be classified as "questionable and ambiguous." Only when a request is deemed "questionable and ambiguous" will the edge device perform differential privacy perturbation on the 16-dimensional embedding vector of the handshake metadata and the 16-dimensional embedding vector of the flow statistics obtained in step 1, as well as the 16-dimensional feature vector output from the message length temporal modality after LSTM encoding in step 2. This perturbation employs a Gaussian mechanism and sets a privacy budget. , Sensitivity The estimate is 0.1. Calculate the noise standard deviation. Add to the three vectors respectively (This indicates that the mean is 0 and the variance is 0) After adding noise (to the normally distributed data), the noisy vector is encapsulated and uploaded to the cloud. This mechanism ensures that even if the intermediate features are intercepted, the original traffic content cannot be deduced, thus strictly protecting user privacy.

[0022] Step 3: The cloud receives three 16-dimensional embedding vectors (with added noise) uploaded from the edge and initiates the feature fusion network. First, it maps the 16-dimensional vector of each modality to a 64-dimensional query vector using three independent linear projection layers. Key vector Sum value vector ( (Representing three modalities). Cross-modal attention is calculated as follows: for any two modalities... and Calculate attention score (scaling factor) ), and then for each ,exist Attention weights are obtained by performing Softmax on the dimension. Then the mode The output after fusing all modal information is This yields interactive enhancement representations for three modalities. , , Then, a learnable modal gating vector is used. (Three-dimensional, Softmax normalized) We perform a weighted summation on them to obtain the single-step fusion vector. Considering the time evolution characteristics of traffic, t represents the sequence number of the current sliding window, and three consecutive sliding windows ( ) generated A GRU network with residual connections is fed in sequentially: The GRU hidden layer dimension is 128, and the final value is... As the 128-dimensional joint multimodal representation vector of this flow This network can automatically learn which modalities are more important for the current judgment and capture the dynamic changes in encrypted communication behavior through temporal modeling. In actual deployment, the edge probe maintains a cache queue for each active flow. Regardless of the judgment result of the current window, the basic features required for single-step fusion of the three most recent windows are temporarily cached. When a flow is continuously judged as "suspicious and ambiguous," the uploaded content includes the complete trimodal feature vectors of the current window and the two previous windows; when a window is directly judged as "clearly secure" or "clearly malicious" at the edge but does not trigger an upload, the cloud inputs the GRU of that window. Set it to the zero vector, and keep the hidden state of the previous time step undecayed under residual connection, thereby maintaining the coherence of time series modeling.

[0023] Step 4: get Then, two discriminative branches are started in parallel. Branch one is the application identification branch, which is initiated by the fully connected layer. constitute, Given a preset number of application categories (e.g., web pages, video, instant messaging, and file transfer, Softmax outputs the application probability distribution). The loss function is the classification cross-entropy. Branch two is the anomaly detection branch based on prototype contrastive learning: in the offline phase, it collects joint representations of a large amount of known normal encrypted traffic. The K-means algorithm is used. Clustering yielded 20 normal behavior prototype vectors, which are stored in the cloud. During online inference, the current... The cosine similarity with each prototype is taken as the maximum value. Set an adaptive threshold. : Calculate the scores of all samples on the normal validation set. The 5th percentile is used as the threshold to ensure that the false alarm rate for normal samples does not exceed 5%. If so, it is judged as abnormal, and the abnormality confidence level is calculated. Finally, the outputs of the two branches are combined to generate a risk score: This value is mapped to the range 0-100; where The system identifies the maximum probability value of the branch output for each application; traffic is divided into three levels based on risk score: 0-30 is low risk, allowing normal communication and recording application tags; 31-70 is medium risk, triggering secondary verification or bandwidth restriction; 71-100 is high risk, automatically issuing blocking policies to edge nodes. This progressive dual-branch discrimination can identify anomalies in known application types as well as discover completely unknown zero-day encryption attacks.

[0024] To ensure the entire framework functions collaboratively, a phased training strategy is adopted. In the first phase, millions of traffic data points labeled as safe, malicious, and suspicious are used for supervised training of the edge encoder and pre-classifier. The loss function is the pre-classification cross-entropy, enabling the edge to acquire preliminary trichotomy capabilities. In the second phase, the edge encoder parameters are frozen. The cloud uses "suspicious / ambiguous" samples uploaded from the edge, along with real application labels and anomaly annotations, to train the attention fusion network and application recognition branch, and update the prototype library and thresholds. The anomaly detection branch relies solely on the joint representation of normal samples for clustering, without the involvement of malicious samples. After online deployment, the system can periodically or based on feedback to update the thresholds and prototypes to adapt to changes in network protocols and traffic. The prototype library is updated every 7 days, and updates are performed on samples deemed low-risk and with high application recognition confidence within the past 7 days by the cloud. Joint characterization was collected from flow samples with a maximum value greater than 0.8. A candidate normal sample set is formed. K-means clustering (K=20) is then performed again on this candidate set to obtain new prototype vectors. Cosine similarity matching is performed between the new and old prototypes. If the similarity between the new prototype and any old prototype is less than 0.7, it is marked as a prototype awaiting review and will not be added to the online service until it is confirmed by manual sampling. This prevents abnormal samples from contaminating the prototype library. The threshold τ is synchronously recalculated and updated based on the 5th percentile of the new prototype set on the validation set.

[0025] The above formulas are all dimensionless calculations. Dimensionless calculations can be performed using various methods such as standardization, which will not be elaborated here. The formulas are derived from software simulations based on a large amount of collected data, and the preset parameters in the formulas can be set by those skilled in the art according to the actual situation.

[0026] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, ATA hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state ATA hard disk.

[0027] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0028] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0029] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0030] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0031] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0032] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable ATA hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0033] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A comprehensive detection method for the driver status of operating vehicles based on multimodal fusion, characterized in that, Includes the following steps: Collect network traffic data and simultaneously extract raw data in three modes from each flow. The three modes include encrypted handshake metadata, flow statistical features, and packet length time sequence. The raw data of the three modalities are preprocessed to obtain a modal representation with a unified dimension. The three preprocessed modal representations are input into three sub-networks inside the edge gateway, and the corresponding deep feature vectors are extracted respectively. The three deep feature vectors are concatenated and fed into the pre-classifier, which outputs the safe probability, malicious probability, and suspicious probability. The system makes a quick prediction based on the difference between the probability of safety and the probability of malice. If the difference is greater than the set marginal threshold, it is directly judged as either clearly safe or clearly malicious and the corresponding action is taken. Otherwise, it enters the fixed probability threshold judgment process. If the final determination is that the problem is questionable and ambiguous, differential privacy perturbation is applied to the three deep feature vectors before uploading them to the cloud. The cloud receives the noisy feature vector, generates a single-step fusion vector through cross-modal attention mechanism and gated fusion, and uses a recurrent neural network to perform temporal modeling on the fusion vector of the continuous sliding window to obtain a joint multimodal representation vector. Based on the joint multimodal representation vector, two branches, application identification and anomaly detection, are executed in parallel. The risk score is generated by fusing the outputs of the two branches. Risk levels are determined by risk scores, and corresponding control strategies are implemented for each level.

2. The method for comprehensive detection of driver status of operating vehicles based on multimodal fusion according to claim 1, characterized in that, The encrypted handshake metadata is obtained by parsing the ClientHello and ServerHello messages of the TLS handshake protocol, including a list of cipher suites, SNI domain name, certificate chain hash value and signature algorithm field. For QUIC traffic, the initial version number and connection parameters are extracted. If a stream does not have a complete handshake process, the corresponding bit is filled with zero; The flow statistics include flow duration, total number of bidirectional packets, total number of bytes, uplink / downlink packet ratio, and packet arrival interval statistics; the message length time sequence is taken as the payload length of the first few data packets of the flow, and zeros are added if it is insufficient.

3. The method for comprehensive detection of driver status of operating vehicles based on multimodal fusion according to claim 1, characterized in that, The preprocessing includes: Z-score normalization is performed on the original vectors of each modality; principal component analysis is used to reduce the dimensionality of the handshake metadata and flow statistics features to 16 dimensions; sinusoidal position coding is added to the message length time sequence to preserve the order information of the packets, resulting in a sequence carrying position information.

4. The method for comprehensive detection of driver status of operating vehicles based on multimodal fusion according to claim 1, characterized in that, The three sub-networks include: The three sub-networks consist of a three-layer multilayer perceptron for processing handshake metadata, a one-dimensional convolutional network for processing streaming statistical features, and a long short-term memory network for processing message length time-series sequences. Each sub-network outputs a 16-dimensional deep feature vector. The pre-classifier is a three-classifier consisting of a fully connected layer and a Softmax layer.

5. The method for comprehensive detection of driver status of operating vehicles based on multimodal fusion according to claim 1, characterized in that, The fixed probability threshold decision process is as follows: If the security probability is greater than the first set threshold, it is determined to be clearly secure; if the malicious probability is greater than the second set threshold, it is determined to be clearly malicious; if neither of the above two conditions is met, it is classified as questionable and ambiguous. The differential privacy perturbation uses a Gaussian mechanism, adding normally distributed noise to the three feature vectors according to the preset privacy budget, sensitivity, and noise standard deviation.

6. The method for comprehensive detection of driver status of operating vehicles based on multimodal fusion according to claim 1, characterized in that, The cross-modal attention mechanism is as follows: For each modality's feature vector, the query, key, and value vectors are obtained through linear projection. The attention score between any two modalities is calculated and normalized to obtain the weight. The value vectors of each modality are weighted and summed to obtain the interaction enhancement representation. The gated fusion is achieved by using a learnable gated vector to perform a weighted summation of the interaction enhancement representations of the three modalities, resulting in a single-step fusion vector; The temporal modeling employs a gated recurrent unit network with residual connections, which sequentially inputs the single-step fusion vectors of three consecutive sliding windows and outputs a joint multimodal representation vector.

7. The method for comprehensive detection of driver status of operating vehicles based on multimodal fusion according to claim 1, characterized in that, The probability distribution of the application category to which the output traffic of the identified branch belongs; The anomaly detection branch is based on prototype contrastive learning. It calculates the cosine similarity between the joint multimodal representation vector and multiple pre-stored normal behavior prototype vectors, takes the maximum similarity, and if the maximum similarity is lower than the adaptive threshold, it is determined to be an anomaly, and an anomaly confidence score is calculated. The risk score is calculated by weighting the anomaly confidence level and the maximum probability value of the application identification branch.

8. The method for comprehensive detection of driver status of operating vehicles based on multimodal fusion according to claim 1, characterized in that, The risk levels are divided into three levels: low risk, medium risk, and high risk, which correspond to allowing normal communication and recording application tags, triggering secondary verification or limiting bandwidth, and automatically sending blocking policies to edge nodes, respectively.

9. The method for comprehensive detection of driver status of operating vehicles based on multimodal fusion according to claim 1, characterized in that, It also includes a phased training strategy: The first phase uses labeled traffic data to perform supervised training on the edge encoder and pre-classifier; The second stage freezes the edge encoder parameters, uses questionable and ambiguous samples received from the cloud and their real annotations to train the attention fusion network and application recognition branch, and updates the prototype library and adaptive threshold. The prototype library is updated regularly. During the update, joint representations are collected from samples that have recently been judged to be low-risk and have high application recognition confidence. New prototypes are obtained by re-clustering and cosine similarity matching is performed between the new prototype and the old prototype. If the similarity between the new prototype and any old prototype is lower than a set threshold, it can only be activated after manual sampling confirmation.

Images