Driver abnormity monitoring system based on large model collaborative decision

Abstract

The invention discloses a driver abnormity monitoring system based on large model collaborative decision-making. The system comprises a multi-path camera, a voice interaction module, a DTOPA visual pre-alignment module, a Qwen-Apan lightweight language reasoning module, a collaborative decision-making module and a safety response control module. Wherein the multi-path camera is used for extracting an environment video stream in a vehicle in real time; the voice interaction module is used for reminding a driver and interacting with the driver; the TOPA visual pre-alignment module is used for converting a video stream into structured text description; the Qwen-Apan lightweight language reasoning module is used for performing semantic analysis and abnormal risk assessment on the text description; the collaborative decision module is used for triggering a voice interaction and vehicle control instruction based on the risk assessment result; the safety response control module is used for executing side parking, emergency contact person notification or multi-level early warning operation; according to the invention, the safety of aided driving is obviously improved.

Description

Technical Field

[0001] The present invention relates to the technical field of assisted driving safety processing, and in particular to a driver abnormality monitoring system based on large-model collaborative decision-making. Background Art

[0002] Traditional driver monitoring systems mostly rely on simple physiological indicators for fatigue detection, lacking the ability to understand complex scenarios, particularly those involving assisted driving. Current mainstream large language models often focus on either perception or reasoning. Perception tasks focus on precise visual feature extraction and scene recognition, while reasoning tasks require logical analysis and decision-making based on understanding. These two approaches have fundamentally different optimization goals and evaluation criteria, and both lack true intelligent analytical capabilities. Existing DMS systems offer relatively simple recognition results and can only provide simple beep or vibration warnings. Furthermore, they lack reliable coordination with the latest generation of electronic vehicle systems, preventing effective data exchange and functional linkage with these systems. Summary of the Invention

[0003] Purpose of the invention: The purpose of the present invention is to provide a driver abnormality monitoring system based on large-scale model collaborative decision-making to solve the problems of the existing driver monitoring system such as single perception dimension, lack of scene understanding and reasoning capabilities, and weak response mechanism.

[0004] Technical solution: The driver abnormality monitoring system based on large-model collaborative decision-making described in the present invention includes: multi-channel cameras, a voice interaction module, a DTOPA visual pre-alignment module, a Qwen-Apan lightweight language reasoning module, a collaborative decision-making module, and a safety response control module; wherein, the multi-channel cameras are used to extract the in-vehicle environment video stream in real time; the voice interaction module is used to remind the driver and interact with the driver, and finally feed back the results to the entire system as part of the decision-making; the DTOPA visual pre-alignment module is used to convert the video stream into a structured text description; the Qwen-Apan lightweight language reasoning module is used to perform semantic analysis and abnormal risk assessment on the text description; the collaborative decision-making module is used to trigger voice interaction and vehicle control instructions based on the risk assessment results; the safety response control module is used to perform pull-over parking, emergency contact notification or multi-level warning operations.

[0005] Furthermore, the DTOPA visual pre-alignment module is specifically designed as follows: a multimodal data augmentation strategy is adopted to generate a driving text video (DTV) dataset to simulate the temporal characteristics of real videos; the CLIP-ViT-L model is used to align image and text features; CLIP dependency is eliminated through visual capability internalization technology, and multi-scale convolutional embedding and modality-adaptive normalization are used.

[0006] Furthermore, the multimodal data enhancement strategy is as follows: LLM is used to automatically generate multimodal pre-training data to create a large-scale driving text video DTV dataset containing text videos and their associated annotations.

[0007] Furthermore, the Qwen-Apan lightweight language reasoning module is as follows: the GPTQ-Int8 quantization scheme is used to compress the model weights; the KV cache is dynamically allocated based on attention entropy.

[0008] Furthermore, the collaborative decision-making module is specifically as follows: defining the driver status understanding index DSU; triggering three levels of response based on the abnormality level: voice reminder, emergency contact notification, and assisted side parking; the DSU formula is as follows: ; Among them, w risk is the risk weighted function; groundtruth i is the basic facts of the i-th sample, including facts such as the in-vehicle scene, driver status, and risk level information. N is the total number of test samples (a positive integer). α is the description accuracy weight coefficient, which controls the importance of description accuracy in the overall evaluation. β is the state classification accuracy weight coefficient, which controls the importance of classification accuracy in the overall evaluation. i is the sample index from 1 to N. represents the accuracy of driver status description, It represents the classification accuracy of driver status, and its calculation formula is as follows: ; ; in, represents the i-th driver state description generated by DTOPA, represents the standard description of the dataset annotation; ROUGE-L represents the longest common subsequence ratio, Indicates whether the corresponding label is mentioned in the j-th driver state description; represents the true driver state label, 、 Both represent the total number of test samples; j is the classification sample index from 1 to M.

[0009] Furthermore, by simulating the diverse behavioral patterns of the Lianfeng Horned Toad, the AdamH optimizer was designed to achieve parameter optimization of the multi-strategy collaborative module.

[0010] Furthermore, the CH chaos map initialization parameters are introduced to integrate the behavior patterns of the Lianfeng horned toad, namely foraging, defense, escape, and social update gradient directions. The CH chaos map initialization parameter formula is as follows: ; Where α≥0, β≥0 and α+β≤1, u∈[0,4], γ ∈[2.5,3.0], are mapping control parameters, z i is the current chaotic state value, z i+1 is the next state value output by the CH chaotic map, and satisfies z i+1 ∈[0,1]; w i is the intermediate weight value of the i-th iteration used to generate the chaotic sequence; λ is the interpolation parameter, which controls the balance between deterministic and random components; r is the random factor, which introduces random perturbations to enhance ergodicity; π is pi, which is approximately equal to 3.14159; sin is the sine function, cos is the cosine function, and || is the absolute value function, which ensures that the output is non-negative.

[0011] The present invention provides a method for monitoring driver abnormality, which is implemented based on a driver abnormality monitoring system using a large model collaborative decision-making system, and includes the following steps: (1) Capture in-car video streams at 25 FPS through multiple cameras; (2) Generate scene text descriptions in real time using the DTOPA model; (3) Use the Qwen-Apan model to parse the text and output a risk assessment; (4) If the detection is abnormal, start the voice interaction confirmation state; (5) Perform security response operations based on the interaction results.

[0012] Furthermore, in step (1), a multimodal data enhancement strategy is adopted to generate a driving text video DTV dataset.

[0013] Furthermore, in step (2), a 21-second sliding window mechanism is used to randomly select 20 frames to form an analysis batch, and the compressed frame data is transmitted through WebSocket.

[0014] Beneficial Effects: Compared with the existing technology, the present invention has the following significant advantages: (1) It achieves a technological leap from traditional "feature monitoring" to in-depth "scene understanding", accurately identifying complex driving behaviors such as fatigue, distraction, and abnormal posture. (2) It builds a lightweight large-scale model system that can run efficiently on on-board edge computing devices, eliminating cloud dependence and ensuring low-latency response and user data privacy. (3) It establishes an intelligent closed loop of "perception-understanding decision-interaction-execution". When risks are detected, it can confirm the status through voice interaction and perform safety interventions such as pulling over by linking the vehicle assistance system, significantly improving the safety of assisted driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of the present invention; Figure 2 Schematic diagram of the visual driving pre-alignment module of the present invention. DETAILED DESCRIPTION

[0016] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0017] like Figure 1 As shown, an embodiment of the present invention provides a driver abnormality monitoring system based on large-scale model collaborative decision-making. Multiple cameras continuously collect in-vehicle video streams and transmit the DTOPA visual pre-alignment module, which is responsible for "translating" the video frames into structured text descriptions. Subsequently, the Qwen-Apan lightweight language inference module performs semantic analysis on the text to determine whether there is an abnormality. Once an abnormality is detected, the system will trigger the human-computer interaction module to communicate with the driver by voice, and the safety response control module will decide whether to take measures such as alarm, notification of emergency contacts or auxiliary vehicle pullover according to the response results; it includes multi-channel cameras, voice interaction module, DTOPA visual pre-alignment module, Qwen-Apan lightweight language reasoning module, collaborative decision-making module, and safety response control module; among them, multi-channel cameras are used to extract the video stream of the in-vehicle environment in real time; the voice interaction module is used to remind the driver and interact with the driver; the DTOPA visual pre-alignment module is used to convert the video stream into a structured text description; the Qwen-Apan lightweight language reasoning module is used to perform semantic analysis and abnormal risk assessment on the text description; the collaborative decision-making module is used to trigger voice interaction and vehicle control instructions based on the risk assessment results; the safety response control module is used to execute pullover, emergency contact notification or multi-level warning operations.

[0018] Among them, the DTOPA visual pre-alignment module is as follows: First, a text pre-alignment framework is employed. Using a multimodal data augmentation strategy, serialized text descriptions and their associated annotations simulate real-world video-text pairs, enabling a preliminary pre-alignment of the text-only LLM with the video modality at the conceptual level. To bridge the representational gap between synthesized text videos and real-world videos, the CLIP model is used as a feature extractor during initial pre-training to effectively fuse image and text features, thereby aligning the video image and text modalities. Then, through internalization of visual capabilities, the CLIP model is removed. This preserves the alignment functionality between the video image and text modalities while significantly reducing the performance requirements placed on the end-side platform during deployment. Furthermore, three new pre-alignment tasks are trained using a unified autoregressive language modeling objective to achieve pre-alignment between text videos and LLM. The introduction of a support memory mechanism bridges the gap between video image and text modalities through a training-free projection process.

[0019] Among them, Figure 2As shown in the figure, the multimodal data augmentation strategy uses LLM to automatically generate multimodal pre-training data to create a large-scale Driver-oriented Text Video (DTV) dataset containing text videos and their associated annotations, thereby addressing the data scarcity problem in visual understanding tasks. Specifically, LLM automatically generates text videos consisting of consecutive text frames. Each text video contains 100-200 sequential text frames, each frame including a frame description and multiple object descriptions. This simulates the temporal characteristics of real videos and ensures that the generated text videos capture dynamic changes and event development processes. To ensure the diversity and domain coverage of the DTV dataset, different conditional prompts are used as generation guides, including video titles and video descriptions from driver anomaly datasets such as Driver Anomaly Detection and Driver Behaviors. This multi-source conditional prompt strategy ensures the breadth and practicality of the generated data.

[0020] CLIP model: To bridge the gap between text videos and real videos, CLIP-ViT-L is adopted as the feature extractor to align image and text modalities during initial pre-training.

[0021] For text videos, given a text frame , CLIP text encoder from frame description and detailed object description Extract the frame representation from the equation as follows: ; (1) in, Represents the SWAF (Scene-Aware Weighted Fusion, SAWF) function, represents the CLIP text encoder. Therefore, a text video containing n text frames is represented as ; Among them, w c and w d It is the preset global-local information balance hyperparameter responsible for dynamic attention weights.

[0022] For a real video, given an image frame , CLIP image encoder extracts frame-level visual features , the formula is as follows: ; (2) in, It is projected into the LLM space through the linear layer and expressed as ,in represents the language model, Represents the projection layer that projects CLIP features into LLM space, PreProcess is an image preprocessing function, S env Represents the state of the environment (such as whether the image needs denoising, whether the lighting is balanced, etc.); Pre-alignment Tasks: The DTOPA model introduces three tasks to pre-align text and video with LLM. The text and video summarization task generates a detailed description based on a given text and video to summarize the entire text and video content. The text and video question-answering task predicts the answer based on a given text and video and a question. The multiple-choice text and video question-answering task is responsible for selecting the correct answer from a list of candidate answers given a text and video, a question, and multiple options. The model uses an autoregressive language modeling objective to train these three tasks, as shown below: ; (3) in, is the total loss function of the DTOPA model, N is the sequence length, and λ is the regularization hyperparameter used to balance the loss. represents the i-th target annotation, log is the logarithmic function, p represents the probability that the large model can accurately predict the next correct word at each step when generating the target text based on the input, t <i represents all t before the i-th one; R is the task-oriented regularization term. When the generated text V' lacks description of key behaviors, the value of this term will increase.

[0023] Since CLIP model pre-training achieves image-text feature alignment, this aligned representation makes zero-shot inference possible without additional fine-tuning. However, the modality gap phenomenon (i.e., CLIP image features and CLIP text features are located in completely different regions of the feature space) prevents the direct use of visual features. As text features To bridge this modality gap, the DTOPA model uses a support memory mechanism to project CLIP visual features into CLIP text feature space. This untrained projection process is expressed as: ; (4) in, is the final feature projected from the visual feature to the text space, represents the pre-built CLIP text feature in the memory of size N, sim represents the similarity calculation function between vectors, and τ is the temperature coefficient used to control the smoothness of the attention distribution. is the original visual feature. In zero-shot reasoning, we use as a representation of real video.

[0024] Using visual capability internalization techniques, the CLIP module's functionality is internalized into DTOPA, allowing the module to be removed. Its core goal is to integrate visual perception directly into the language model while retaining the video understanding capabilities learned in the first stage. This stage encodes visual information by introducing low-rank adapters in the first few layers of the LLM. Specifically, for the first Nvit layers of the LLM, DTOPA is applied to all linear layers (including QKV projection and feedforward layers): ; (5) Where W represents the pre-trained weight matrix, ΔW=BA represents the low-rank adaptation update, and B∈R d×r and A∈R r×d are low-rank decomposition matrices, respectively. h is the output of the final linear layer, x is the input, d is the original dimension, and r is the low-rank dimension. After training, the parameters can be seamlessly merged into the trained base LLM from steps 1) to 4, achieving zero additional computational overhead during inference.

[0025] ; (6) Among them, W merged is the combined weight matrix, which has zero additional computational overhead during inference.

[0026] To replace the external CLIP visual encoder, DTOPA designed a direct visual embedding module. It first divides the video frame into patches through a convolutional layer and proposes a "multi-scale convolutional embedding and modality-adaptive normalization" scheme. The convolution is no longer single-scale, but parallel multi-scale convolution to capture visual features of different granularities: ; (7) Where I represents the input video frame image and p represents the patch size. Position encoding is then added and projected into the LLM space. patches is the patch feature after multi-scale convolution, Concat is the feature splicing operation; the feature map after the convolution operation is flattened and transposed to obtain the sequence representation, and then RMS normalization is applied to stabilize the training process. The x in the following formula is norm is the normalized sequence feature, RMSNorm represents RMS normalization, x seq The features after serialization are flattened and serialized from two-dimensional image patches into one-dimensional sequence feature representations: ; (8) The calculation formula for RMS normalization is: ; (9) Where d is the feature dimension, x iis the i-th dimension feature, ε is a numerical stability constant (can be set to 1e-8), and γ is a learnable scaling parameter.

[0027] The final visual embedding is obtained through a linear projection, which maps the features of the intermediate dimensions to the hidden dimensions of DTOPA, ensuring that the visual representation can be processed in the same space as the textual representation. This design enables visual information to be processed directly in a form that DTOPA can understand, without the need for external intermediaries. Compared to other visual models such as LLAVA, which have ViT + high training costs + traditional architecture, this design allows the added parameters to be seamlessly incorporated into the LLM during inference, thereby eliminating structural complexity and minimizing training and computational overhead. The inference speed per token is over 1.6 times faster, and memory usage is reduced by 15%, making DTOPA more suitable for deployment on the end side with limited computing power.

[0028] The Qwen-Apan lightweight language inference module is based on the Qwen3-1.7B model and adopts the GPTQ-Int8 post-training quantization scheme to compress the model weights into 8-bit integers, significantly reducing the graphics memory / memory usage, enabling it to be stably deployed on low-power platforms such as Nvidia Jetson Orin NX, shortening the inference time and making it more suitable for abnormal driver status detection tasks.

[0029] Although the Qwen3-1.7B model is lightweight compared to larger models, when deployed on embedded devices, it still faces obstacles such as high inference latency, long startup time, and easy memory overflow. Therefore, other technologies need to be introduced to compress the model size, reduce computing requirements, and improve inference efficiency.

[0030] Furthermore, to further optimize the memory usage efficiency of the KV cache, this paper introduces a KV cache allocation strategy based on the attention entropy between the decision reasoning text and the video description generated by DTOPA. Traditional KV cache management methods use a uniform cache size for all Transformer layers, ignoring the differences in attention density between different layers when processing different information. In the application of abnormal driver status monitoring, different attention layers have significant differences in the degree of attention they pay to different information.

[0031] The KV cache allocation strategy uses attention entropy to quantify the attention distribution characteristics of each layer, thereby using a dynamic method to determine the optimal KV cache allocation scheme. Specifically, the attention entropy calculation formula is as follows: ; (10) in, is the cross-modal attention entropy of the lth layer; represents the probability distribution of attention of m1 to m2 in the i-th attention head of layer l, where m1 and m2 represent two different modalities; Nhead is the number of attention heads. Based on the attention entropy, the cache allocation strategy is: ; (11) Among them, K l is the KV cache size of the lth layer, K base is the base cache size, and are the mean and standard deviation of the attention entropy of all layers, η is the adjustment factor, Cm(l) is the contribution factor, which represents the contribution of module m to the decision at layer l; M represents the set of all modalities.

[0032] GPTQ is a gradient-based post-training quantization technique that significantly compresses model size and improves inference speed while minimizing accuracy loss. For each layer, the algorithm uses second-order information (Hessian matrix) to guide the quantization process, ensuring that quantization error is minimized.

[0033] Here, we use a variant of the DSQ algorithm that preserves key weights. During the quantization process, rather than treating all weights equally, we identify a small number of weights that are most critical to "judging dangerous driving behavior" and quantize them with higher precision. Specifically, we solve the following optimization problem: ; (12) in, represents the original weight, represents the quantized weight, Represents the input activations of this layer. Msaliency is a "weight saliency mask" of the same dimension as W, where the values ​​corresponding to weights critical to safety judgment are 1, and the rest are 0. This mask can be obtained by analyzing gradients or feature contributions. λ is a hyperparameter used to balance overall quantization error and the protection of critical weights. is the L2 norm, and argmin means finding the quantized weights that minimize the expression.

[0034] Although GPTQ technology is widely used in LLM deployment due to its excellent compression capabilities and fast inference advantages, it is still difficult to meet the operation and storage requirements in resource-constrained edge devices or vehicle-mounted platforms, and inference still relies on some high-performance GPUs or hardware platforms with floating-point computing capabilities, which limits its practical value in lightweight terminals. Therefore, the present invention introduces the Int8 integer quantization strategy within the GPTQ framework to form a GPTQ-Int8 solution with both spacing compression and high efficiency. By compressing the weights into 8-bit integer representations and retaining the high-precision expression of key structures, the stability of inference accuracy is maintained while significantly reducing the model size. Compared with the traditional FP16-GPTQ model, the model size of the Int8 version can be further compressed by more than 50%, significantly reducing memory, video memory and bandwidth requirements, making it more suitable for low-power environments. GPTQ-Int8 has better memory access and cache hit rates, and significantly reduces overall execution latency. While maintaining language understanding and semantic reasoning capabilities, it achieves the best balance between model compression, inference acceleration and deployment cost.

[0035] Collaborative decision-making module: As a core technology in the current field of artificial intelligence, the Large Language Model (LLM) can act as a cross-modal semantic bridge, converting visual features into structured text semantics, and effectively solving the gap between visual signals and semantic reasoning in video understanding. However, the current language annotation efficiency of video-text datasets collected from the Internet is low, and there are only simple label annotations, which are difficult to use for training. To this end, the present invention proposes a collaborative decision-making mechanism algorithm that integrates the dual models of visual-driving pre-alignment and lightweight language reasoning to achieve a technological leap from "feature monitoring" to "scene understanding."

[0036] Driver state understanding evaluation index DSU: The system proposes a new evaluation index DSU calculation formula as follows: ; (13) Among them, w risk is a risk-weighted function that maps the groundtruth to a risk weight, which enables DSU to evaluate not only accuracy but also risk-aware accuracy. represents the accuracy of driver status description, It represents the classification accuracy of driver status, and its calculation formula is as follows: ; (14) ; (15) in, represents the driver state description generated by DTOPA, represents the standard description of the dataset annotation, ROUGE-L represents the longest common subsequence ratio, Indicates whether the corresponding label is mentioned in the j-th driver state description, Represents the true driver status label, If is an indicator function that returns 1 when the condition is true and 0 when it is false, M and N both represent the total number of test samples, and j is the classification sample index from 1 to M.

[0037] Traditional Adam and AdamW optimizers adopt a monotonically decreasing learning rate adjustment strategy. Although this design helps to ensure convergence stability, it can easily cause the algorithm to fall into a local optimal solution prematurely in complex loss landscapes. To address these fundamental defects of traditional optimizers, this paper proposes the AdamH optimizer by simulating the diverse behavior patterns of the lotus-peaked toad, realizing a multi-strategy collaborative parameter optimization mechanism.

[0038] When applied to the training of large models, the first step is to map the parameter space of the neural network to the position space of the Lianfeng Horned Toad. Let the network parameter vector be , each individual represents a complete set of network parameter configurations, and each parameter is initialized by a cubic chaos map. For each training batch, the fitness of each individual (parameter configuration) is calculated: .

[0039] It is important to note that the common initialization methods currently used in the training of large models are often zero initialization, uniform distribution initialization, normal distribution initialization, etc. Numerous studies have shown that such initialization strategies can easily lead to overly concentrated distribution of individuals in the population, thereby weakening the algorithm's local development capabilities and global exploration performance.

[0040] To address these issues, chaotic mapping is often used to enhance initialization. However, traditional single chaotic mapping (such as Logistic or Sine) can also lead to chaotic degradation. For example, when μ=4, the output of the Logistic mapping tends to fall into a periodic orbit, with the point set densely distributed near the diagonal of the phase space; the Sine mapping produces high-density aggregation in the boundary area, significantly reducing the ergodicity.

[0041] In order to improve the performance of the AdamH algorithm in the initialization phase, the present invention introduces a CH chaotic map to perform initialization, and its mathematical expression is: ; (16) Where α≥0, β≥0 and α+β≤1, u∈[0,4], γ ∈[2.5,3.0], these are mapping control parameters, zi is the current chaotic state value, z i+1 is the next state value output by the CH chaotic map, and satisfies zi+1 ∈[0,1]; w i is the intermediate weight value of the i-th iteration used to generate the chaotic sequence; λ is the interpolation parameter, which controls the balance between deterministic and random components; r is the random factor, which introduces random perturbations to enhance ergodicity; π is pi, which is approximately equal to 3.14159; sin is the sine function, cos is the cosine function, and || is the absolute value function, which ensures that the output is non-negative.

[0042] CH chaotic mapping (CH) is designed to generate complex and uniformly distributed chaotic sequences through a specific two-layer processing architecture. In phase space behavior analysis, the point sets generated by logistic and sinusoidal mapping tend to be concentrated in a limited region, exhibiting significant chaotic degradation. In contrast, CH chaotic mapping exhibits a highly uniform phase space distribution, significantly improving the randomness and robustness of its output sequence.

[0043] After initialization, the population is sorted and ranked based on fitness values, and behavior probabilities are assigned to each individual: ; (17) ; (18) In the above formula, sorted_indices is the individual index sorted by fitness value, f i is the fitness value (loss function value) of the i-th individual, N is the number of individuals in the population, argsort returns the function of the sorted index, P foraging,i is the probability of foraging behavior of the i-th individual, P defense,i is the probability of defensive behavior of the i-th individual, P escape,i is the probability of escape behavior of the i-th individual, P social,i is the probability of social behavior of the i-th individual, rank i represents ranking, e is a natural constant, approximately equal to 2.71828.

[0044] In order to make full use of gradient information, AdamH integrates the gradient direction during the update process, calculates the gradient of the current individual, and integrates the gradient information into the behavior: ; (19) Where, X i gradient represents the individual position after fusion of gradient information, X i current represents the current individual position, as the learning rate that decays over time t, g(·,·) is an adaptive function that adjusts the step size based on the information of the gradient and the Hessian matrix, ∇f(θ i) is the fitness function f with respect to parameter θ i The gradient of θ i is the parameter vector of the i-th individual, H i is the fitness function f in θ i The Hessian matrix of the point.

[0045] According to the above formula, the adaptive learning rate parameter of the i-th individual is Not only decays over time, but also according to the local gradient And the curvature (Hessian matrix Hi) that can be regarded as "terrain" is adaptively adjusted: ; (20) Inheriting the momentum mechanism of the Adam optimizer, AdamH maintains first-order and second-order momentum estimates, continuously saving and updating this momentum information: ;(twenty one) ;(twenty two) Where β1 and β2 are momentum decay parameters; is the first-order moment estimate (momentum) at time t, is the second-order moment estimate at time t, is the updated individual position.

[0046] The final position update formula combining all behavior modes is: ;(twenty three) ;(twenty four) The coefficients in the formula are based on the bias-corrected first-order momentum and second-order momentum When the gradient direction is stable (alignment is high), α H,t becomes smaller, trust the gradient; when the gradient oscillates (low alignment), α H,t becomes larger, reducing dependence on gradients. is the final updated individual position, is the basic value of the dynamic coefficient, is the numerical stability coefficient, is the behavior weight vector, exp(·) is the exponential function, E is a small constant used for numerical stability, and softmax(...) is the Softmax function that converts a numeric vector into a probability distribution whose sum is 1.

[0047] The convergence of the algorithm is monitored by the following indicators: ; (25) In the formula, Convergence is the convergence judgment index, X best is the current optimal individual position, is the convergence threshold. When the average distance is less than this threshold, the algorithm is considered to have converged.

[0048] Repeat the above steps until the termination condition (maximum number of iterations or convergence) is met, and finally output the trained parameters.

Claims

1. A driver abnormality monitoring system based on large model collaborative decision-making, characterized by: include: Multi-channel cameras, voice interaction module, DTOPA visual pre-alignment module, Qwen-Apan lightweight language reasoning module, collaborative decision-making module, and safety response control module; among them, multi-channel cameras are used to extract the in-vehicle environment video stream in real time; the voice interaction module is used to remind the driver and interact with the driver, and finally feed back the results to the entire system as part of the decision-making; the DTOPA visual pre-alignment module is used to convert the video stream into a structured text description; the Qwen-Apan lightweight language reasoning module is used to perform semantic analysis and abnormal risk assessment on the text description; the collaborative decision-making module is used to trigger voice interaction and vehicle control instructions based on the risk assessment results; the safety response control module is used to perform pull-over parking, emergency contact notification or multi-level warning operations.

2. The driver abnormality monitoring system based on large model collaborative decision-making according to claim 1 is characterized in that: The DTOPA visual pre-alignment module is specifically designed as follows: a multimodal data augmentation strategy is used to generate a driving text video (DTV) dataset that simulates the temporal characteristics of real videos; the CLIP-ViT-L model is used to align image and text features; The CLIP dependency is eliminated through visual ability internalization technology, and multi-scale convolutional embedding and modality adaptive normalization are adopted.

3. The driver abnormality monitoring system based on large model collaborative decision-making according to claim 2 is characterized in that: The specific multimodal data enhancement strategy is as follows: LLM is used to automatically generate multimodal pre-training data and create a large-scale driving text video DTV dataset containing text videos and their associated annotations.

4. The driver abnormality monitoring system based on large model collaborative decision-making according to claim 1 is characterized in that: The Qwen-Apan lightweight language inference module is specifically as follows: the GPTQ-Int8 quantization scheme is used to compress model weights; the KV cache is dynamically allocated based on attention entropy.

5. The driver abnormality monitoring system based on large model collaborative decision-making according to claim 1 is characterized in that: The collaborative decision-making module is as follows: define the driver status understanding indicator (DSU); trigger three levels of response based on the abnormality level: voice reminder, emergency contact notification, and assisted pull-over. The DSU formula is as follows: ； Among them, w risk is the risk-weighted function; Indicates the accuracy of driver status description, groundtruth i is the basic facts of the i-th sample, including facts such as the in-vehicle scene, driver status, and risk level information. N is the total number of test samples, which is a positive integer. α is the description accuracy weight coefficient, which controls the importance of description accuracy in the overall evaluation. β is the state classification accuracy weight coefficient, which controls the importance of classification accuracy in the overall evaluation. i is the sample index from 1 to N. It represents the classification accuracy of driver status, and its calculation formula is as follows: ； ； in, represents the i-th driver state description generated by DTOPA, represents the standard description of the dataset annotation; ROUGE-L represents the longest common subsequence ratio, Indicates whether the corresponding label is mentioned in the j-th driver state description; represents the true driver state label, 、 Both represent the total number of test samples; j is the classification sample index from 1 to M.

6. The driver abnormality monitoring system based on large model collaborative decision-making according to claim 5 is characterized in that: By simulating the diverse behavioral patterns of the Lianfeng Horned Toad, the AdamH optimizer is designed to achieve parameter optimization of the multi-strategy collaborative module.

7. The driver abnormality monitoring system based on large model collaborative decision-making according to claim 6 is characterized in that: The CH chaotic map initialization parameters are introduced to integrate the behavior patterns of the Lianfeng horned toad, namely foraging, defense, escape, and social update gradient directions. The CH chaotic map initialization parameter formula is as follows: ； Where, α≥0, β≥0 and α+β≤1, u∈[0,4], γ ∈[2.5,3.0], are mapping control parameters, z i is the current chaotic state value, z i+1 is the next state value output by the CH chaotic map, and satisfies z i+1 ∈[0,1]; w i is the intermediate weight value of the i-th iteration used to generate the chaotic sequence; λ is the interpolation parameter, which controls the balance between deterministic and random components; r is the random factor, which introduces random perturbations to enhance ergodicity; π is pi, which is approximately equal to 3.14159; sin is the sine function, cos is the cosine function, and || is the absolute value function, which ensures that the output is non-negative.

8. A driver abnormality monitoring method, characterized in that: The implementation of the driver abnormality monitoring system based on large model collaborative decision-making includes the following steps: (1) Capture in-car video streams at 25 FPS through multiple cameras; (2) Generate scene text descriptions in real time using the DTOPA model; (3) Use the Qwen-Apan model to parse the text and output a risk assessment; (4) If the detection is abnormal, start the voice interaction confirmation state; (5) Perform security response operations based on the interaction results.

9. A driver abnormality monitoring method according to claim 8, characterized in that: In step (1), a multimodal data enhancement strategy is adopted to generate a driving text video DTV dataset.

10. The method for monitoring driver abnormality according to claim 8, characterized in that: In step (2), a 21-second sliding window mechanism is used to randomly select 20 frames to form an analysis batch, and the compressed frame data is transmitted through WebSocket.

Images