A driver abnormal behavior recognition method and system based on continuous learning

Abstract

This invention discloses a method and system for identifying abnormal driver behavior based on continuous learning, relating to the field of driving behavior detection technology. The method includes dividing a driver monitoring image into image blocks and projecting them into feature vectors, adding positional encoding, and combining extracted prompt features with segmentation into prompt vectors to achieve dual coverage of global and key local features, improving feature extraction accuracy. The two types of vectors are concatenated into a fused feature sequence, input to a pre-trained Vision Transformer model with frozen parameters, and enhanced features are extracted through a self-attention mechanism. Then, a global feature vector is obtained through average pooling. Continuous learning is achieved by outputting a vector of all seen task logic values, using a binary mask for filtering, and classifying using the argmax function. This allows for adaptation to behavior category expansion, balancing real-time performance with stability in complex environments, improving the accuracy of similar behavior recognition, and making it easily deployable in in-vehicle scenarios.

Description

Technical Field

[0001] This invention relates to the field of driving behavior detection technology, specifically to a method and system for identifying abnormal driver behavior based on continuous learning. Background Technology

[0002] With the rapid development of automotive intelligence technology, Advanced Driver Assistance Systems (ADAS) and autonomous driving technologies have made significant progress. However, during the long transition period to fully autonomous driving, human-machine collaborative driving will remain the dominant mode, with drivers continuing to play an irreplaceable core role in vehicle control. Against this backdrop, driver status monitoring systems have become a key technology for ensuring road safety. Their core function is to identify abnormal driver behavior in real time and accurately, including but not limited to operating mobile electronic devices, excessive interaction with passengers, smoking, and taking both hands off the steering wheel. Timely detection and early warning of these abnormal states are important technical means to prevent traffic accidents and improve road safety.

[0003] Currently, deep learning-based methods have become the mainstream technology in the field of abnormal driver behavior recognition, with models such as convolutional neural networks and Vision Transformers demonstrating excellent performance in controlled experimental environments. However, when these models are transferred from closed laboratory environments to real, open, and dynamically changing driving scenarios, their practical application faces severe challenges. Real driving environments are highly complex and uncertain, specifically manifested in variations in in-vehicle environments depending on the vehicle model, different camera installation locations, dynamic changes in lighting conditions with day and night and weather, and the continuous emergence of new distraction behaviors with technological advancements. Once traditional models are trained, their knowledge systems tend to solidify, making it difficult to adapt to the continuously evolving external environment, resulting in a significant decline in model performance over time.

[0004] To address the problem of model aging, traditional solutions rely on retraining or fine-tuning the model using newly acquired data. However, this approach faces significant practical obstacles. High-quality data annotation requires substantial investment of manpower and time, and obtaining a sufficient number of labeled samples is difficult even for certain low-incidence anomalies. This heavy reliance on labeled data severely restricts the iteration speed and deployment flexibility of models in practical applications. More problematic is that if existing models are updated solely with new task data, they inevitably suffer from "catastrophic forgetting," where the learning of new knowledge drastically overwrites and forgets previously learned old task features, leading to a sharp decline in model performance on the original recognition task. This deficiency makes it difficult for existing models to maintain their ability to recognize known anomalies while continuously adapting to emerging scenarios and behavioral patterns, thus greatly limiting their practical value and lifespan in long-term deployment.

[0005] Although existing research attempts to mitigate this problem through methods such as model fine-tuning or ensemble learning, fine-tuning can lead to catastrophic forgetting, while model ensemble results in a linear increase in computing resources and storage overhead, making it difficult to meet the resource constraints of in-vehicle edge computing devices. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the purpose of this invention is to provide a method and system for identifying abnormal driver behavior based on continuous learning, in order to solve the above-mentioned problems described in the prior art.

[0007] A first aspect of the present invention is to provide a method for identifying abnormal driver behavior based on continuous learning, the method comprising: The input driver monitoring image is evenly divided into multiple image blocks, and each image block is projected into a feature vector to form a feature vector sequence. At the same time, a position code is added to each feature vector to inject spatial structure information. Extract prompt features from the driver monitoring image and segment the prompt features into multiple prompt vectors; The multiple cue vectors are concatenated with the feature vector sequence to form a fused feature sequence, which is then input into a pre-trained Vision Transformer model with frozen parameters. Feature extraction is performed through a self-attention mechanism, and an enhanced feature sequence is output. Select the feature vector corresponding to the prompt vector from the enhanced feature sequence and perform average pooling to obtain the global feature vector; The global feature vector is classified and processed to output the original logical value vector corresponding to all seen task categories. The original logical value vector is then filtered by a binary mask bound to the current task, and the argmax function is used to determine the category of abnormal driver behavior.

[0008] According to one aspect of the above technical solution, the step of extracting cue features from the image block and segmenting the cue features into multiple cue vectors includes: The input image is used to extract features through a pre-trained lightweight convolutional neural network, and a multi-dimensional cue feature map is output. The multidimensional cue feature map is flattened in the spatial dimension and transformed into a one-dimensional continuous feature vector to preserve the spatial correlation of the features. The one-dimensional continuous feature vector is evenly divided into multiple segments of the same dimension according to a preset number, and each segment serves as a cue vector for subsequent feature fusion.

[0009] According to one aspect of the above technical solution, the step of extracting features from the input image using a pre-trained lightweight convolutional neural network and outputting a multidimensional cue feature map includes: A pre-trained ResNet-18 network model is used as the basic feature extractor. Global average pooling layers and fully connected layers are removed, while convolutional layer structures are retained to extract spatial features from the input image. The input image is fed into the ResNet-18 network model for forward propagation, and a multi-dimensional cue feature map is output to preserve the multi-scale spatial semantic information of the input image.

[0010] According to one aspect of the above technical solution, the steps of concatenating the multiple cue vectors with the feature vector sequence to form a fused feature sequence, inputting the parameter-frozen pre-trained Vision Transformer model, extracting features through a self-attention mechanism, and outputting an enhanced feature sequence include: The multiple cue vectors are used as prefixes and sequentially concatenated with the feature vector sequence to form a fused feature sequence, wherein the cue vectors are located at the beginning of the sequence and the image feature vectors are located at the end of the sequence. The pre-trained Vision Transformer model with the input parameters of the fused feature sequence is used to perform global interaction and deep enhancement on all features using the built-in multi-head self-attention mechanism. Deep feature extraction is performed through multiple Transformer encoding layers of the Vision Transformer model, outputting an enhanced feature sequence of the same length as the feature vector sequence.

[0011] According to one aspect of the above technical solution, the pre-trained VisionTransformer model with frozen input parameters of the fused feature sequence, and the step of performing global interaction and depth enhancement on all features using the built-in multi-head self-attention mechanism, includes: The fused feature sequence is input into a pre-trained VisionTransformer model containing multiple Transformer coding layers, each of which is equipped with a multi-head self-attention mechanism and a feedforward neural network. The global dependencies between all feature vectors in the fused feature sequence are modeled using the multi-head self-attention mechanism, and the attention weight of each feature vector with all other vectors is calculated. The feedforward neural network is used to perform nonlinear transformation and feature enhancement on the output of the multi-head self-attention mechanism, resulting in an enhanced feature sequence with improved semantic representation capabilities.

[0012] According to one aspect of the above technical solution, the global feature vector is classified to output the original logical value vector corresponding to all seen task categories. The original logical value vector is then filtered using a binary mask bound to the current task. The step of using the argmax function to determine the driver's abnormal behavior category includes: The global feature vector is input into a classification head module containing multiple fully connected layers, and the feature is transformed through a non-linear activation function to output the original logical value vector corresponding to all learned task categories. The corresponding binary mask is dynamically loaded based on the current task identifier, and the output position corresponding to the current task is set to 1, while the output position of non-current tasks is set to 0. The original logical value vector is multiplied element-wise with the binary mask to block the output channels of non-current tasks, retain the valid output of the current task, and the final behavior classification result is determined by the argmax function.

[0013] According to one aspect of the above technical solution, the step of inputting the global feature vector into a classification head module containing multiple fully connected layers, performing feature transformation through a nonlinear activation function, and outputting the original logistic value vector corresponding to all learned task categories includes: The global feature vector is input into the classification head module, which consists of two fully connected layers. The first fully connected layer maps the input feature dimension to the intermediate hidden dimension and uses the ReLU activation function for nonlinear transformation. The intermediate features activated by ReLU are input into the second fully connected layer, which maps the feature dimension to the output dimension equal to the total number of all learned task categories, generating the original logistic vector.

[0014] A second aspect of the present invention is to provide a driver abnormal behavior recognition system based on continuous learning, applied to the method shown in the above-described technical solution, the system comprising: The image processing module is used to uniformly divide the input driver monitoring image into multiple image blocks, project each image block into a feature vector to form a feature vector sequence, and add position encoding to each feature vector to inject spatial structure information. The feature extraction module is used to extract prompt features from the driver monitoring image and segment the prompt features into multiple prompt vectors; The feature enhancement module is used to concatenate the multiple cue vectors with the feature vector sequence into a fused feature sequence, input a pre-trained Vision Transformer model with frozen input parameters, extract features through a self-attention mechanism, and output an enhanced feature sequence. The feature aggregation module is used to select the feature vector corresponding to the prompt vector from the enhanced feature sequence and perform average pooling operation to obtain the global feature vector; The behavior classification module is used to classify the global feature vector, output the original logical value vector corresponding to all seen task categories, and filter the original logical value vector through a binary mask bound to the current task, and use the argmax function to determine the abnormal behavior category of the driver.

[0015] A third aspect of the present invention is to provide a readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method described in the above-described technical solution.

[0016] A fourth aspect of the present invention is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in the above technical solutions.

[0017] Compared with existing technologies, the driver abnormal behavior recognition method and system based on continuous learning shown in this invention have the following advantages: When identifying abnormal driver behavior, this system achieves dual coverage of global and key local features through image block segmentation, location encoding injection, and cue feature extraction and segmentation. This ensures the integrity and spatial relevance of image information while effectively filtering redundant interference, significantly improving the accuracy and efficiency of feature extraction and reducing recognition errors caused by missing or redundant features. Simultaneously, by outputting the original logical value vectors of all known task categories and dynamically filtering them using binary masks, the system overcomes the core pain point of the model, achieving continuous learning capabilities. This allows for compatibility with previously learned categories and rapid adaptation to new tasks, reducing the training cost of adding new tasks and adapting to the dynamic expansion needs of behavior categories. Furthermore, a pre-trained Vision Transformer model with parameter freezing, combined with average pooling dimensionality reduction and self-attention mechanisms, collaboratively optimizes inference efficiency and generalization ability, balancing real-time performance in in-vehicle scenarios and recognition stability in complex environments. Deep fusion guided by cue vectors and focused feature integration enhance the semantic expression and category discrimination of features, effectively improving the recognition accuracy of similar abnormal behaviors. Attached Figure Description

[0018] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart illustrating the driver abnormal behavior recognition method based on continuous learning provided in an embodiment of the present invention. Figure 2A structural diagram of a driver abnormal behavior recognition model based on continuous learning provided in an embodiment of the present invention; Figure 3 This is a structural block diagram of a driver abnormal behavior recognition system based on continuous learning, provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Several embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of the present invention will be more thorough and complete.

[0020] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0022] Example 1 Please see Figure 1 and Figure 2 The first embodiment of the present invention provides a method for identifying abnormal driver behavior based on continuous learning, the method comprising steps S10-S50: Step S10: The input driver monitoring image is evenly divided into multiple image blocks, and each image block is projected as a feature vector to form a feature vector sequence. At the same time, a position code is added to each feature vector to inject spatial structure information.

[0023] In this embodiment, the input driver monitoring image is evenly divided into multiple image blocks and projected into feature vectors to form a feature vector sequence. The purpose is to transform the two-dimensional image into a one-dimensional feature form that the model can process. The even division ensures the integrity and uniformity of the image information and avoids the omission of local features.

[0024] Adding positional encoding is to compensate for the spatial structure information lost during feature vector projection, enabling the model to perceive the relative positional relationship of different image blocks in the original image. This is crucial for recognition tasks that rely on spatial layout, such as driver behavior. For example, spatial information such as the relative position of the hands and steering wheel and the orientation of the head directly affects the accuracy of behavior judgment.

[0025] Specifically, in this embodiment, the input driver monitoring image I (with dimensions H×W×3) is first uniformly divided into a grid consisting of M rows and N columns of image blocks, where H is the image height, W is the image width, and the size of each image block is P×P, where P is the side length of a single image block, satisfying M=H / P and N=W / P. The image height H is divided by the side length P of a single image block to obtain the number of vertical blocks M, and the image width W is divided by the side length P of a single image block to obtain the number of horizontal blocks N.

[0026] Subsequently, a trainable feature mapping module E with dimension L×( ×3), project each image patch into a high-dimensional embedding space to generate a feature vector of dimension L. 0 represents the initial / original feature vector before positional encoding, and i and j are the image patch in the i-th row and j-th column of the grid corresponding to this feature vector, thus forming an initial feature vector sequence containing M×N feature vectors. .

[0027] Then, for the feature vector sequence Each feature vector in Overlay a preset, unique position code. The dimension is (M×N)×L, which is used to inject the two-dimensional spatial structure information of each image patch in the driver monitoring image I into the feature representation, expressed as: This is to ensure that the model can perceive the local and global context of the image.

[0028] Step S20: Extract prompt features from the driver monitoring image and segment the prompt features into multiple prompt vectors.

[0029] In this embodiment, extracting cue features from the input image and segmenting them into multiple cue vectors is a step that condenses and enhances the core information of the image. Abnormal driver behavior often has key feature regions, such as facial expressions and body movements. Extracting cue features can focus on this high-value information and reduce irrelevant background interference. Segmenting it into multiple cue vectors is to adapt to the subsequent fusion process, so that the cue information can form a precise interactive correspondence with the image block feature vectors, laying the foundation for subsequent feature enhancement.

[0030] Specifically, the input image, namely the driver monitoring image I, is input in parallel into the lightweight cue learning module. This module extracts features in a hierarchical manner and outputs a compact cue feature map. Subsequently, the feature processing module reshapes and segments the prompt feature map to generate... A cue vector of dimension L , dimension .

[0031] Step S30: Concatenate the multiple cue vectors with the feature vector sequence to form a fused feature sequence, input the parameter-frozen pre-trained Vision Transformer model, extract features through a self-attention mechanism, and output an enhanced feature sequence.

[0032] In this embodiment, multiple cue vectors and feature vector sequences are concatenated into a fused feature sequence. The pre-trained Vision Transformer model with frozen input parameters is used to extract features through a self-attention mechanism, and the enhanced feature sequence is output.

[0033] The stitching operation organically combines key prompts with global image features, enabling the model to focus on high-value prompt areas while processing global information. By using a pre-trained model and freezing parameters, the model can fully utilize the general visual features it has already learned, avoiding problems such as large sample requirements and low training efficiency caused by training from scratch. It can also capture the global dependencies between features through the self-attention mechanism, achieving in-depth feature enhancement. This makes the fused features more semantically expressive and able to accurately depict the essential characteristics of driver behavior.

[0034] Specifically, The first cue vector is used as a prefix and sequentially concatenated with the feature vectors of the M×N encoded positions to form a feature vector of length M. The N+k fusion feature sequence is expressed as follows: The dimension is (M×N+k)×L. This fused feature sequence is fed into a large-scale pre-trained Vision Transformer model with frozen parameters. In this context, the frozen encoder / feature mapping function utilizes its built-in self-attention mechanism to perform global interaction and depth enhancement on all features. For the l-th layer encoder, the attention calculation expression is:

[0035] Where Q, K, and V are obtained from the input through linear transformation, and are the query vector, key vector, and value vector, respectively. T Let K be the transpose of matrix K. Using the scaling factor, the L-layer encoder outputs an enhanced feature sequence with stronger semantic representation capabilities. The dimension is (M×N+k)×L.

[0036] Step S40: Select the feature vector corresponding to the prompt vector from the enhanced feature sequence and perform average pooling to obtain the global feature vector.

[0037] In this embodiment, feature vectors corresponding to the cue vectors are selected from the enhanced feature sequence and subjected to average pooling to obtain a global feature vector. This step focuses and integrates the enhanced features. Since the cue vectors correspond to key information in the image, selecting their corresponding feature vectors can filter out irrelevant and redundant features. The average pooling operation can integrate multiple key feature vectors into a global feature vector with unified dimensions and comprehensive information. This retains the core information of the key features while reducing the feature dimensionality, providing efficient and accurate input for subsequent classification processing.

[0038] Specifically, from the output enhanced feature sequence, the set of feature vectors of the k output feature vectors corresponding to the cue vectors at the beginning of the sequence is precisely selected. By performing average pooling on this set of output feature vectors, their scattered semantic information is aggregated, and finally condensed into a global feature vector of dimension L representing the entire driver monitoring image I, expressed as: ,in, This is the scaling factor. Let be the i-th output feature vector.

[0039] Step S50: Classify the global feature vector, output the original logical value vector corresponding to all seen task categories, and filter the original logical value vector through a binary mask bound to the current task, and use the argmax function to determine the driver's abnormal behavior category.

[0040] In this embodiment, the global feature vector is classified and output as a vector of original logical values ​​corresponding to all seen task categories. After filtering by a binary mask bound to the current task, the argmax function is used to determine the category of abnormal behavior.

[0041] The classification process is the core step that maps feature vectors to specific behavior categories, outputting the original logical value vectors of all seen task categories. This reflects the characteristics of continuous learning and is compatible with previously learned task categories. The binary mask filtering operation is designed to adapt to the current task requirements, shielding outputs from non-current tasks to avoid category interference and ensure the accuracy of the classification results. The argmax function is then used to quickly determine the behavior category with the highest probability from the filtered valid outputs, completing the final recognition.

[0042] Specifically, the global feature vector Input a trainable classification head module This module outputs a raw logical value vector corresponding to all seen task categories (including historical and current tasks). Subsequently, a binary mask bound to the currently active task is invoked. For the original logical value vector To perform element-level filtering, the expression is: = ⊙ Output channels not related to the current task are masked. Finally, based on the activated output channels, the specific category of abnormal driver behavior in driver monitoring image I is determined by the argmax function. .

[0043] Compared with existing technologies, the driver abnormal behavior recognition method based on continuous learning shown in this embodiment has the following advantages: When identifying abnormal driver behavior, this system achieves dual coverage of global and key local features through image block segmentation, location encoding injection, and cue feature extraction and segmentation. This ensures the integrity and spatial relevance of image information while effectively filtering redundant interference, significantly improving the accuracy and efficiency of feature extraction and reducing recognition errors caused by missing or redundant features. Simultaneously, by outputting the original logical value vectors of all known task categories and dynamically filtering them using binary masks, the system overcomes the core pain point of the model, achieving continuous learning capabilities. This allows for compatibility with previously learned categories and rapid adaptation to new tasks, reducing the training cost of adding new tasks and adapting to the dynamic expansion needs of behavior categories. Furthermore, a pre-trained Vision Transformer model with parameter freezing, combined with average pooling dimensionality reduction and self-attention mechanisms, collaboratively optimizes inference efficiency and generalization ability, balancing real-time performance in in-vehicle scenarios and recognition stability in complex environments. Deep fusion guided by cue vectors and focused feature integration enhance the semantic expression and category discrimination of features, effectively improving the recognition accuracy of similar abnormal behaviors.

[0044] Example 2 The second embodiment of the present invention also provides a method for identifying abnormal driver behavior based on continuous learning, which is basically similar to the method shown in the first embodiment, except that: In this embodiment, the step of extracting cue features from the image patch and segmenting the cue features into multiple cue vectors includes: The input image is used to extract features through a pre-trained lightweight convolutional neural network, and a multi-dimensional cue feature map is output. The multidimensional cue feature map is flattened in the spatial dimension and transformed into a one-dimensional continuous feature vector to preserve the spatial correlation of the features. The one-dimensional continuous feature vector is evenly divided into multiple segments of the same dimension according to a preset number, and each segment serves as a cue vector for subsequent feature fusion.

[0045] Specifically, this embodiment uses a lightweight convolutional neural network, which can ensure the efficiency of feature extraction, avoid the computational overhead of complex models, and accurately extract key features in the image by leveraging the advantages of pre-trained models. The output multi-dimensional cue feature map can retain multi-scale information of features and adapt to the feature expression needs of different drivers' abnormal behaviors. For example, features of different scales, such as subtle changes in facial expressions and obvious body movements, can all be reflected in the multi-dimensional feature map.

[0046] Furthermore, the multidimensional cue feature map is flattened in the spatial dimension and transformed into a one-dimensional continuous feature vector. The purpose is to convert the two-dimensional feature map into a one-dimensional vector form for subsequent segmentation operations. At the same time, the spatial correlation of features is strictly preserved during the flattening process to ensure that features at different locations can still maintain their original relative relationships in the one-dimensional vector, avoiding the loss of spatial information. This is because key features of driver behavior often depend on the spatial correlation of different parts, such as the relative positions of the hands and face.

[0047] Furthermore, the one-dimensional continuous feature vector is evenly divided into multiple segments with the same dimension according to a preset number. Even division ensures that the dimension of each prompt vector is consistent, which facilitates subsequent splicing and interaction with the feature vector sequence. The preset number is determined according to the processing capability of the model and the feature expression requirements. This ensures that each prompt vector contains sufficient key information, while avoiding insufficient or redundant feature expression caused by a single prompt vector being too large or too small in dimension. This ensures that the prompt vector can effectively assist in subsequent feature fusion and enhancement.

[0048] In this embodiment, the steps of extracting features from the input image using a pre-trained lightweight convolutional neural network and outputting a multidimensional cue feature map include: A pre-trained ResNet-18 network model is used as the basic feature extractor. Global average pooling layers and fully connected layers are removed, while convolutional layer structures are retained to extract spatial features from the input image. The input image is fed into the ResNet-18 network model for forward propagation, and a multi-dimensional cue feature map is output to preserve the multi-scale spatial semantic information of the input image.

[0049] It should be noted that ResNet-18, as a classic lightweight convolutional neural network, has advantages such as simple structure, high computational efficiency, and strong feature extraction capabilities, making it very suitable for real-time processing of driver monitoring images.

[0050] In this embodiment, the corresponding global average pooling layer and fully connected layer are removed from the model architecture because these two layers are mainly used for the final classification task. The core of this step is to extract intermediate spatial features. Retaining the convolutional layer structure can give full play to its ability to capture local features and spatial relationships, and accurately extract spatial features related to driver behavior in the input image.

[0051] Furthermore, during the forward propagation of the input image into the ResNet-18 network model, the convolutional layers extract low-level features (such as edges and textures) and high-level features (such as facial contours and limb shapes) of the image step by step through sliding window operations. The output of the multi-dimensional cue feature map can retain the multi-scale spatial semantic information extracted by different convolutional layers. This information covers the detailed and overall features of the driver's behavior, providing rich and comprehensive basic data for subsequent cue vector segmentation and feature fusion, ensuring that the cue features can accurately reflect the key information of the driver's behavior.

[0052] In this embodiment, the steps of concatenating the multiple cue vectors with the feature vector sequence to form a fused feature sequence, inputting the parameter-frozen pre-trained Vision Transformer model, extracting features through a self-attention mechanism, and outputting an enhanced feature sequence include: The multiple cue vectors are used as prefixes and sequentially concatenated with the feature vector sequence to form a fused feature sequence, wherein the cue vectors are located at the beginning of the sequence and the image feature vectors are located at the end of the sequence. The pre-trained Vision Transformer model with the input parameters of the fused feature sequence is used to perform global interaction and deep enhancement on all features using the built-in multi-head self-attention mechanism. Deep feature extraction is performed through multiple Transformer encoding layers of the Vision Transformer model, outputting an enhanced feature sequence of the same length as the feature vector sequence.

[0053] Specifically, in this embodiment, the concatenation method of using multiple cue vectors as prefixes and sequentially concatenating them with the feature vector sequence enables the pre-trained Vision Transformer model to prioritize key cue features during processing. At the same time, it combines global image features to achieve global feature interaction guided by key information. The layout with cue vectors first conforms to the processing logic of the Transformer model's self-attention mechanism, enabling the model to focus on core features more quickly and improving the efficiency and accuracy of feature enhancement.

[0054] Furthermore, in the pre-trained Vision Transformer model with the input parameters of the fused feature sequence frozen, the above parameter freezing setting can not only make full use of the general visual features already learned by the pre-trained model and avoid the model forgetting past knowledge when training on new tasks (reflecting the continuous learning characteristics), but also reduce training costs; while the multi-head self-attention mechanism can capture the dependencies between features from multiple dimensions at the same time, realize the global interaction of features, and fully integrate the cue features and image features to enhance each other and improve the semantic expressive ability of the features.

[0055] Furthermore, deep feature extraction is performed through multiple Transformer encoding layers of the Vision Transformer model, which can gradually refine the semantic expression of features and transform shallow features into more abstract and discriminative deep features. The final output is an enhanced feature sequence of the same length as the original feature vector sequence. This is to ensure that the feature vectors corresponding to the prompt vectors can be accurately selected in the subsequent steps, maintain the structural consistency of the feature sequence, and provide suitable input for the subsequent pooling and classification steps.

[0056] In this embodiment, the steps of freezing the input parameters of the pre-trained VisionTransformer model with the fused feature sequence and performing global interaction and depth enhancement on all features using the built-in multi-head self-attention mechanism include: The fused feature sequence is input into a pre-trained VisionTransformer model containing multiple Transformer coding layers, each of which is equipped with a multi-head self-attention mechanism and a feedforward neural network. The global dependencies between all feature vectors in the fused feature sequence are modeled using the multi-head self-attention mechanism, and the attention weight of each feature vector with all other vectors is calculated. The feedforward neural network is used to perform nonlinear transformation and feature enhancement on the output of the multi-head self-attention mechanism, resulting in an enhanced feature sequence with improved semantic representation capabilities.

[0057] Specifically, in this embodiment, the superposition of multiple Transformer encoding layers can achieve layer-by-layer feature deepening and gradually improve the semantic expressive power of the features; while the combination of multi-head self-attention mechanism and feedforward neural network is the core of Transformer model to efficiently process features. The former is responsible for capturing the global dependencies between features, and the latter is responsible for the nonlinear transformation of features. The two work together to achieve deep feature enhancement.

[0058] Furthermore, a multi-head self-attention mechanism is used to model the global dependencies between all feature vectors in the fused feature sequence. Calculating the attention weights of each feature vector with all other vectors is the key to achieving global feature interaction. The identification of abnormal driver behavior requires comprehensive consideration of feature associations in different regions of the image, such as the coordination of head posture and hand movements. The multi-head self-attention mechanism can accurately capture these cross-regional dependencies. By allocating attention weights, the model focuses on feature associations related to behavior recognition and weakens the influence of irrelevant associations.

[0059] Furthermore, while the features output by the multi-head self-attention mechanism capture the dependencies, they still require further transformation and enhancement to better adapt to classification tasks. Feedforward neural networks, through non-linear activation functions such as ReLU, can perform complex transformations on features, mine deep semantic information of features, improve the discriminative ability of features, and ensure that the output enhanced feature sequence can accurately characterize the essential features of driver behavior, laying a solid foundation for subsequent global feature vector extraction and classification processing.

[0060] In this embodiment, the steps of classifying the global feature vector, outputting the original logical value vector corresponding to all seen task categories, filtering the original logical value vector using a binary mask bound to the current task, and determining the driver's abnormal behavior category using the argmax function include: The global feature vector is input into a classification head module containing multiple fully connected layers, and the feature is transformed through a non-linear activation function to output the original logical value vector corresponding to all learned task categories. The corresponding binary mask is dynamically loaded based on the current task identifier, and the output position corresponding to the current task is set to 1, while the output position of non-current tasks is set to 0. The original logical value vector is multiplied element-wise with the binary mask to block the output channels of non-current tasks, retain the valid output of the current task, and the final behavior classification result is determined by the argmax function.

[0061] Specifically, the multiple fully connected layers in the classification head module enable precise mapping from feature vectors to class logistic values. Non-linear activation functions such as ReLU enhance the model's ability to handle complex features and capture the non-linear relationship between features and classes. Outputting the original logistic value vectors of all learned task classes embodies the core idea of ​​continuous learning, enabling the model to be compatible with all previously trained task classes and avoiding forgetting when adding new tasks.

[0062] Furthermore, dynamically loading the corresponding binary mask based on the current task identifier and setting the output position corresponding to the current task to 1 and the output position of non-current tasks to 0 is to adapt to the classification requirements of the current task. Since the original logical value vector contains the category information of all learned tasks, direct classification will be affected by the category of non-current tasks. However, in this embodiment, by dynamically loading the binary mask, the category output channel corresponding to the current task can be accurately filtered out, ensuring that the classification focuses on the current task.

[0063] Furthermore, the element-wise multiplication of the original logical value vector with the binary mask effectively masks the outputs of non-current tasks, allowing the model to focus only on the category judgment of the current task and avoid interference from irrelevant categories. The argmax function can quickly select the category with the highest probability from the effective output as the final recognition result, ensuring the efficiency and accuracy of classification, while meeting the need for continuous learning throughout the process and flexibly responding to the recognition of abnormal behavior in different task scenarios.

[0064] In this embodiment, the steps of inputting the global feature vector into a classification head module containing multiple fully connected layers, performing feature transformation through a non-linear activation function, and outputting the original logistic value vector corresponding to all learned task categories include: The global feature vector is input into the classification head module, which consists of two fully connected layers. The first fully connected layer maps the input feature dimension to the intermediate hidden dimension and uses the ReLU activation function for nonlinear transformation. The intermediate features activated by ReLU are input into the second fully connected layer, which maps the feature dimension to the output dimension equal to the total number of all learned task categories, generating the original logistic vector.

[0065] Specifically, the design of two fully connected layers in the classification head module ensures the depth of feature transformation while avoiding excessive computational overhead and overfitting problems caused by overcomplexity.

[0066] The dimension mapping of the first fully connected layer can transform the global feature vector into an intermediate feature dimension that is more suitable for classification. The ReLU activation function can introduce non-linear factors, enhance the model's ability to express complex features, effectively capture the non-linear relationship between features and categories, and improve the discriminative power of features.

[0067] The core function of the second fully connected layer is to achieve a precise mapping from intermediate features to category logical values. The output dimension is equal to the total number of learned task categories, ensuring that each category has a corresponding logical value output. This reflects the characteristics of continuous learning and is compatible with all previously learned task categories.

[0068] The resulting raw logistic value vector contains prediction probability information for each category, providing comprehensive and accurate input for subsequent binary mask filtering and argmax function to determine the final classification result, ensuring that the classification result accurately reflects the true category of the driver's abnormal behavior.

[0069] Example 3 Please see Figure 2 The third embodiment of the present invention provides a driver abnormal behavior recognition system based on continuous learning, the system comprising: Image processing module 10 is used to uniformly divide the input driver monitoring image into multiple image blocks, project each image block into a feature vector to form a feature vector sequence, and add position encoding to each feature vector to inject spatial structure information. Feature extraction module 20 is used to extract prompt features from the driver monitoring image and segment the prompt features into multiple prompt vectors; Feature enhancement module 30 is used to concatenate the multiple cue vectors with the feature vector sequence to form a fused feature sequence, input a pre-trained Vision Transformer model with frozen input parameters, extract features through a self-attention mechanism, and output an enhanced feature sequence. The feature aggregation module 40 is used to select the feature vector corresponding to the prompt vector from the enhanced feature sequence and perform average pooling operation to obtain the global feature vector; The behavior classification module 50 is used to classify the global feature vector, output the original logical value vector corresponding to all seen task categories, and filter the original logical value vector through a binary mask bound to the current task, and use the argmax function to determine the driver's abnormal behavior category.

[0070] Compared with existing technologies, the driver abnormal behavior recognition system based on continuous learning shown in this embodiment has the following advantages: When identifying abnormal driver behavior, this system achieves dual coverage of global and key local features through image block segmentation, location encoding injection, and cue feature extraction and segmentation. This ensures the integrity and spatial relevance of image information while effectively filtering redundant interference, significantly improving the accuracy and efficiency of feature extraction and reducing recognition errors caused by missing or redundant features. Simultaneously, by outputting the original logical value vectors of all known task categories and dynamically filtering them using binary masks, the system overcomes the core pain point of the model, achieving continuous learning capabilities. This allows for compatibility with previously learned categories and rapid adaptation to new tasks, reducing the training cost of adding new tasks and adapting to the dynamic expansion needs of behavior categories. Furthermore, a pre-trained Vision Transformer model with parameter freezing, combined with average pooling dimensionality reduction and self-attention mechanisms, collaboratively optimizes inference efficiency and generalization ability, balancing real-time performance in in-vehicle scenarios and recognition stability in complex environments. Deep fusion guided by cue vectors and focused feature integration enhance the semantic expression and category discrimination of features, effectively improving the recognition accuracy of similar abnormal behaviors.

[0071] Example 4 A fourth embodiment of the present invention provides a readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the methods described in the above embodiments.

[0072] Example 5 A fifth embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the methods described in the above embodiments.

[0073] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0074] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for identifying abnormal driver behavior based on continuous learning, characterized in that, The method includes: The input driver monitoring image is evenly divided into multiple image blocks, and each image block is projected into a feature vector to form a feature vector sequence. At the same time, a position code is added to each feature vector to inject spatial structure information. Extract prompt features from the driver monitoring image and segment the prompt features into multiple prompt vectors; The multiple cue vectors are concatenated with the feature vector sequence to form a fused feature sequence, which is then input into a pre-trained Vision Transformer model with frozen parameters. Feature extraction is performed through a self-attention mechanism, and an enhanced feature sequence is output. Select the feature vector corresponding to the prompt vector from the enhanced feature sequence and perform average pooling to obtain the global feature vector; The global feature vector is classified and processed to output the original logical value vector corresponding to all seen task categories. The original logical value vector is then filtered by a binary mask bound to the current task, and the argmax function is used to determine the category of abnormal driver behavior.

2. The driver abnormal behavior recognition method based on continuous learning according to claim 1, characterized in that, The steps of extracting cue features from the image patch and segmenting the cue features into multiple cue vectors include: The input image is used to extract features through a pre-trained lightweight convolutional neural network, and a multi-dimensional cue feature map is output. The multidimensional cue feature map is flattened in the spatial dimension and transformed into a one-dimensional continuous feature vector to preserve the spatial correlation of the features. The one-dimensional continuous feature vector is evenly divided into multiple segments of the same dimension according to a preset number, and each segment serves as a cue vector for subsequent feature fusion.

3. The driver abnormal behavior recognition method based on continuous learning according to claim 2, characterized in that, The steps of extracting features from the input image using a pre-trained lightweight convolutional neural network and outputting a multidimensional cue feature map include: A pre-trained ResNet-18 network model is used as the basic feature extractor. Global average pooling layers and fully connected layers are removed, while convolutional layer structures are retained to extract spatial features of the input image. The input image is fed into the ResNet-18 network model for forward propagation, and a multi-dimensional cue feature map is output to preserve the multi-scale spatial semantic information of the input image.

4. The driver abnormal behavior recognition method based on continuous learning according to claim 3, characterized in that, The steps of concatenating the multiple cue vectors with the feature vector sequence to form a fused feature sequence, inputting it into a pre-trained VisionTransformer model with frozen parameters, extracting features through a self-attention mechanism, and outputting an enhanced feature sequence include: The multiple cue vectors are used as prefixes and sequentially concatenated with the feature vector sequence to form a fused feature sequence, wherein the cue vectors are located at the beginning of the sequence and the image feature vectors are located at the end of the sequence. The pre-trained Vision Transformer model with the input parameters of the fused feature sequence is used to perform global interaction and depth enhancement on all features using the built-in multi-head self-attention mechanism. Deep feature extraction is performed through multiple Transformer encoding layers of the Vision Transformer model, outputting an enhanced feature sequence of the same length as the feature vector sequence.

5. The driver abnormal behavior recognition method based on continuous learning according to claim 4, characterized in that, The steps of freezing the input parameters of the fused feature sequence into the pre-trained Vision Transformer model and performing global interaction and depth enhancement on all features using the built-in multi-head self-attention mechanism include: The fused feature sequence is input into a pre-trained Vision Transformer model containing multiple Transformer coding layers, each of which is equipped with a multi-head self-attention mechanism and a feedforward neural network. The global dependencies between all feature vectors in the fused feature sequence are modeled using the multi-head self-attention mechanism, and the attention weight of each feature vector with all other vectors is calculated. The feedforward neural network is used to perform nonlinear transformation and feature enhancement on the output of the multi-head self-attention mechanism, resulting in an enhanced feature sequence with improved semantic representation capabilities.

6. The driver abnormal behavior recognition method based on continuous learning according to any one of claims 1-5, characterized in that, The steps of classifying the global feature vector, outputting the original logical value vector corresponding to all seen task categories, filtering the original logical value vector through a binary mask bound to the current task, and using the argmax function to determine the driver's abnormal behavior category include: The global feature vector is input into a classification head module containing multiple fully connected layers, and the feature is transformed through a non-linear activation function to output the original logical value vector corresponding to all learned task categories. The corresponding binary mask is dynamically loaded based on the current task identifier, and the output position corresponding to the current task is set to 1, while the output position of non-current tasks is set to 0. The original logical value vector is multiplied element-wise with the binary mask to block the output channels of non-current tasks, retain the valid output of the current task, and the final behavior classification result is determined by the argmax function.

7. The driver abnormal behavior recognition method based on continuous learning according to claim 6, characterized in that, The steps of inputting the global feature vector into a classification head module containing multiple fully connected layers, performing feature transformation through a non-linear activation function, and outputting the original logistic value vector corresponding to all learned task categories include: The global feature vector is input into the classification head module, which consists of two fully connected layers. The first fully connected layer maps the input feature dimension to the intermediate hidden dimension and uses the ReLU activation function for nonlinear transformation. The intermediate features activated by ReLU are input into the second fully connected layer, which maps the feature dimension to the output dimension equal to the total number of all learned task categories, generating the original logistic vector.

8. A driver abnormal behavior recognition system based on continuous learning, characterized in that, The system, applicable to the method of any one of claims 1-7, comprises: The image processing module is used to uniformly divide the input driver monitoring image into multiple image blocks, project each image block into a feature vector to form a feature vector sequence, and add position encoding to each feature vector to inject spatial structure information. The feature extraction module is used to extract prompt features from the driver monitoring image and segment the prompt features into multiple prompt vectors; The feature enhancement module is used to concatenate the multiple cue vectors with the feature vector sequence into a fused feature sequence, input a pre-trained Vision Transformer model with frozen input parameters, extract features through a self-attention mechanism, and output an enhanced feature sequence. The feature aggregation module is used to select the feature vector corresponding to the prompt vector from the enhanced feature sequence and perform average pooling operation to obtain the global feature vector; The behavior classification module is used to classify the global feature vector, output the original logical value vector corresponding to all seen task categories, and filter the original logical value vector through a binary mask bound to the current task, and use the argmax function to determine the abnormal behavior category of the driver.

9. A readable storage medium having computer instructions stored thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-7.

Images