A multi-modal driving behavior recognition method, system, device and storage medium

Abstract

This invention relates to a multimodal driving behavior recognition method, system, device, and storage medium, belonging to the field of intelligent driving technology. It includes: acquiring image sequences and corresponding driver behavior category labels; constructing a driving behavior recognition model; inputting the image sequences into the recognition model, extracting spatial semantic features through intra-frame branching, extracting temporal dynamic features through inter-frame branching, and extracting textual features of the category labels through an attention mechanism module; using the textual features as a query vector, dynamically aggregating spatial semantic features and temporal dynamic features to construct spatial view prototypes and temporal view prototypes respectively, and calculating the matching loss for the two types of views accordingly; constructing a joint loss function based on the two types of view matching loss and cross-view alignment loss, and using this function to train the recognition model; inputting the in-vehicle driver behavior image sequence to be recognized into the trained recognition model to complete the in-vehicle driver behavior recognition. This invention improves the accuracy of driving behavior recognition.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and in particular to a multimodal driving behavior recognition method, system, device and storage medium. Background Technology

[0002] With the rapid iteration and upgrading of autonomous driving and intelligent driving technologies, driving safety monitoring has become a core and critical component of intelligent vehicle systems. Driver fatigue, distracted driving, and other abnormal behaviors are significant contributing factors to road traffic accidents. Therefore, achieving real-time and accurate identification of driver behavior is crucial for mitigating driving risks and improving the safety and reliability of intelligent driving systems. Currently, vision-based behavior recognition is an important means of achieving this goal.

[0003] Current vehicle-mounted visual recognition solutions mostly use a single visible light camera to collect image data and utilize visual models to determine driver behavior. However, real-world driving environments are complex and varied, with frequent occurrences of challenging conditions such as low-light nighttime conditions, strong daytime glare, alternating light and dark tunnels, obstructions from objects, and facial occlusion. The imaging stability of a single visible light modality is extremely poor, easily leading to missing image information and feature distortion, resulting in a significant drop in driver behavior recognition accuracy and failing to meet the high-precision monitoring requirements of complex driving environments. To overcome the shortcomings of single-modal recognition, existing technologies are gradually introducing multimodal fusion solutions. By integrating data from multiple sensors, such as visible light cameras and infrared cameras, and combining the imaging advantages of different modalities, these solutions compensate for the recognition deficiencies of single-modal methods in abnormal lighting and obstructed scenarios, effectively improving the comprehensiveness and robustness of driver behavior recognition. However, current mainstream multimodal visual fusion recognition technologies still face many core technical challenges, hindering their practical application in real-world embedded vehicle systems. On the one hand, existing multimodal fusion methods are relatively crude, mostly employing shallow fusion strategies such as simple feature splicing and fixed-weight weighted averaging, without refined correlation modeling of multimodal features. This fails to fully leverage the complementary advantages of different sensor data, resulting in poor effectiveness of multimodal feature fusion. Simultaneously, existing technologies generally ignore the prior textual semantic information contained in driver behavior labels, lacking the ability of high-level textual semantics to guide and constrain visual feature aggregation. This leads to insufficient model differentiation of fine-grained similar driving behaviors such as yawning, nodding, and looking down at a mobile phone, resulting in weak generalization and anti-interference capabilities in complex in-vehicle scenarios. On the other hand, driver behavior recognition is a composite task integrating spatial semantic understanding and temporal dynamic analysis. It requires not only accurately extracting spatial features such as driver head posture, eye state, and hand position from single-frame images, but also capturing dynamic temporal behavioral features such as the duration of yawning, nodding frequency, and handheld device use based on continuous video frame sequences. Existing recognition models struggle to simultaneously achieve high-precision spatial semantic modeling and complex temporal dynamic modeling under constraints of low computing power and low memory overhead. They often need to balance recognition accuracy with computational cost, failing to achieve accurate bidirectional modeling of spatiotemporal features, which seriously affects the integrity and accuracy of driving behavior recognition.

[0004] In addition, existing high-precision visual recognition models mostly rely on large-scale general-purpose visual pre-trained backbone networks, requiring parameter fine-tuning to adapt to in-vehicle driving behavior recognition scenarios. However, real-world in-vehicle scenarios exhibit typical small-sample training characteristics. Directly performing full parameter fine-tuning on large pre-trained models generates enormous computational and memory overhead, far exceeding the computing power and storage resource limits of in-vehicle embedded devices. Furthermore, it easily leads to overfitting on small-sample data, resulting in poor model adaptability. To address the resource consumption issue of full parameter fine-tuning, existing technologies propose lightweight fine-tuning schemes based on adapters, reducing the number of adjustable parameters by inserting a small number of trainable adapter parameters. However, during training, the gradient still needs to flow through the frozen, complete backbone network throughout the training process, generating huge dynamic memory consumption. This makes it unsuitable for the low-computing-power, low-memory deployment environment of in-vehicle systems, resulting in extremely poor practicality and feasibility.

[0005] In summary, existing driver behavior recognition technologies generally suffer from numerous technical shortcomings, including low effectiveness of multimodal feature fusion, weak fine-grained behavior discrimination, insufficient spatiotemporal modeling accuracy, high model training and inference overhead, poor generalization with small samples, and difficulty in adapting to in-vehicle embedded deployment scenarios. Currently, the industry lacks a multimodal driver behavior recognition solution that can simultaneously achieve high-precision recognition, low memory and computational overhead, and effectively integrate visual spatiotemporal features with textual semantic priors. This makes it difficult to meet the real-time, highly robust driver safety monitoring requirements of complex intelligent driving scenarios. Summary of the Invention

[0006] Therefore, the technical problem to be solved by the present invention is to overcome the problems of the existing multimodal fusion strategy being simple and crude, having low fusion effectiveness, ignoring the semantic information of the behavior label text, resulting in insufficient ability to distinguish fine-grained similar driving behaviors, weak generalization performance, low driving behavior recognition accuracy, poor real-time performance, and weak practicality.

[0007] In a first aspect, to solve the above-mentioned technical problems, the present invention provides a multimodal driving behavior recognition method, comprising: S1. Obtain the training dataset; wherein, the training dataset includes image sequences of driver behavior inside the vehicle and driver behavior category labels corresponding to the image sequences; use the image sequences as the initial original image data; S2. Construct a driving behavior recognition model; wherein, the driving behavior recognition model includes intra-frame branching, inter-frame branching, and an attention mechanism module; S3. Input the initial original image data into the driving behavior recognition model, extract features from the initial original image data through the intra-frame branch to obtain spatial semantic features; extract temporal information from the initial original image data through the inter-frame branch to obtain temporal dynamic features; and extract features from the driver behavior category label through the attention mechanism module to obtain text features. S4. Using the text features as query vectors, dynamically aggregate the spatial semantic features and the temporal dynamic features to construct spatial view prototypes and temporal view prototypes respectively. S5. Using the spatial view prototype and the temporal view prototype, calculate the spatial view matching loss and the temporal view matching loss respectively; construct a joint loss function based on the spatial view matching loss, the temporal view matching loss and the cross-view alignment loss; train the driving behavior recognition model using the joint loss function to obtain the optimal driving behavior recognition model. S6. Input the in-vehicle driver behavior image sequence to be identified into the optimal driving behavior recognition model to obtain the target driving behavior classification result and complete the in-vehicle driver behavior recognition.

[0008] In one embodiment of the present invention, in step S3, the process of extracting spatial semantic features from the initial original image data through the intra-frame branch includes using the initial original image data as an input image; the intra-frame branch uses a visual transform network as a visual encoder to perform hierarchical iterative feature encoding on the input image; wherein, in the visual transform network, the first... The steps for the layer to perform feature encoding on the input image are as follows: For the The output features of the first layer are normalized, and a multi-head self-attention mechanism is used to globally correlate the normalized features with information, linking the output with the first layer. The output features of the layer are residually added and fused to obtain intermediate features; wherein, the mathematical expression of the intermediate features is: ; The intermediate features are then subjected to layer normalization again, and the normalized features are nonlinearly mapped using a feedforward network to obtain the transformed features. The transformed features are then fused with the intermediate features by residual addition to obtain the first... The spatial semantic features of the layer; wherein, the mathematical expression of the spatial semantic features is: ; in, Indicates the first Spatial semantic features output by the layer Indicates the first Spatial semantic features output by the layer Indicates the first Intermediate features of the layer output, This indicates a multi-head self-attention mechanism. Indicates a feedforward network. Representation layer normalization.

[0009] In one embodiment of the present invention, the step of extracting temporal information from the initial original image data through the inter-frame branch to obtain temporal dynamic features in step S3 is as follows: The inter-frame branch includes a motion guidance module and a time adaptation block; the motion guidance module extracts motion features from the initial original image data to obtain initial motion features; The initial motion features are fused with multi-layer spatial semantic features from the intra-frame branch using the time adaptation block to obtain temporal dynamic features.

[0010] In one embodiment of the present invention, the step of extracting motion features from the initial original image data through the motion guidance module to obtain initial motion features is as follows: The initial original image includes an original frame sequence, and a frame difference sequence is calculated based on the original frame sequence. The original frame sequence and the frame difference sequence are used as input sequences, respectively. Multi-scale depthwise separable convolution is used to extract features from the input sequences to obtain multiple multi-scale features. The expression for the multi-scale features is: ; Multiple multi-scale features are fused and convolved pointwise to obtain fused features; wherein the expression of the fused features is: ; The initial motion features are obtained by averaging the fused features of the original frame sequence and the fused features of the frame difference sequence; the expression of the initial motion features is: ; in, Indicates the first Multi-scale features, Represents the input sequence. Represents the original frame sequence. Represents the frame difference sequence. Indicates the initial motion characteristics. Indicates the first A multi-scale depthwise separable convolution function This represents pointwise convolution. Represents the feature concatenation function. This indicates the motion guidance module. This indicates average pooling.

[0011] In one embodiment of the present invention, the temporal adaptation block fuses the initial motion features with multi-layer spatial semantic features from the intra-frame branch; wherein the temporal adaptation block in the first... The calculation expression for each module is: ; ; in, Indicates the inter-frame branch number The temporal dynamic characteristics at each module Temporal convolution operations are represented as follows: express, This indicates a multi-head self-attention mechanism. Representation layer normalization, This indicates a downsampling operation. Indicates the first branch in the intra-frame Spatial semantic features extracted from layers.

[0012] In one embodiment of the present invention, the mathematical expressions for the spatial view prototype and the temporal view prototype are: ; ; ; in, Represents the query vector. This represents the spatial semantic features of the intra-frame branch output. Representing text features; , and Represents the learnable projection matrix; and These represent the key vector and the value vector, respectively. Represents the view prototype. Indicates the projection dimension. Represents the feature concatenation function. This represents the attention mechanism. Represents the normalization function. This represents the matrix transpose symbol.

[0013] In one embodiment of the present invention, step S5, which involves calculating the spatial view matching loss and the temporal view matching loss using the spatial view prototype and the temporal view prototype respectively, is as follows: The degree of matching between the query set and the support set on the spatial view prototype and the temporal view prototype is calculated using an ordered temporal alignment metric. The spatial view matching loss and the temporal view matching loss are calculated based on the matching degree.

[0014] In one embodiment of the present invention, the cross-view alignment loss is used to constrain the spatial semantic features, the temporal dynamic features, and the text features to align in the semantic space; wherein, the mathematical expression of the cross-view alignment loss is: ; in, Indicates cross-view alignment loss. Representing text features, This represents the spatial semantic features of the intra-frame branch output. Indicates the first The temporal dynamic characteristics at each module.

[0015] In one embodiment of the present invention, in step S5, the mathematical expression of the joint loss function is: ; in, Indicates joint loss, This represents the spatial view matching loss. This indicates the time-view matching loss. Indicates the balance hyperparameters, This indicates the cross-view alignment loss.

[0016] Secondly, to solve the above-mentioned technical problems, the present invention provides a multimodal driving behavior recognition system for implementing the above-mentioned multimodal driving behavior recognition method, comprising: An acquisition module is used to acquire a training dataset; wherein, the training dataset includes image sequences of driver behavior inside the vehicle and driver behavior category labels corresponding to the image sequences; the image sequences are used as initial original image data; A model building module is used to build a driving behavior recognition model; wherein, the driving behavior recognition model includes intra-frame branching, inter-frame branching, and attention mechanism modules; The dual-branch feature extraction module is used to input the initial original image data into the driving behavior recognition model, extract features from the initial original image data through the intra-frame branch to obtain spatial semantic features, extract temporal information from the initial original image data through the inter-frame branch to obtain temporal dynamic features, and extract features from the driver behavior category label through the attention mechanism module to obtain text features. The prototype building module is used to dynamically aggregate the spatial semantic features and the temporal dynamic features using the text features as query vectors, and to build spatial view prototypes and temporal view prototypes respectively. The training module is used to calculate the spatial view matching loss and the temporal view matching loss using the spatial view prototype and the temporal view prototype, respectively; to construct a joint loss function based on the spatial view matching loss, the temporal view matching loss and the cross-view alignment loss; and to train the driving behavior recognition model using the joint loss function to obtain the optimal driving behavior recognition model. The recognition output module is used to input the image sequence of the driver's behavior inside the vehicle to be recognized into the optimal driving behavior recognition model, output the target driving behavior classification result, and complete the recognition of the driver's behavior inside the vehicle.

[0017] Thirdly, to solve the above-mentioned technical problems, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned multimodal driving behavior recognition method.

[0018] Fourthly, to solve the above-mentioned technical problems, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned multimodal driving behavior recognition method.

[0019] Compared with the prior art, the above-described technical solution of the present invention has the following advantages: (1) The multimodal driving behavior recognition method, system, device, and storage medium described in this invention decouple spatial semantic feature extraction and temporal dynamic modeling into two independent branches for parallel execution. The dual-branch processing architecture includes an intra-frame branch with completely frozen parameters and an inter-frame branch with updatable parameters; by completely blocking the back propagation of gradients to the frozen backbone network, the memory usage during the training phase is significantly reduced while ensuring the model's strong feature representation capability, thus adapting to the lightweight fine-tuning requirements of vehicle terminal computing power and storage resources. By introducing a motion guidance module and a time adaptation block, accurate capture of the temporal dynamics of complex driving actions is achieved, making up for the shortcomings of traditional methods in dynamic feature modeling. By using text semantic guidance for prototype construction, the model's ability to distinguish differences in similar actions and fine-grained similar behaviors is significantly enhanced, greatly improving the recognition accuracy and robustness under complex lighting and occlusion conditions.

[0020] (2) The present invention adopts a multi-task joint training objective, and combines spatial view matching loss, temporal view matching loss and cross-view alignment loss calculated based on ordered temporal alignment metric. Only the lightweight inter-frame branches are fine-tuned. On the basis of achieving ordered alignment constraints of spatial and temporal view features, the training computing power and storage resource consumption are significantly reduced, which can be adapted to the deployment requirements of vehicle-mounted low computing power terminals. Attached Figure Description

[0021] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0022] Figure 1 This is a flowchart of a multimodal driving behavior recognition method according to a preferred embodiment of the present invention; Figure 2This is a schematic diagram of the dual-branch processing framework in a preferred embodiment of the present invention; Figure 3 This is a structural diagram of a multimodal driving behavior recognition system according to a preferred embodiment of the present invention. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0024] Example 1: Reference Figure 1 As shown, the present invention provides a multimodal driving behavior recognition method, including but not limited to the following steps: S1. Obtain the training dataset; the training dataset includes image sequences of driver behavior inside the vehicle and corresponding driver behavior category labels; use the image sequences as the initial original image data; S2. Construct a driving behavior recognition model; the driving behavior recognition model includes intra-frame branching, inter-frame branching, and attention mechanism modules; S3. Input the initial original image data into the driving behavior recognition model. Extract features from the initial original image data through intra-frame branch to obtain spatial semantic features; extract temporal information from the initial original image data through inter-frame branch to obtain temporal dynamic features; and extract features from the driver behavior category label through the attention mechanism module to obtain text features. S4. Using text features as query vectors, dynamically aggregate spatial semantic features and temporal dynamic features to construct spatial view prototypes and temporal view prototypes respectively. S5. Using the spatial view prototype and the temporal view prototype, calculate the spatial view matching loss and the temporal view matching loss respectively; construct a joint loss function based on the spatial view matching loss, the temporal view matching loss and the cross-view alignment loss; train the driving behavior recognition model using the joint loss function to obtain the optimal driving behavior recognition model. S6. Input the image sequence of the driver's behavior inside the vehicle to be identified into the optimal driving behavior recognition model to obtain the target driving behavior classification result and complete the driver's behavior recognition inside the vehicle.

[0025] This invention discloses a multimodal driving behavior recognition method that employs a dual-branch architecture to completely block gradient backpropagation to the frozen backbone network, reducing memory consumption during model training and enabling efficient fine-tuning on in-vehicle terminals with limited computing and storage resources. By introducing a motion guidance module and a time adaptation block, it achieves accurate capture of the temporal dynamics of complex driving actions, overcoming the shortcomings of traditional methods in dynamic feature modeling. Utilizing textual semantic guidance for prototype construction significantly enhances the model's ability to distinguish differences in similar actions and fine-grained similar behaviors, greatly improving recognition accuracy and robustness under complex lighting and occlusion conditions. Furthermore, this invention reduces training memory overhead, preserves feature details, and improves the accuracy and robustness of driving behavior recognition, making it particularly suitable for driver behavior recognition tasks that integrate multimodal data such as visual images and text labels in intelligent driving scenarios, effectively ensuring driving safety.

[0026] Specifically, refer to Figure 2 As shown, the driving behavior recognition model includes a dual-branch processing framework, comprising a parameter-frozen intra-frame branch and a trainable inter-frame branch, with the gradient backpropagation paths of the two branches being independent. In this embodiment, the intra-frame branch extracts spatial semantic features through a pre-trained visual encoder; the inter-frame branch processes the original image sequence and the corresponding frame difference sequence through a motion-guided module to achieve temporal dynamic feature extraction and fusion. Furthermore, based on a semantically guided attention mechanism, the text features output by the text encoder from the driver behavior label are used as queries to dynamically aggregate spatial and temporal features, generating spatial view prototypes and temporal view prototypes respectively; the corresponding spatial matching loss and temporal matching loss are calculated using the dual-view prototypes, and the cross-view alignment loss is fused to jointly supervise the training of the inter-frame branch, outputting the optimal driving behavior recognition model. The following describes each step in detail.

[0027] Specifically, in step S1, image sequences from onboard visible light sensors or infrared sensors are acquired to form the initial visual input. These image sequences are simultaneously captured by onboard cameras positioned at multiple locations, such as the dashboard, around the rearview mirrors, and the upper right corner of the windows, achieving complete multi-view coverage of the target area. The acquired content includes data on abnormal driver behaviors (such as using a mobile phone, making calls, adjusting the radio, drinking water, etc.) and normal driving behaviors. Each activity type can include both unobstructed and obstructed conditions (such as wearing a hat or sunglasses). Simultaneously, text labels for the driver behavior categories corresponding to the image sequences are acquired, and all data are labeled with the start and end times of the behavioral activities. During the model training phase, synthetic driver datasets (such as SYNDD1) or publicly available action datasets can also be used as core or auxiliary data sources.

[0028] Furthermore, the acquired image sequences undergo preprocessing, which includes keyframe filtering and center cropping. Specifically, image processing tools are used to load video or image sequences, perform temporal cropping and center cropping to extract key regions, and standardize the frame image resolution (e.g., uniformly adjust to 224×224 pixels). For interactions between the driver and objects inside the vehicle, continuous image sequences and corresponding frame difference sequences are extracted. The preprocessed images are then input into a parameter-frozen CLIP visual encoder (ViT-B / 32) for spatial semantic feature extraction.

[0029] Furthermore, the driver behavior category labels are labeled and aligned. The specific steps are as follows: the extracted image sequences are labeled and classified according to a preset format (e.g., "behavior category-volunteer number-time number-viewpoint number"), and the category text label corresponding to each image sequence is clearly defined (e.g., "using phone", "drinking", etc.). This ensures strict temporal and semantic alignment between the visual image sequences and their corresponding text labels, and constructs prompt words for subsequent dual-branch network processing and multimodal input data steps for prototype construction.

[0030] Specifically, in step S3, the intra-frame branch extracts spatial semantic features through a pre-trained visual encoder, the inter-frame branch processes the original image sequence and the corresponding frame difference sequence through a motion-guided module (also known as a motion primer module), and the specific implementation steps for obtaining text features based on a semantically guided attention mechanism are as follows: S310. Feature extraction is performed on the initial original image data through an intra-frame branch to obtain spatial semantic features. This process includes using the initial original image data as the input image; the intra-frame branch employs a VisionTransformer (ViT) network as a visual encoder to perform hierarchical iterative feature encoding on the input image. Specifically, for any input original image... The first in the visual transformation network The steps for the layer to perform feature encoding on the input image are as follows: S311, regarding the first The output features of the first layer are normalized, and a multi-head self-attention mechanism is used to globally correlate the normalized features with information, linking the output with the first layer. The output features of the layer are summed and fused using residuals to obtain intermediate features; the mathematical expression for the intermediate features is: ; in, Indicates the first Spatial semantic features output by the layer Indicates the first Intermediate features of the layer output, This indicates a multi-head self-attention mechanism. Representation layer normalization.

[0031] S312. Perform layer normalization on the intermediate features again, and then perform nonlinear mapping on the normalized features through a feedforward network to obtain the transformed features; then fuse the transformed features with the intermediate features by adding the residuals to obtain the first... The spatial semantic features of the layer; where the mathematical expression of the spatial semantic features is: ; in, Indicates the first Frame in Spatial semantic features output by the layer; This represents a feedforward network. In this embodiment of the invention, all parameters of this intra-frame branch are frozen during training.

[0032] Furthermore, spatial semantic features are performed using intra-frame branching, and the following exemplary steps are given: For example, a parameter-frozen CLIP visual encoder (ViT-B / 32 configuration) is used as the spatial feature extractor, with no adjustable parameters throughout the process, reusing pre-trained knowledge. The specific feature extraction process is as follows: Step 1: Video frame sampling processing. The input complete video sequence is uniformly sampled, with 8 key images extracted from each video segment, and all sampled frames are uniformly scaled to a standard size of 224×224 pixels.

[0033] Step 2: Image Patch Division. The preprocessed single-frame image is divided into blocks, uniformly divided into multiple non-overlapping image patches of size 32×32.

[0034] Step 3: Multi-scale Spatial Feature Encoding and Aggregation. The segmented image patches are input into the ViT base module of the CLIP model for feature encoding, outputting multi-scale spatial features across multiple network layers. To fully explore and enrich the spatial semantic cues of the image, spatial semantic features from layers 2, 4, 6, 8, 10, and 12 of the ViT network are selected for fusion and aggregation, ultimately generating global spatial semantic features. .

[0035] The core of this invention lies in a memory-efficient dual-branch feature extraction and semantic-guided fusion mechanism. The intra-frame branch employs a frozen CLIP visual encoder to extract spatial semantic features. All parameters are frozen during training to avoid the memory overhead of gradient backpropagation. The inter-frame branch is a lightweight trainable network that processes the raw RGB frame sequence through a motion-guided module. With frame difference sequence .

[0036] S320. Temporal information is extracted from the initial original image data through inter-frame branching to obtain temporal dynamic features. In this embodiment of the invention, the inter-frame branch contains 5.4M adjustable parameters, specifically used to capture inter-frame motion changes. The inter-frame branch includes a motion guidance module and a temporal adaptation block; wherein, the motion guidance module is constructed based on multi-scale depth-separable 3D convolution.

[0037] Furthermore, the inter-frame branch extraction process for temporal dynamic features includes: in terms of motion feature preprocessing, calculating the RGB difference sequence and average difference frame of adjacent frames, and inputting them together with the original frame sequence into the motion guidance module; in terms of motion feature extraction, extracting motion features through dual-branch multi-scale depth-separable 3D convolution, taking into account both spatial resolution and dynamic information capture; in terms of temporal dependency modeling, fusing motion features with multi-scale spatial features of intra-frame branches through a lightweight temporal adaptation block, and using temporal convolution and multi-head self-attention to model temporal dependencies, outputting temporal dynamic features (also known as temporal features). The specific implementation steps are as follows: S321. The specific steps for extracting motion features from the initial original image data using the motion guidance module to obtain the initial motion features are as follows: S3211. The initial original image includes the original frame sequence, and a frame difference sequence is calculated based on the original frame sequence. The original frame sequence and the frame difference sequence are used as input sequences respectively. Multi-scale depthwise separable convolution is used to extract features from the input sequences to obtain multiple multi-scale features. The expression for the multi-scale features is as follows: ; in, Indicates the first Multi-scale features, Represents the input sequence. Indicates the first A multi-scale depthwise separable convolution function. In this embodiment of the invention, multi-scale features are extracted through two parallel branches, wherein the kernel size of the multi-scale depthwise separable convolution is preferably... , .

[0038] S3212. Multiple multi-scale features are fused and convolved pointwise to obtain fused features; the expression for the fused features is: ; in, This represents pointwise convolution. Represents the feature concatenation function. This indicates the motion guidance module.

[0039] S3213. The fusion features of the original frame sequence and the fusion features of the frame difference sequence are averaged to obtain the initial motion features; the expression for the initial motion features is: ; in, Represents the original frame sequence. Represents the frame difference sequence. Indicates the initial motion characteristics. This indicates the motion guidance module. This indicates average pooling.

[0040] It should be noted that a video is composed of multiple frames; the original frame sequence refers to the image sequence composed of frame-by-frame images obtained after decomposing the video. The frame difference sequence is an image sequence composed of difference images obtained by performing difference operations on two consecutive frames using the inter-frame difference method.

[0041] S322. The initial motion features are fused with multi-layer spatial semantic features from intra-frame branches using a temporal adaptation block to obtain temporal dynamic features. The temporal adaptation block is located in the [missing information - likely a specific step or stage]. The calculation expression for each module is: ; ; in, Indicates the inter-frame branch number The temporal dynamic characteristics at each module express, express, This indicates a multi-head self-attention mechanism. Representation layer normalization, Indicates a downsampling operation; Indicating the first branch in the intra-frame ViT Spatial semantic features extracted from the ViT layer, i.e., from the ViT layer... Spatial semantic features extracted from layers.

[0042] Furthermore, in this embodiment of the invention, the multi-layer spatial semantic features are preferably the 2nd, 4th, 6th, 8th, 10th, and 12th layers of the visual encoder, and are input to the temporal adaptation block through sparse connection as multi-scale spatial priors.

[0043] S330. To address the intra-class discrepancy problem in few-shot learning, a semantically guided attention mechanism is introduced. Therefore, the attention mechanism module extracts features from the driver behavior category labels to obtain text features. Specifically, text features are obtained through a text encoder using the category labels of the support set samples. .

[0044] Furthermore, the mathematical expression for converting behavior category labels into text features using the CLIP text encoder is as follows: ; in, The text encoder representing CLIP. This refers to text prompts constructed using categories.

[0045] In step S330, the action category labels (such as "using phone" and "drinking") are converted into text semantic features using the CLIP text encoder. In step S4, these text features are then... As the query vector (Q), the intra-frame branch Inter-frame branching Features are dynamically aggregated as keys (K) and values ​​(V) through Transformer attention blocks. Simultaneously, spatial semantic features are based on intra-frame branches. Generate a spatial view prototype Temporal dynamic features based on inter-frame branching Generate a time view prototype Together, these two constitute the behavioral dual-view template of this method.

[0046] Specifically, in step S4, a dual-view prototype is constructed guided by textual features. For support set samples, a semantically enhanced prototype is generated through cross-modal attention computation. The mathematical expressions for constructing the spatial view prototype and the temporal view prototype are as follows: ; ; ; in, Represents the query vector. This represents the spatial semantic features of the intra-frame branch output. Representing text features; , and Represents the learnable projection matrix; and These represent the key vector and the value vector, respectively. This represents the view prototype, used for subsequent matching. Represents the projection dimension, i.e., the key vector. The dimension; Represents the feature concatenation function. This represents the attention mechanism. Represents the normalization function. This represents the matrix transpose symbol. This process is applied to spatial features. and temporal dynamic characteristics Generate a spatial view prototype and time view prototype .

[0047] Furthermore, in this embodiment of the invention, a dual-scene cross-attention calculation logic is adopted to adapt labeled support set samples and unlabeled query set samples respectively, ensuring that multimodal features can be accurately aligned in different scenarios, as detailed below: Scenario 1: Support set samples with labels. The support set samples contain explicit driving behavior labels, and textual semantic features can be used to enhance modality alignment. The mathematical expressions for constructing the spatial view prototype and the temporal view prototype are as follows: ; ; ; in, This represents a learnable projection matrix that shares keys and values. Represents cross-modal attention output features; This represents the two types of modal features to be fused.

[0048] Scenario 2: Unlabeled query set samples. The query set samples have no behavioral labels; modality alignment is achieved solely based on visual features. The mathematical expressions for constructing the spatial view prototype and the temporal view prototype are: ; ; ; ; in, Represents support vectors, , and Represents the learnable projection matrix; and These represent the key vector and the value vector, respectively.

[0049] Specifically, in step S5, a multi-task training objective is adopted to balance classification accuracy and semantic consistency. Specifically, spatial view prototypes and temporal view prototypes are used to calculate spatial view matching loss and temporal view matching loss, respectively. The specific steps are as follows: S510. Calculate the matching degree between the query set and the support set on the spatial view prototype and the temporal view prototype using an ordered temporal alignment metric. Specifically, calculate the matching loss for the spatial view prototype and the temporal view prototype respectively, compare the similarity between the query sample and the support set prototype using the temporal alignment similarity calculation method (OTAM metric), and then optimize the classification effect using cross-entropy loss.

[0050] S520. Calculate the spatial view matching loss based on the matching degree. Loss of matching with time view The specific mathematical expression is: ; ; in, This represents the summation operation; Indicates the authenticity of the behavior; Represents the natural logarithm function; Represents the classification probability of a spatial view; This represents the classification probability of the time view.

[0051] Specifically, cross-view alignment loss is used as an auxiliary loss to constrain spatial semantic features. Temporal dynamic characteristics Text features Alignment is performed in the semantic space to avoid feature shift. The mathematical expression for the cross-view alignment loss is: ; in, Indicates cross-view alignment loss. Representing text features, This represents the spatial semantic features of the intra-frame branch output. Indicates the first The temporal dynamic characteristics at each module This represents the L2 norm.

[0052] Furthermore, the joint loss function used to train the driving behavior recognition model balances classification accuracy and feature consistency by weighted summation of the main matching loss and the cross-view alignment loss. The specific mathematical expression is: ; in, This represents the joint loss, i.e., the total global loss value; This represents the spatial view matching loss. This indicates the time-view matching loss. Indicates cross-view alignment loss; The value represents the balancing hyperparameter, i.e., the weight coefficient of the cross-view alignment loss, ranging from 0.5 to 3.0, with optimal performance between 0.5 and 1.0. This training mechanism ensures that the model captures both spatial details and temporal dynamics with low memory consumption, improving recognition accuracy, robustness, and efficiency.

[0053] Furthermore, during the training of the driving behavior recognition model using the joint loss function, a uniform sparse sampling strategy is adopted to extract 8 frames from each driver behavior video for processing, and the resolution of the frame images is standardized to 224×224 pixels.

[0054] Furthermore, a transfer learning strategy is employed during model training. Models pre-trained on general behavior recognition datasets (such as visual encoders and text encoders) are selected as the foundation and then specialized for in-vehicle driver behavior recognition scenarios to address the scarcity of labeled data in in-vehicle environments. The specific steps for optimizing the driver behavior recognition model using transfer learning are as follows: First, pre-trained models that are highly relevant to the behavior recognition task are selected, and then adapted to the driving behavior recognition dataset through transfer learning.

[0055] Secondly, after completing the transfer adaptation, the training hyperparameters are configured, setting core parameters such as learning rate, training epochs, and batch size. Based on the driving behavior dataset, the transferred model is iteratively fine-tuned. During training, the model continuously learns and integrates features from the new dataset, dynamically updating its feature representation system to improve its recognition adaptability and feature extraction capabilities for various driving behaviors.

[0056] Finally, a knowledge distillation module is introduced to construct a teacher-student model architecture, using the trained high-precision model as the teacher model and a lightweight, small model as the student model. Through the knowledge distillation algorithm, the driving behavior feature knowledge and recognition logic learned by the teacher model are transferred to the student model. This significantly reduces the number of model parameters and computational overhead, achieving model lightweighting while preserving the original recognition accuracy to the maximum extent, ensuring stable driving behavior recognition performance.

[0057] Specifically, in step S6, a visible light and / or infrared image sequence of the driving scene to be identified is acquired, and this image sequence is input into the optimal driving behavior recognition model obtained in step S5 to obtain the target driving behavior classification result. The target driving behavior classification result can be used for subsequent driver fatigue driving, distracted driving behavior recognition, and safety warnings.

[0058] Specifically, when applying this model for reasoning and identification, the probability distribution of the target driving behavior category is output based on the similarity matching score between the query sample and the spatial view prototype and the temporal view prototype, thereby determining the specific driving behavior category (such as fatigued driving, distracted driving, etc.). Subsequently, a risk assessment value is calculated based on the identified target behavior category and its preset risk rules, and different levels of safety warnings are triggered when the threshold is exceeded.

[0059] Furthermore, in the inference stage of the driving behavior recognition model, a spatial and temporal dual-view similarity aggregation method is used to complete the driving behavior recognition prediction. The spatial view similarity score and temporal view similarity score between the query sample and the support set samples are calculated separately. The two types of similarity scores are then summed to obtain the final behavior recognition prediction score. The specific calculation formula is as follows: ; in, This represents the final behavior recognition prediction score. Indicates the spatial view similarity score. This represents the similarity score of the time view.

[0060] Furthermore, the abnormal behavior identification and risk level classification stage specifically includes: Behavior Classification: Based on the final score, the driver's behavior category is output, including safe driving, distracted driving, and fatigued driving; Risk assessment: Establish risk level standards and classify different abnormal behaviors into levels (such as low risk, medium risk, and high risk). Warning Trigger: Different levels of warning mechanisms are triggered based on the risk level, and the identification results are recorded for subsequent optimization.

[0061] Furthermore, embodiments of the present invention may selectively introduce a user feedback mechanism to collect driver feedback on the recognition results and update the trainable inter-frame branch parameters through incremental learning to continuously optimize the model's adaptability to specific driver behavior habits. This feedback iteration mechanism can continuously optimize model parameters using real user scenario data, effectively improving the overall accuracy of driving behavior recognition while optimizing model adaptability and improving the user experience.

[0062] Furthermore, to verify the performance of the method described in this invention in the public action recognition task, a comparative experiment was conducted. The specific experimental setup and result analysis are as follows: The core action recognition method of this invention is based on ME-FSAR (Motion-Enhanced Few-Shot Action Recognition, abbreviated as ME-FSAR). The performance of this method on publicly available action recognition datasets (such as the motion-sensitive dataset SSV2, the scene-related dataset UCF101 / Kinetics, etc.) has been experimentally verified, and the specific comparison results are shown in Table 1. Table 1 shows the performance comparison results of the recognition datasets. In this comparative experiment, the methods compared include: Matching Network (MatchingNet), Compound Memory Network (CMN++), Ordered Temporal Alignment Method (OTAM), Temporal-Relational Cross Transformers (TRX), Spatio-Temporal Relation Modeling (STRM), Motion-Enhanced Few-Shot Action Recognition with PlusFusion (ME-FSAR(plus)), and Motion-Enhanced Few-Shot Action Recognition with Max Fusion (ME-FSAR(max)). Since driving behavior falls under the sub-segment of action recognition tasks, the performance advantages of ME-FSAR can be transferred to driving behavior recognition.

[0063] Table 1:

[0064] Experimental results demonstrate that the method employed in this invention exhibits superior performance on publicly available action recognition datasets. On the SSV2-Small dataset, the 1-shot / 5-shot accuracies reach 60.0% / 64.5%, respectively; on UCF101, 94.8% / 97.5%; on Kinetics, 89.0% / 92.0%; and on HMDB-51, 72.7% / 82.0%. All these metrics outperform traditional ResNet-based methods, objectively validating the effectiveness of the multimodal driving behavior recognition model provided by this invention in behavior recognition tasks.

[0065] The core of the method described in this invention lies in constructing a memory-efficient dual-branch processing architecture, decoupling spatial semantic feature extraction and temporal dynamic modeling into two independent branches. This architecture includes an intra-frame branch with completely frozen parameters and a trainable inter-frame branch. By completely cutting off the backpropagation path of gradients to the frozen backbone network, it significantly reduces training memory consumption while maintaining strong representational capabilities. During processing, visible light or infrared image sequences of vehicles are acquired, and spatial semantic features are extracted using the intra-frame branch based on a visual transformation network. Simultaneously, the motion guidance module of the inter-frame branch processes the original image and frame difference sequence. The motion guidance module is constructed based on multi-scale depth-separable 3D convolution, which can balance spatial resolution and dynamic information capture, and fuses the initial temporal features with multi-layer spatial features from the intra-frame branch through a temporal adaptation block. To address the intra-class variance problem in low-sample scenarios, this invention introduces a semantically guided attention mechanism, transforming behavioral category labels into textual semantic features through a text encoder, and using these as queries to dynamically aggregate visual features, thereby constructing spatial and temporal view prototypes. During the training phase, the model employs a multi-task joint training objective, combining spatial view matching loss, temporal view matching loss, and cross-view alignment loss calculated based on ordered temporal alignment metrics, with only lightweight inter-frame branches fine-tuned. In implementation, this method uses a uniform sparse sampling strategy, standardizing each video segment to 8 frames with a resolution of 224 x 224 pixels. Experiments demonstrate that this method significantly outperforms traditional methods in accuracy on multiple public datasets. The method described in this invention ultimately outputs driving behavior classification results by calculating the similarity score between the query sample and the dual-view prototype, triggering a safety warning when the risk exceeds a threshold. It also supports the introduction of a user feedback mechanism for incremental model optimization, thereby achieving continuous adaptation to specific driver behavior habits.

[0066] Example 2: Based on the same inventive concept, this embodiment provides a multimodal driving behavior recognition system. The principle of solving the problem is similar to that of the multimodal driving behavior recognition method provided in Embodiment 1, and the repeated parts will not be described again.

[0067] Reference Figure 3 As shown, this embodiment provides a multimodal driving behavior recognition system for implementing the multimodal driving behavior recognition method described in Embodiment 1, including: The acquisition module is used to acquire the training dataset; the training dataset includes image sequences of driver behavior inside the vehicle and corresponding driver behavior category labels; the image sequences are used as the initial original image data; The model building module is used to build a driving behavior recognition model; the driving behavior recognition model includes intra-frame branching, inter-frame branching, and attention mechanism modules. The dual-branch feature extraction module is used to input the initial original image data into the driving behavior recognition model. It extracts features from the initial original image data through the intra-frame branch to obtain spatial semantic features; it extracts temporal information from the initial original image data through the inter-frame branch to obtain temporal dynamic features; and it extracts features from the driver behavior category labels through the attention mechanism module to obtain text features. The prototype building module is used to dynamically aggregate spatial semantic features and temporal dynamic features using text features as query vectors, and to build spatial view prototypes and temporal view prototypes respectively. The training module is used to calculate the spatial view matching loss and the temporal view matching loss using the spatial view prototype and the temporal view prototype, respectively; construct a joint loss function based on the spatial view matching loss, the temporal view matching loss and the cross-view alignment loss; and train the driving behavior recognition model using the joint loss function to obtain the optimal driving behavior recognition model. The recognition output module is used to input the image sequence of the driver's behavior inside the vehicle to be recognized into the optimal driving behavior recognition model, and output the target driving behavior classification result to complete the recognition of the driver's behavior inside the vehicle.

[0068] Specifically, the acquisition module is used to collect multimodal data, including three in-vehicle cameras, an audio acquisition device, and a vehicle status sensor. The three in-vehicle cameras are deployed on the dashboard, near the rearview mirror, and in the upper right corner of the window, respectively, to collect video data of driver behavior. The audio acquisition device is used to collect in-vehicle audio data. The vehicle status sensor is used to collect vehicle driving status data (such as speed, acceleration, steering angle, etc.).

[0069] Specifically, the multimodal driving behavior recognition system described in this embodiment further includes a data preprocessing module, which is connected to the acquisition module and is used to preprocess the collected multimodal data. The data preprocessing module includes a video preprocessing unit, an audio preprocessing unit, and a data synchronization unit; wherein, the video preprocessing unit is used for video format conversion, cropping, and annotation; the audio preprocessing unit is used for audio noise reduction, short-time analysis, and pre-emphasis processing; and the data synchronization unit is used to achieve time synchronization of video, audio, and driving status data.

[0070] Specifically, the dual-branch feature extraction module is connected to the data preprocessing module and is used to extract the spatial and temporal features of the visual modality. It includes an intra-frame feature extraction unit and an inter-frame feature extraction unit. The intra-frame feature extraction unit uses a frozen CLIP ViT encoder to extract spatial features; the inter-frame feature extraction unit extracts temporal dynamic features through a motion guidance module and a time adaptation block.

[0071] Specifically, the prototype building module is connected to the dual-branch feature extraction module to build action prototypes. The prototype building module includes a semantic feature extraction unit and an attention aggregation unit. The semantic feature extraction unit extracts the semantic features of action labels through the CLIP text encoder; the attention aggregation unit realizes the dynamic aggregation of semantic features and visual features through the Transformer attention block, and outputs a dual-view prototype.

[0072] Specifically, the multimodal driving behavior recognition system described in this embodiment further includes a multimodal fusion module, which is connected to the prototype construction module and the data preprocessing module, and is used to fuse multimodal features. The multimodal fusion module includes an attention weight calculation unit and a cross-modal fusion unit; wherein, the attention weight calculation unit is used to calculate the fusion weights of different modalities; the cross-modal fusion unit is used to achieve multimodal feature fusion through a linear layer, a fusion layer, and a feedforward network.

[0073] Specifically, the training module is connected to the multimodal fusion module for model training and optimization. The training module includes a loss function calculation unit and a fine-tuning optimization unit; the loss function calculation unit is used to calculate the main matching loss and cross-view alignment loss; the fine-tuning optimization unit is used to fine-tune and compress the model through transfer learning and knowledge distillation.

[0074] Specifically, the multimodal driving behavior recognition system described in this embodiment also includes a risk assessment module and a user feedback module. The recognition output module outputs driving behavior classification results based on similarity matching scores. The risk assessment module provides risk warnings based on the output results of the recognition output module; the user feedback module collects correction information for incremental model iteration to improve system adaptability.

[0075] Furthermore, the recognition output module is connected to the training module and includes a dual-view matching unit. The risk assessment module is connected to the recognition output module and includes a risk level assessment unit and an early warning unit. The dual-view matching unit calculates similarity and outputs the behavior category; the risk level assessment unit classifies the risk level according to the behavior category; and the early warning unit triggers corresponding warnings according to the risk level.

[0076] Furthermore, the user feedback module is connected to the risk assessment module to collect user feedback information and transmit the feedback data to the training module to achieve iterative optimization of the model.

[0077] Example 3: This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multimodal driving behavior recognition method described in Embodiment 1.

[0078] Example 4: This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a multimodal driving behavior recognition method as described in Embodiment 1.

[0079] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0081] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0082] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0083] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A multimodal driving behavior recognition method, characterized in that, include: S1. Obtain the training dataset; wherein, the training dataset includes image sequences of driver behavior inside the vehicle and driver behavior category labels corresponding to the image sequences; use the image sequences as the initial original image data; S2. Construct a driving behavior recognition model; wherein, the driving behavior recognition model includes intra-frame branching, inter-frame branching, and an attention mechanism module; S3. Input the initial original image data into the driving behavior recognition model. Extract features from the initial original image data using the intra-frame branch to obtain spatial semantic features; extract temporal information from the initial original image data using the inter-frame branch to obtain temporal dynamic features; extract features from the driver behavior category label using the attention mechanism module to obtain text features; wherein, obtaining the spatial semantic features includes using the initial original image data as the input image; the intra-frame branch uses a visual transformation network as a visual encoder to perform hierarchical iterative feature encoding on the input image; wherein, in the visual transformation network, the first... The steps for the layer to perform feature encoding on the input image are as follows: For the The output features of the first layer are normalized, and a multi-head self-attention mechanism is used to globally correlate the normalized features with information, linking the output with the first layer. The output features of the layer are residually added and fused to obtain intermediate features; wherein, the mathematical expression of the intermediate features is: ； The intermediate features are then subjected to layer normalization again, and the normalized features are nonlinearly mapped using a feedforward network to obtain the transformed features. The transformed features are then fused with the intermediate features by residual addition to obtain the first... The spatial semantic features of the layer; wherein, the mathematical expression of the spatial semantic features is: ； in, Indicates the first Spatial semantic features output by the layer Indicates the first Spatial semantic features output by the layer Indicates the first Intermediate features of the layer output, This indicates a multi-head self-attention mechanism. Indicates a feedforward network. Representation layer normalization; S4. Using the text features as query vectors, dynamically aggregate the spatial semantic features and the temporal dynamic features to construct spatial view prototypes and temporal view prototypes respectively. S5. Using the spatial view prototype and the temporal view prototype, calculate the spatial view matching loss and the temporal view matching loss respectively; construct a joint loss function based on the spatial view matching loss, the temporal view matching loss and the cross-view alignment loss; train the driving behavior recognition model using the joint loss function to obtain the optimal driving behavior recognition model. S6. Input the in-vehicle driver behavior image sequence to be identified into the optimal driving behavior recognition model to obtain the target driving behavior classification result and complete the in-vehicle driver behavior recognition.

2. The multimodal driving behavior recognition method according to claim 1, characterized in that, In step S3, the step of extracting temporal information from the initial original image data through the inter-frame branch to obtain temporal dynamic features is as follows: The inter-frame branch includes a motion guidance module and a time adaptation block; the motion guidance module extracts motion features from the initial original image data to obtain initial motion features; The initial motion features are fused with multi-layer spatial semantic features from the intra-frame branch using the time adaptation block to obtain temporal dynamic features.

3. The multimodal driving behavior recognition method according to claim 2, characterized in that, The steps for extracting motion features from the initial original image data using the motion guidance module to obtain the initial motion features are as follows: The initial original image includes an original frame sequence, and a frame difference sequence is calculated based on the original frame sequence. The original frame sequence and the frame difference sequence are used as input sequences, respectively. Multi-scale depthwise separable convolution is used to extract features from the input sequences to obtain multiple multi-scale features. The expression for the multi-scale features is: ； Multiple multi-scale features are fused and convolved pointwise to obtain fused features; wherein the expression of the fused features is: ； The initial motion features are obtained by averaging the fused features of the original frame sequence and the fused features of the frame difference sequence. The expression for the initial motion feature is: ； in, Indicates the first Multi-scale features, Represents the input sequence. Represents the original frame sequence. Represents the frame difference sequence. Indicates the initial motion characteristics. Indicates the first A multi-scale depthwise separable convolution function This represents pointwise convolution. Represents the feature concatenation function. This indicates the motion guidance module. This indicates average pooling.

4. The multimodal driving behavior recognition method according to claim 2, characterized in that, The temporal adaptation block fuses the initial motion features with multi-layer spatial semantic features from the intra-frame branch; The time adaptation block is in the first The calculation expression for each module is: ； ； in, Indicates the inter-frame branch number The temporal dynamic characteristics at each module Represents temporal convolution. Indicates batch normalization, This indicates a multi-head self-attention mechanism. Representation layer normalization, This indicates a downsampling operation. Indicates the first branch in the intra-frame Spatial semantic features extracted from layers.

5. The multimodal driving behavior recognition method according to claim 1, characterized in that, The mathematical expressions for the spatial view prototype and the temporal view prototype are: ； ； ； in, Represents the query vector. This represents the spatial semantic features of the intra-frame branch output. Representing text features; , and Represents the learnable projection matrix; and These represent the key vector and the value vector, respectively. Represents the view prototype. Indicates the projection dimension. Represents the feature concatenation function. This represents the attention mechanism. Represents the normalization function. This represents the matrix transpose symbol.

6. The multimodal driving behavior recognition method according to claim 1, characterized in that, In step S5, the steps for calculating the spatial view matching loss and the temporal view matching loss using the spatial view prototype and the temporal view prototype are as follows: The degree of matching between the query set and the support set on the spatial view prototype and the temporal view prototype is calculated using an ordered temporal alignment metric. The spatial view matching loss and the temporal view matching loss are calculated based on the matching degree.

7. The multimodal driving behavior recognition method according to claim 1, characterized in that, The cross-view alignment loss is used to constrain the spatial semantic features, the temporal dynamic features, and the text features to align in the semantic space; wherein, the mathematical expression of the cross-view alignment loss is: ； in, Indicates cross-view alignment loss. Representing text features, This represents the spatial semantic features of the intra-frame branch output. Indicates the first The temporal dynamic characteristics at each module.

8. The multimodal driving behavior recognition method according to claim 1, characterized in that, In step S5, the mathematical expression for the joint loss function is: ； in, Indicates joint loss, This represents the spatial view matching loss. This indicates the time-view matching loss. Indicates the balance hyperparameters, This indicates the cross-view alignment loss.

9. A multimodal driving behavior recognition system, used to implement the multimodal driving behavior recognition method according to any one of claims 1 to 8, characterized in that, include: An acquisition module is used to acquire a training dataset; wherein, the training dataset includes image sequences of driver behavior inside the vehicle and driver behavior category labels corresponding to the image sequences; the image sequences are used as initial original image data; A model building module is used to build a driving behavior recognition model; wherein, the driving behavior recognition model includes intra-frame branching, inter-frame branching, and attention mechanism modules; The dual-branch feature extraction module is used to input the initial original image data into the driving behavior recognition model, extract features from the initial original image data through the intra-frame branch to obtain spatial semantic features, extract temporal information from the initial original image data through the inter-frame branch to obtain temporal dynamic features, and extract features from the driver behavior category label through the attention mechanism module to obtain text features. The prototype building module is used to dynamically aggregate the spatial semantic features and the temporal dynamic features using the text features as query vectors, and to build spatial view prototypes and temporal view prototypes respectively. The training module is used to calculate the spatial view matching loss and the temporal view matching loss using the spatial view prototype and the temporal view prototype, respectively; to construct a joint loss function based on the spatial view matching loss, the temporal view matching loss and the cross-view alignment loss; and to train the driving behavior recognition model using the joint loss function to obtain the optimal driving behavior recognition model. The recognition output module is used to input the image sequence of the driver's behavior inside the vehicle to be recognized into the optimal driving behavior recognition model, output the target driving behavior classification result, and complete the recognition of the driver's behavior inside the vehicle.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a multimodal driving behavior recognition method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements a multimodal driving behavior recognition method as described in any one of claims 1-8.

Images