Application method based on RTDETR-DRLCS model in driver attention state detection

Abstract

The invention discloses an application method in driver attention state detection based on an RTDETR-DRLCS model, and the method comprises the steps: making a driver expression data set FKCN, carrying out the expansion and preprocessing of the data set, and dividing the data set; building an RTDETR-r18 basic model, initializing model parameters, and training the basic model by adopting a training set; the RTDETR-r18 basic model is improved, and an improved RTDETR-DRLCS model is obtained; the improved RTDETR-DRLCS model is verified, and the model index of the improved RTDETR-DRLCS model is compared with that of an RTDETR-r18 basic model; and evaluating the attention state of the driver according to the real-time detection result of the improved RTDETR-DRLCS model on the driver expression.

Description

Technical Field

[0001] The invention relates to an application method based on an RTDETR-DRLCS model in driver attention state detection. Background Art

[0002] Motor vehicles have an increasing impact on people's work and life. While bringing convenience to life, they also increase the risk of road traffic accidents. Studies have shown that most of the causes of traffic accidents are personal factors of the driver, and the number of traffic accidents such as serious injuries and deaths caused by negative emotions of the driver during driving tasks is also increasing year by year.

[0003] In addition, studies have shown that low levels of anger can disrupt attention and prevent drivers from focusing on traffic; sadness and depression can narrow the driver's attention span, leading to reduced attention and increased reaction time. Driver distraction can affect their control of the vehicle, thereby increasing the risk of traffic accidents. Therefore, it is of great significance to monitor the driver's emotional changes during driving to determine whether their attention is distracted. Summary of the invention

[0004] The present invention aims to solve the problems existing in the above-mentioned prior art and provides an application method based on the RTDETR-DRLCS model in driver attention state detection.

[0005] The technical solutions adopted in the present invention are:

[0006] A method for applying the RTDETR-DRLCS model in driver attention state detection includes the following steps:

[0007] S1: Create the driver expression dataset FKCN, expand and preprocess the dataset, and divide the dataset into training set, validation set, and test set;

[0008] S2: Build the RTDETR-r18 basic model, initialize the model parameters, and train the basic model using the training set;

[0009] S3: The RTDETR-r18 basic model is improved to obtain an improved RTDETR-DRLCS model, wherein the improvements are as follows:

[0010] The 3×3 ordinary convolutional layer of the residual module in the RTDETR-r18 base model is replaced by a 3×3 grouped convolutional layer and a 1×1 convolutional layer, and the split-fusion strategy is used to improve the 3×3 grouped convolutional layer;

[0011] Improve the traditional 2D sine / cosine position encoding method of the AIFI module in the RTDETR-r18 base model to a learnable position encoding method;

[0012] The DFusion_SC module is introduced into the CCFM module of the RTDETR-r18 base model, and the network architecture of the CCFM module is reconstructed to combine dynamic sampling with adaptive fusion;

[0013] S4: Validate the improved RTDETR-DRLCS model and compare the model indicators with the RTDETR-r18 base model;

[0014] S5: Evaluate the driver's attention state based on the real-time detection results of the driver's expression based on the improved RTDETR-DRLCS model.

[0015] Furthermore, the data set FKCN includes several images, which are divided into five categories according to expressions, namely: neutral, angry, happy, sad and surprised. Each type of expression is further classified into driving environment and non-driving environment, where the driving environment means that the user is in the driving position, and the non-driving environment means that the user is not in the driving position.

[0016] Furthermore, the split-merge strategy is:

[0017] The 3×3 grouped convolution layer is split into three sub-layers, namely: 3×3 grouped convolution layer, 1×1 convolution layer and BN normalization layer;

[0018] In the training phase: two convolution outputs are obtained by performing independent convolution operations through the 3×3 grouped convolution layer and the 1×1 convolution layer, and the two convolution outputs are added to the result after the BN normalization layer processing;

[0019] In the inference phase:

[0020] The convolution kernel and bias of the 1×1 convolution layer, the convolution kernel and bias of the 3×3 grouped convolution layer, and the BN normalization layer are fused into a 3×3 equivalent convolution kernel and bias, and returned. Then, a new convolution layer is created using the returned convolution kernel and bias.

[0021] Furthermore, in the inference stage, the formula for fusing the convolutional layer with the BN normalization layer is:

[0022] Conv(x)=W(x)+b

[0023] Where W is the convolution kernel, x is the input data, b is the bias term, and Conv(x) is the output of the convolution operation;

[0024]

[0025] Among them, BN(x) is the output of batch normalization, x is the input data, γ is the scaling factor of BN, mean is the mean of the batch data, var is the variance of the batch data, and β is the offset factor of BN;

[0026]

[0027] Among them, BN(Conv(x)) is the output after the convolution layer and normalization are fused;

[0028] The formula for creating a new convolution layer using the returned convolution kernel and bias is:

[0029]

[0030] BN(Conv(x))=Wfused(x)+bfused

[0031] Among them, W fused (x) Convolution weight after fusion, b fused The convolution bias after fusion, fused is the convolution parameter after the fusion of Conv and BN.

[0032] Furthermore, the traditional 2D sine / cosine position encoding method in the AIFI module is improved to a learnable position encoding method. The process of learnable position encoding is:

[0033] 1) Convert the input data into tensor form and obtain the number of channels, height and width of the tensor;

[0034] 2) Generate learnable position embeddings of tensors;

[0035] 3) Flatten the tensor and transpose it;

[0036] 4) Pass the processed tensor and position embedding.

[0037] Furthermore, the DFusion_SC module includes a DySample module and a Fusion_SC module. The Fusion_SC module is a fusion module that improves the Fusion module in the CCFM module based on stacked Conv3XC and SDI modules; the DySample module replaces the upsampling in the CCFM module.

[0038] The present invention has the following beneficial effects:

[0039] The present invention analyzes the driver's facial expression through an improved RTDETR-DRLCS model to evaluate his attention state. In order to solve the problem of lack of expression data in driving scenes, the present invention adopts a self-made data set FKCN and processes the data set. Secondly, the residual module in the backbone network is optimized to improve the extraction effect of the driver's expression features, while reducing the number of model parameters. Then the traditional 2D sine / cosine position encoding method in the AIFI module is improved to a learnable position encoding method to improve the model's adaptability to diverse driving environments. Finally, by introducing the DFusion_SC multi-module collaborative mechanism, the network architecture of the CCFM module is systematically reconstructed, dynamic sampling is combined with adaptive fusion, the extraction quality of the feature map is improved, and the overall performance of the CCFM module is optimized. Based on the above improvements, the model can capture and analyze the driver's expression features in real time and accurately, and then accurately reflect the driver's attention state. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of the present invention.

[0041] Figure 2 This is the algorithm structure diagram of the improved RTDETR-DRLCS model.

[0042] Figure 3 This is the backbone network structure diagram of the improved RTDETR-DRLCS model.

[0043] Figure 4 Flowchart for learnable position encoding.

[0044] Figure 5 This is the structural diagram of the improved CCFM module.

[0045] Figure 6 PR curve diagram of the prediction results. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with the accompanying drawings.

[0047] like Figure 1 As shown, the present invention provides an application method based on the RTDETR-DRLCS model in driver attention state detection, comprising the following steps:

[0048] S1: Create the driver expression dataset FKCN, expand and preprocess the dataset, and divide the dataset into training set, validation set, and test set;

[0049] S2: Build the RTDETR-r18 basic model, initialize the model parameters, and train the basic model using the training set;

[0050] S3: The RTDETR-r18 basic model is improved to obtain an improved RTDETR-DRLCS model, wherein the improvements are as follows:

[0051] The 3×3 ordinary convolutional layer of the residual module in the RTDETR-r18 base model is replaced by a 3×3 grouped convolutional layer and a 1×1 convolutional layer, and the split-fusion strategy is used to improve the 3×3 grouped convolutional layer;

[0052] Improve the traditional 2D sine / cosine position encoding method of the AIFI module in the RTDETR-r18 base model to a learnable position encoding method;

[0053] The DFusion_SC module is introduced into the CCFM module of the RTDETR-r18 base model, and the network architecture of the CCFM module is reconstructed to combine dynamic sampling with adaptive fusion;

[0054] S4: Validate the improved RTDETR-DRLCS model and compare the model indicators with the RTDETR-r18 base model;

[0055] S5: Evaluate the driver's attention state based on the real-time detection results of the driver's expression based on the improved RTDETR-DRLCS model.

[0056] The present invention analyzes the driver's facial expression through an improved RTDETR-DRLCS model to evaluate his attention state. In order to solve the problem of lack of expression data in driving scenes, the present invention adopts a self-made data set FKCN and processes the data set. Secondly, the residual module in the backbone network is optimized to improve the extraction effect of the driver's expression features, while reducing the number of model parameters. Then the traditional 2D sine / cosine position encoding method in the AIFI module is improved to a learnable position encoding method to improve the model's adaptability to diverse driving environments. Finally, by introducing the DFusion_SC multi-module collaborative mechanism, the network architecture of the CCFM module is systematically reconstructed, dynamic sampling is combined with adaptive fusion, the extraction quality of the feature map is improved, and the overall performance of the CCFM module is optimized. Based on the above improvements, the model can capture and analyze the driver's expression features in real time and accurately, and then accurately reflect the driver's attention state.

[0057] The present invention uses a self-made dataset FKCN, which is composed of some public datasets KMU-FED driving environment expression pictures and KDEF non-driving environment pictures as well as driving environment pictures on the Internet. The driving environment refers to the user being in the driving position, and the non-driving environment refers to the user being in the non-driving position.

[0058] KMU-FED contains six driving environment expressions: happy, angry, fearful, disgusted, surprised, and sad. 1 to 3 images of anger, happiness, sadness, and surprise in the KMU-FED dataset are selected to avoid the influence of continuous images on the results. The KDEF dataset contains images of side faces, half side faces, and front faces. The self-made process of the dataset FKCN is as follows:

[0059] First, we obtain facial expression pictures of drivers in different driving and non-driving environments. To address the problem of lack of facial expression pictures of drivers in driving environments and simple features caused by continuous images, we use a variety of data enhancement methods such as affine transformation, shearing, and translation to expand the dataset, and use the enhanced pictures to replace some continuous images in the dataset.

[0060] After the dataset is expanded, the facial expressions of drivers in the pictures are labeled using the LabelImg annotation tool to obtain the self-made dataset FKCN;

[0061] Before the data is input into the model, the images in the dataset are preprocessed by fixed-size scaling, and the image size is scaled to 640×640.

[0062] Table 1 summarizes the relationship between emotion, attention and driving behavior.

[0063]

[0064] Table 1

[0065] Figure 2-3 They are the algorithm architecture diagram and backbone network structure diagram of the improved RTDETR-DRLCS model respectively.

[0066] Optimization of the backbone network structure of the improved RTDETR-DRLCS model:

[0067] The 3×3 ordinary convolution layer of the residual module is replaced by a 3×3 grouped convolution layer and a 1×1 convolution layer, and the split-fusion strategy is used to improve the 3×3 grouped convolution layer.

[0068] Specifically:

[0069] 1) Initialize the number of input channels c1, the number of output channels c2, the convolution kernel size k, the convolution stride s, the padding size p, and the number of groups g for group convolution. By inheriting the parent class parameters, the number of groups g is greater than 1.

[0070] 2) Create a 3×3 grouped convolutional layer, a 1×1 convolutional layer, and an optional initialized batch normalization layer; set the activation function to SiLU.

[0071] 3) A fusion splitting strategy is adopted for the 3×3 group convolution layer, which is split into a 3×3 group convolution layer, a 1×1 convolution layer and a BN normalization layer.

[0072] During the training phase:

[0073] 4) If there is a BN normalization layer, batch normalization is performed on the input data x.

[0074] 5) The input data x passes through a 3×3 grouped convolutional layer and a 1×1 convolutional layer respectively to obtain two convolution outputs.

[0075] 6) Add the two convolution outputs to the result after the BN normalization layer processing.

[0076] In the inference phase:

[0077] 7) Obtain the equivalent convolution kernel and bias and create an equivalent convolution layer.

[0078] By calling the method, the convolution kernel and bias of the 1×1 convolution layer, the convolution kernel and bias of the 3×3 grouped convolution layer, and the BN normalization layer are fused into a 3×3 equivalent convolution kernel and bias, and returned. Then, a new convolution layer is created using the returned convolution kernel and bias.

[0079] 8) The output data after the new convolution layer is added to the output data after the 1×1 convolution layer to obtain the final output result.

[0080] In the inference stage, the formula for fusing the convolutional layer with the BN normalization layer is:

[0081] Conv(x)=W(x)+b

[0082] Where W is the convolution kernel, x is the input data, b is the bias term, and Conv(x) is the output of the convolution operation;

[0083]

[0084] Among them, BN(x) is the output of batch normalization, x is the input data, γ is the scaling factor of BN, mean is the mean of the batch data, var is the variance of the batch data, and β is the offset factor of BN;

[0085]

[0086] Among them, BN(Conv(x)) is the output after the convolution layer and normalization are fused;

[0087] The formula for creating a new convolution layer using the returned convolution kernel and bias is:

[0088]

[0089] BN(Conv(x))=Wfused(x)+bfused

[0090] Among them, W fused (x) Convolution weight after fusion, b fused The convolution bias after fusion, fused is the convolution parameter after the fusion of Conv and BN.

[0091] like Figure 4 The figure shows the flow chart of learnable position coding. The position coding in the AIFI module is 2D sine / cosine position coding. This fixed position coding is predefined and will not be adjusted according to data changes, resulting in insufficient flexibility of position coding. The fixed position coding calculation is as follows:

[0092] For position pos and dimension i:

[0093]

[0094] Where PE is the position code; pos is the position index; i is the index of the embedding dimension; d model is the embedding dimension for each position.

[0095] The improved learnable positional encoding can establish a dynamic learning mechanism, obtain input data, call the LearnedPositionalEncoding instance to flexibly store the position index, and generate a learnable position embedding. The input tensor is then processed to adapt to the use and calculation of the subsequent multi-head self-attention and feedforward network, and the processed tensor and position embedding are passed. Finally, the position encoding is adaptively adjusted and updated through the back-propagation mechanism, and the expression of position information is continuously optimized, so that the improved AIFI module can capture more complex spatial relationships, provide the model with richer and more detailed position information, and enhance the model's representation ability and flexibility.

[0096] The process of improved learnable position encoding is:

[0097] 1) Convert the input data into a tensor and obtain the number of channels, height, and width of the tensor.

[0098] 2) Generate learnable positional embeddings of tensors:

[0099] By calling the Learned Positional Encoding instance, setting the maximum number of positional embeddings and the embedding dimension, and using nn.Embedding to create an embedding layer to generate a learnable positional encoding; then using register_buffer to register a fixed buffer to store the position index; finally, using the embedding layer to generate the corresponding positional embedding.

[0100] 3) Flatten the tensor and transpose it:

[0101] According to the form of tensor, use flatten and permute functions to flatten and transpose the input tensor to convert it into a tensor of the same shape.

[0102] 4) Pass the processed tensor and position embedding.

[0103] By calling the parent class method, the processed tensor and the generated position embedding are transmitted, so that the attention weights between tensors are calculated in conjunction with the multi-head self-attention mechanism.

[0104] This improvement enables the AIFI module to dynamically adjust and update the position encoding based on the input data to capture more complex spatial relationships and provide the model with richer and more detailed position information.

[0105] Figure 5 The improved CCFM module structure diagram is shown in Figure 2. The DFusion_SC module is used to collaboratively modify the network structure of the CCFM module, so that the improved CCFM module combines dynamic sampling with adaptive fusion, and systematically optimizes the overall performance of the CCFM module.

[0106] Among them, the DFusion_SC module contains the DySample and Fusion_SC modules.

[0107] The DySample module in the DFusion_SC module is used to replace the upsampling in CCFM. This module can automatically adjust the sampling coordinates according to the resolution and sampling strategy of the feature map, calculate the offset through the convolution layer, and finally perform a bilinear interpolation sampling operation based on the initial position coordinates. This allows the upsampled feature map to more accurately reconstruct the spatial relationship of the original features, reduce the blur and distortion that may be introduced during the upsampling process, and provide the quality of the feature map for the subsequent feature fusion module.

[0108] The calculation process of the DySample module is as follows:

[0109] Calculate the offset:

[0110] ofset(x)=W ofset .x+b ofset

[0111]

[0112] Among them, P init is the initial position offset; x is the input feature map, with a shape of (B, C, H, W); standard coordinate grid:

[0113] coordsh =arange(H)+0.5

[0114] coords w =arange(W)+0.5 stacking and transforming dimensions:

[0115] coords=transpose(stack(coords w ,coords h ),(1,2)) calculate the normalized coordinates:

[0116]

[0117] The Fusion_SC module is a fusion module that improves the Fusion module in the CCFM module based on the stacked Conv3XC and SDI modules.

[0118] For the Fusion module in the CCFM module, the stacked Conv3XC modules are used to improve the feature extraction module in the Fusion module, so that the improved feature extraction module can effectively fuse feature information from different levels, thereby improving the feature expression ability of the model.

[0119] The feature tensors output by the improved feature extraction module and the 1×1 convolutional layer are size-aligned and input into the SDI module for fusion.

[0120] By improving the CCFM module, the feature fusion module in CCFM can effectively fuse multi-level feature information, enhance feature expression capabilities, and greatly improve the performance of the model in complex scenarios.

[0121] The stacked Conv3XC is connected sequentially, that is, the output of the first Conv3XC module is used as the input of the second Conv3XC module, and the output of the second Conv3XC module is used as the input of the third Conv3XC module.

[0122] The SDI module processes the input through GSConv for feature processing, so that the RTDETR-DRLCS model can better capture and reorganize feature information. Finally, the input feature tensors are fused by element-by-element multiplication, so that the final feature tensor contains richer information.

[0123] By improving the network structure of the CCFM module through multi-module collaboration, the blur and distortion that may be introduced in the upsampling process can be reduced, the quality of the feature map can be improved, and the model can be integrated with feature information at different levels to improve the feature expression ability of the model.

[0124] Figure 6- is the PR curve of the RTDETR-DRLCS model in the self-made data set FKCN, and Table 2 is the detection result diagram. The accuracy of the RTDETR-DRLCS model in driver expression detection reached 97.7%. Compared with the RTDETR-r18 benchmark model, the accuracy of the RTDETR-DRLCS model increased by 2.6%. Although the FPS of the model has decreased and the latency has increased, the detection speed can still exceed the standard of real-time detection. Combined with the theoretical research on the relationship between driver expression state and attention in the existing literature in Table 1, the extracted driver expressions are evaluated and analyzed. When the RTDETR-DRLCS model detects that the driver's expression is angry, sad, or surprised, the driver is in a distracted state; when the RTDETR-DRLCS model detects that the driver's expression is neutral or happy, the driver is in a focused state.

[0125] This further verifies the innovation and effectiveness of evaluating driver attention status through facial expression detection.

[0126]

[0127] Table 2

[0128] The above description is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements without departing from the principle of the present invention, and these improvements should also be regarded as within the protection scope of the present invention.

Claims

1. An application method based on the RTDETR-DRLCS model in driver attention state detection, characterized in that: The steps include: S1: Create the driver expression dataset FKCN, expand and preprocess the dataset, and divide the dataset into training set, validation set, and test set; S2: Build the RTDETR-r18 basic model, initialize the model parameters, and train the basic model using the training set; S3: The RTDETR-r18 basic model is improved to obtain an improved RTDETR-DRLCS model, wherein the improvements are as follows: The 3×3 ordinary convolutional layer of the residual module in the RTDETR-r18 base model is replaced by a 3×3 grouped convolutional layer and a 1×1 convolutional layer, and the split-fusion strategy is used to improve the 3×3 grouped convolutional layer; Improve the traditional 2D sine / cosine position encoding method of the AIFI module in the RTDETR-r18 base model to a learnable position encoding method; The DFusion_SC module is introduced into the CCFM module of the RTDETR-r18 base model, and the network architecture of the CCFM module is reconstructed to combine dynamic sampling with adaptive fusion; S4: Validate the improved RTDETR-DRLCS model and compare the model indicators with the RTDETR-r18 base model; S5: Evaluate the driver's attention state based on the real-time detection results of the driver's expression based on the improved RTDETR-DRLCS model.

2. The method for applying the RTDETR-DRLCS model in driver attention state detection according to claim 1, characterized in that: The dataset FKCN includes several images, which are divided into five categories according to facial expressions, namely: neutral, angry, happy, sad and surprised. Each type of facial expression is further classified into driving environment and non-driving environment. The driving environment refers to the user being in the driving position, and the non-driving environment refers to the user being in the non-driving position.

3. The method for applying the RTDETR-DRLCS model in driver attention state detection according to claim 1, characterized in that: The split-fusion strategy is: The 3×3 grouped convolution layer is split into three sub-layers, namely: 3×3 grouped convolution layer, 1×1 convolution layer and BN normalization layer; In the training phase: two convolution outputs are obtained by performing independent convolution operations through the 3×3 grouped convolution layer and the 1×1 convolution layer, and the two convolution outputs are added to the result after the BN normalization layer processing; In the inference phase: The convolution kernel and bias of the 1×1 convolution layer, the convolution kernel and bias of the 3×3 grouped convolution layer, and the BN normalization layer are fused into a 3×3 equivalent convolution kernel and bias, and returned. Then, a new convolution layer is created using the returned convolution kernel and bias.

4. The method for applying the RTDETR-DRLCS model in driver attention state detection as claimed in claim 3, characterized in that: In the inference stage, the formula for fusing the convolutional layer with the BN normalization layer is: Conv(x)=W(x)+b Where W is the convolution kernel, x is the input data, b is the bias term, and Conv(x) is the output of the convolution operation; Among them, BN(x) is the output of batch normalization, x is the input data, γ is the scaling factor of BN, mean is the mean of the batch data, var is the variance of the batch data, and β is the offset factor of BN; Among them, BN(Conv(x)) is the output after the convolution layer and normalization are fused; The formula for creating a new convolution layer using the returned convolution kernel and bias is: BN(Conv(x))=Wfused(x)+bfused Among them, W fused (x) Convolution weight after fusion, b fused The convolution bias after fusion, fused is the convolution parameter after the fusion of Conv and BN.

5. The method for applying the RTDETR-DRLCS model in driver attention state detection according to claim 1, characterized in that: The traditional 2D sine / cosine position encoding method in the AIFI module is improved to a learnable position encoding method. The process of learnable position encoding is as follows: 1) Convert the input data into tensor form and obtain the number of channels, height and width of the tensor; 2) Generate learnable position embeddings of tensors; 3) Flatten the tensor and transpose it; 4) Pass the processed tensor and position embedding.

6. The method for applying the RTDETR-DRLCS model in driver attention state detection according to claim 1, characterized in that: The DFusion_SC module includes a DySample module and a Fusion_SC module. The Fusion_SC module is a fusion module that improves the Fusion module in the CCFM module based on stacked Conv3XC and SDI modules; the DySample module replaces the upsampling in the CCFM module.

Images