A method and device for identifying driver distraction behavior in a smart cockpit

Abstract

This invention discloses a method and apparatus for driver distraction behavior recognition in smart cockpits. The method includes: constructing a first model; inputting an initial distraction behavior image into the first model and extracting driver posture recognition results from the initial distraction behavior image; inputting the initial distraction behavior image into a second model for training and extracting key object recognition results from the initial distraction behavior image; and calculating distraction behavior recognition results based on the driver posture recognition results and key object recognition results. This invention enhances the accurate localization and feature extraction of key information in shallow semantics of images by embedding a CA attention module, and also improves the accuracy of the classification network while maintaining a lightweight network model. Furthermore, this invention fuses the recognition results of posture and object, reducing the probability of misclassifying similar distracted driving behaviors and improving the model's accuracy and generalization. It can be widely applied in the field of automotive driving technology.

Description

Technical Field

[0001] This invention relates to the field of automotive driving technology, and in particular to a method and device for recognizing driver distraction behavior in smart cockpits. Background Technology

[0002] A 2019 statistical report from a certain country's traffic safety administration indicated that traffic accidents caused by distracted driving occurred frequently, resulting in 3,142 deaths and an increase of approximately 1.1% in injuries compared to 2018. In recent years, the number of motor vehicle drivers has surged, leading to a growing number of traffic safety problems. More than 90% of major traffic accidents are related to commercial vehicles, and these accidents are often highly serious. Therefore, the driving status of commercial vehicle drivers requires corresponding attention. Furthermore, driver behavior recognition is a crucial component of interactive intelligent cockpit systems. Thus, effectively recognizing the posture of commercial vehicle drivers and accurately and promptly classifying distracted driving behaviors are essential for achieving human-machine interaction in intelligent cockpit systems for commercial vehicles, effectively improving driving safety.

[0003] Existing smart cockpit systems utilize multimodal interaction methods including voice, touch, gesture, vision, and facial expression recognition. Their primary aim is to meet driver needs, "understand" the driver's requirements, and enhance the driver's enjoyment through proactive responses. However, for drivers of commercial vehicles, while ensuring a positive driving experience, greater emphasis should be placed on driving safety. Therefore, introducing driver distraction detection based on visual posture into the smart cockpit can further improve safety.

[0004] Current research on driver distraction detection falls into three categories: data based on natural driving behavior of vehicles, data based on driver posture images from vision, and data based on driver physiological parameters from wearable devices. In research on distracted driving, there is a lack of extensive image data of commercial vehicle drivers in natural driving states. This results in existing visual posture algorithms being unable to effectively detect distracted behavior in commercial vehicle drivers, exhibiting low accuracy and poor generalization, thus failing to meet the needs of widespread applications.

[0005] Disadvantages of existing technology:

[0006] 1. Visual pose estimation methods are widely used due to their non-invasive and real-time monitoring capabilities. For acquired driver video images, one approach involves a two-stage algorithm: first, acquiring the driver's skeletal joint diagram and corresponding coordinate data, then inputting this data into a deep learning network to extract motion features for classification. However, the accuracy of this type of algorithm is easily affected by the pose estimation algorithm in the first stage, resulting in poor real-time performance.

[0007] 2. One approach is to use an end-to-end algorithm to directly extract image features through a deep learning network and output classification results. While this type of algorithm has the advantages of low computational complexity and good real-time performance, it is prone to ignoring key features in the image, resulting in poor generalization. Summary of the Invention

[0008] In view of this, embodiments of the present invention provide a highly accurate method for recognizing driver distraction behavior in smart cockpits.

[0009] On one hand, embodiments of the present invention provide a method for recognizing driver distraction behavior in smart cockpits, including:

[0010] Construct the first model;

[0011] Input the initial distraction behavior image into the first model, and extract the driver posture recognition result from the initial distraction behavior image;

[0012] The initial distraction behavior image is input into the second model for training, and the key object recognition results in the initial distraction behavior image are extracted.

[0013] Based on the driver posture recognition result and the key object recognition result, the distraction behavior recognition result is calculated.

[0014] Optionally, the step of constructing the first model includes:

[0015] Construct the first convolutional layer, the second convolutional layer, and the fully connected layer;

[0016] Construct a CA attention module, which includes two average pooling layers and two 1×1 convolutional layers;

[0017] Construct an inverse residual module, which includes two 1×1 convolutional layers and one 3×3 depth-separable convolutional layer;

[0018] A feature extraction module is constructed, which includes two 3×3 convolutional layers, two 1×1 convolutional layers, and several transformer models;

[0019] A first model is constructed based on the first convolutional layer, the second convolutional layer, the fully connected layer, the CA attention module, the inverted residual module, and the feature extraction module.

[0020] Optionally, the step of inputting the initial distracted behavior image into the first model and extracting the driver posture recognition result from the initial distracted behavior image includes:

[0021] The initial distraction behavior image is downsampled by the first convolutional layer to obtain a first feature map;

[0022] The first feature map is encoded along the horizontal and vertical directions by the CA attention module to obtain the second feature map;

[0023] The second feature map is downsampled using the inverted residual module to obtain the third feature map;

[0024] The feature extraction module performs feature extraction and feature fusion on the third feature map to obtain the fourth feature map;

[0025] The second convolutional layer adjusts the number of output channels of the fourth feature map to the number of channels of the initial distraction behavior image;

[0026] The driver's posture recognition result in the initial distraction behavior image is obtained by predicting and classifying the fourth feature map through the fully connected layer.

[0027] Optionally, the step of encoding the first feature map along the horizontal and vertical directions using the CA attention module to obtain the second feature map includes:

[0028] The first feature map is average pooled along the horizontal and vertical directions to obtain feature information in the horizontal and vertical directions.

[0029] Based on the feature information of each channel in the horizontal and vertical directions, we obtain the horizontal feature set and the vertical feature set;

[0030] The horizontal feature set and the vertical feature set are concatenated and convolved to obtain an intermediate feature map.

[0031] After normalization and nonlinear operations, the intermediate feature maps are used to obtain horizontally independent feature maps and vertically independent feature maps.

[0032] The horizontal independent feature map and the vertical independent feature map are passed through the second convolutional layer and the sigmoid activation function to obtain the horizontal attention weight and the vertical attention weight;

[0033] The first feature map is weighted based on the horizontal attention weight and the vertical attention weight to obtain the second feature map.

[0034] Optionally, in the step of inputting the initial distraction behavior image into the second model for training and extracting the key object recognition results in the initial distraction behavior image, the second model is a YOLOv5 model, which includes an input module, a backbone module and an output module.

[0035] The input module performs data augmentation on the initial distraction behavior image using the Mosaic data augmentation algorithm, and then adjusts the image size to the target size to obtain a standard image;

[0036] The standard is sliced ​​using the backbone module, and then spliced ​​and convolved to obtain the initial feature map.

[0037] The initial feature map is classified and predicted by the output module to obtain the key object recognition result.

[0038] Optionally, the step of calculating the distraction behavior recognition result based on the driver posture recognition result and the key object recognition result includes:

[0039] The probability matrices for the two distraction behaviors of using a mobile phone and drinking water in the driver posture recognition results are concatenated with the probability matrices for the key object recognition results to obtain the target recognition probability matrix.

[0040] The target recognition probability matrix is ​​fused through calculation to obtain the distraction behavior recognition result.

[0041] Optionally, in the step of calculating the distraction behavior recognition result based on the driver posture recognition result and the key object recognition result using the minimum value rule, the driver posture recognition result includes safe driving behavior, using a mobile phone, drinking water, talking to the passenger and tidying up one's appearance; the key object recognition result includes communication tools and drinking tools.

[0042] On the other hand, embodiments of the present invention also provide a driver distraction behavior recognition device for smart cockpits, comprising:

[0043] The first module is used to build the first model;

[0044] The second module is used to input the initial distraction behavior image into the first model and extract the driver posture recognition result from the initial distraction behavior image.

[0045] The third module is used to input the initial distraction behavior image into the second model for training and extract the key object recognition results in the initial distraction behavior image.

[0046] The fourth module is used to calculate the distraction behavior recognition result based on the driver posture recognition result and the key object recognition result.

[0047] On the other hand, embodiments of the present invention also provide an electronic device, including a processor and a memory; the memory is used to store a program; the processor executes the program to implement the aforementioned driver distraction behavior recognition method for smart cockpits.

[0048] On the other hand, embodiments of the present invention also provide a computer-readable storage medium storing a program that is executed by a processor to implement the aforementioned driver distraction behavior recognition method for smart cockpits.

[0049] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0050] The embodiments of the present invention include at least the following beneficial results: By constructing a first model, the embodiments of the present invention enhance the accurate localization and feature extraction of key information in the shallow semantics of images, and can also improve the accuracy of the classification network while ensuring the lightweight network model; by fusing the pose recognition result identified by the first model with the object recognition result extracted by the second model to obtain the final recognition result, the embodiments of the present invention reduce the probability of similar distracted driving behaviors being misclassified, and improve the accuracy and generalization of the model. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 A flowchart illustrating the steps of a driver distraction behavior recognition method for smart cockpits, provided by an embodiment of the present invention;

[0053] Figure 2 A structural diagram of the first model provided in an embodiment of the present invention;

[0054] Figure 3 A flowchart illustrating a driver distraction behavior recognition method for smart cockpits, provided as an embodiment of the present invention;

[0055] Figure 4 This is a block diagram of a driver distraction behavior recognition device for smart cockpits, provided as an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] On one hand, embodiments of the present invention provide a method for recognizing driver distraction behavior in smart cockpits, referring to... Figure 1 The method specifically includes steps S100-S400.

[0058] S100: Construct the first model.

[0059] Specifically, the first model is a Mobi leViT-CA model based on an attention mechanism, formed by embedding the CA attention module between the first convolutional layer and the inverse residual module of the Mobi leViT model. This model is used to extract the driver's pose features from the initial distracted behavior image and enhance local attention to key object information for driving behavior classification. (Refer to...) Figure 2 This is a structural diagram of the Mobi leViT-CA model.

[0060] Optionally, a first convolutional layer, a second convolutional layer, and a fully connected layer are constructed; the first convolutional layer is 3×3 in size, the second convolutional layer is 1×1 in size, and the fully connected layer is 1×1 in size; a CA attention module is constructed, which includes two average pooling layers and two 1×1 convolutional layers; an inverse residual module (Mobi leV2) is constructed, which includes two 1×1 convolutional layers and one 3×3 depthwise separable convolutional layer, and the first model contains 7 inverse residual modules; a feature extraction module (Mobi leViT block) is constructed, which includes two 3×3 convolutional layers, two 1×1 convolutional layers, and several transformer models, and the first model contains 3 feature extraction modules. (Refer to...) Figure 2 Here is a structural diagram of the Mobi leViT-CA model, by Figure 2 It can be seen that there are 7 inverted residual modules (Mobi leV2, Figure 2 The model can be divided into five modules: layer 1 (represented as MV2) and layer 5 (represented as layer 2). In layer 3, L=2, indicating that the feature extraction module has two transformer models. In layer 4, L=4, indicating that the feature extraction module has four transformer models. In layer 5, L=3, indicating that the feature extraction module has three transformer models.

[0061] S200: Input the initial distraction behavior image into the first model and extract the driver posture recognition result from the initial distraction behavior image.

[0062] Specifically, firstly, an initial image of distracting behaviors is acquired. The distracting behaviors in this invention are using a mobile phone, drinking water, talking to the passenger, and grooming. The driver posture recognition results include using a mobile phone, drinking water, talking to the passenger, grooming, and safe driving behaviors. Step S200 specifically includes steps S210-S260.

[0063] S210: The initial distraction behavior image is downsampled through the first convolutional layer to obtain a first feature map.

[0064] Specifically, the size of the first convolutional layer is 3×3. The initial image of the distracted behavior is input into the first model, and it is first downsampled through the first convolutional layer to obtain the first feature map;

[0065] S220: The first feature map is encoded along the horizontal and vertical directions by the CA attention module to obtain the second feature map.

[0066] Specifically, CA attention is a mobile network attention mechanism that decomposes channel attention into two 1D feature encoding processes, aggregating features along two spatial directions respectively. This allows for the capture of long-range dependencies along one spatial direction while preserving precise location information along the other. The generated feature maps are then encoded into a pair of direction-aware and location-sensitive attention maps, which can be complementaryly applied to the input feature map to enhance the representation of the object of interest. The input size of the CA attention module is the same as the output size of the first convolutional layer, i.e., 16×16. After passing through the CA attention module, the first feature map yields a second feature map of the same size with stronger feature representation, enhancing the representation and attention of key information in the shallow semantics of driving behavior images. Step S220 specifically includes steps S221-S226.

[0067] S221: Perform average pooling on the first feature map along the horizontal and vertical directions to obtain feature information in the horizontal and vertical directions.

[0068] Specifically, the first feature map X∈R c×H×W Average pooling is performed along the horizontal and vertical directions respectively to obtain the feature information of each channel in the horizontal and vertical directions. C is the number of channels, H is the height of the channel, and W is the width of the channel.

[0069] S222: Based on the feature information of each channel in the horizontal and vertical directions, obtain the horizontal feature set and the vertical feature set.

[0070] Specifically, the expressions for the horizontal feature set and the vertical feature set obtained based on the feature information in the horizontal and vertical directions are as follows:

[0071]

[0072]

[0073] Where C is the number of channels, H is the height of a channel, and W is the width of a channel. Let be the horizontal feature set, representing the set of local features along the horizontal direction of the c-th channel with height h; Let x be the vertical feature set, representing the set of local features along the vertical direction of the c-th channel with width w. c (h,i) represents a feature with height h and width i in the c-th horizontal channel, x c (j,w) represents a feature with height j and width w in the c-th channel in the vertical direction.

[0074] S223: Concatenate and convolve the horizontal feature set and the vertical feature set to obtain an intermediate feature map.

[0075] Specifically, the horizontal and vertical feature sets are concatenated and transformed using a convolution function to obtain an intermediate feature map with spatial information. The expression for the intermediate feature map is as follows:

[0076] f=δ(F1([z h ,z w ]))

[0077] Where f represents the intermediate feature map, δ() is the non-linear activation function, F1() is the shared 1×1 convolution transformation function, and z h For the set of horizontal features, z w It is a set of vertical features.

[0078] S224: After normalizing and performing nonlinear operations on the intermediate feature map, horizontal independent feature maps and vertical independent feature maps are obtained.

[0079] Specifically, the intermediate feature map is normalized and subjected to nonlinear operations to obtain the horizontally independent feature map f. h and vertically independent feature maps f w Among them, the horizontally independent feature map f h ∈R C / r ×H Vertically independent feature map f w ∈R C / r ×W Where C is the number of channels, r is the reduction ratio of the control block size, and W is the width of the channel.

[0080] S225: Pass the horizontal independent feature map and the vertical independent feature map through the second convolutional layer and the sigmoid activation function to obtain the horizontal attention weight and the vertical attention weight.

[0081] Specifically, the horizontal independent feature map and the vertical independent feature map are passed through the second convolutional layer and the sigmoid activation function to obtain the horizontal attention weight g. h and vertical attention weight g w .

[0082] S226: The first feature map is weighted based on the horizontal attention weight and the vertical attention weight to obtain the second feature map.

[0083] Specifically, the formula for the weighted calculation is as follows:

[0084]

[0085] in, The horizontal attention weights on the c-th channel are... Let y be the vertical attention weight on the c-th channel. c (i,j) represents the attention output on the c-th channel, x c (i,j) represents the features of the first feature map input on the c-th channel.

[0086] S230: The second feature map is downsampled by the inverted residual module to obtain the third feature map.

[0087] Specifically, the inverted residual module is the Mobi leV2 inverted residual module, which downsamples the second feature map to obtain the third feature map.

[0088] S240: The feature extraction module performs feature extraction and feature fusion on the third feature map to obtain the fourth feature map.

[0089] Specifically, the feature extraction module is a Mobi leViT module, which performs local feature extraction, global feature extraction, and feature fusion on the feature map. In the local feature extraction part, a 3×3 convolutional kernel is used to complete the local feature extraction task. Then, a 1×1 convolutional kernel is used to adjust the number of its output channels to the number of input channels required by the global feature extraction module. In the feature fusion part, a 1×1 convolutional kernel is used to restore the number of its output channels. The restored feature map is then concatenated with the initial input feature map using a skip connection method. Finally, a 3×3 convolutional kernel is used to complete the feature fusion task. The feature extraction module performs feature extraction and feature fusion on the third feature map to obtain the fourth feature map.

[0090] S250: The number of output channels of the fourth feature map is adjusted to the number of channels of the initial distraction behavior image through the second convolutional layer.

[0091] Specifically, the second convolutional layer has a size of 1×1, and the number of output channels of the fourth feature map is adjusted to the number of channels of the initial distraction behavior image through the second convolutional layer.

[0092] S260: The fourth feature map is predicted and classified through the fully connected layer to obtain the driver posture recognition result in the initial distraction behavior image.

[0093] Specifically, the fully connected layer outputs 5 neurons, corresponding to the 5 categories of driver behavior.

[0094] S300: Input the initial distraction behavior image into the second model for training, and extract the key object recognition results in the initial distraction behavior image.

[0095] Specifically, the second model is a YOLOv5 model, including an input module, a backbone module, and an output module. The YOLOv5 model identifies objects held by the driver, determining the object's category information. Key object recognition results include communication tools and drinking utensils; communication tools include mobile phones, and drinking utensils include water cups and bottles. Step S300 includes steps S310-S330.

[0096] S310: The input module performs data augmentation on the initial distraction behavior image using the Mosaic data augmentation algorithm, and then adjusts the image size to the target size to obtain a standard image.

[0097] Specifically, the Mosaic data augmentation algorithm randomly crops four images and then stitches them together into one image as training data. That is, it reads four images each time, and then performs operations such as flipping, scaling, and color gamut adjustment on these four images to generate new image data. The size of the new image is then adjusted to the target size, which is 608×608×3, to obtain a standard image.

[0098] S320: The standard is sliced ​​using the backbone module, and then spliced ​​and convolved to obtain the initial feature map.

[0099] Specifically, the attention structure in the backbone module completes the 32x downsampling of the standard image. First, a slicing operation is performed, taking a value for every other pixel. Similar to neighbor downsampling, the original image data is divided into 4 parts. Then, these 4 parts are concatenated along the channel dimension. Finally, the concatenated feature map is convolved to obtain the initial feature map. The attention structure consists of 4 convolutional layers.

[0100] S330: The initial feature map is classified and predicted by the output module to obtain the key object recognition result.

[0101] Specifically, the output module performs object category prediction on the relevant features extracted from the initial feature map to obtain the key object recognition result; the GIOU (Generalized Intersection over Union) calculation result used for prediction is used as the loss value for the target bounding box regression fitting, and the prediction category is set to 2 categories, namely communication tools and drinking tools. In one embodiment of the present invention, YOLOv5s is selected as the pre-training weight, the iteration batch size is set to 16, and the total number of iterations is 150.

[0102] S400: Based on the driver posture recognition result and the key object recognition result, the distraction behavior recognition result is calculated.

[0103] Specifically, during the decision fusion process, if YOLOv5 detects a key object in the image—a communication tool—it determines that the distracted driving behavior is using a mobile phone; if it detects a key object in the image—a drinking tool—it determines that the distracted driving behavior is drinking water. Step S400 specifically includes steps S410-S420:

[0104] S410: Concatenate the recognition probability matrices of the two distraction behaviors of using a mobile phone and drinking water in the driver posture recognition results with the recognition probability matrix of the key object recognition results to obtain the target recognition probability matrix.

[0105] Specifically, the recognition results output by the first model are represented by the set m1 = {m1(n1), m1(n2), ..., m1(n...}. c )} represents, where, ·1(n c Let be the recognition rate of the c-th behavior in classifier 1, where c is the type of behavior and n represents the current test sample number. The recognition results output by the second model are represented by the set m2 = {m2(n1), m2(n2), ..., m2(n...}. c )} represents, where, m2(n c Let be the recognition rate of the c-th behavior in classifier 2, where c is the type of behavior and n represents the current test sample number. The probability matrices of the first model for recognizing distracting behaviors like using a mobile phone and drinking water are concatenated with the probability matrix of the second model for recognizing key objects, using the set {Value_P}. ij (n), i=1,2...k,j=1,2,...c} represents, where Value_P ij (n) is the target recognition probability matrix, k indicates that there are k classifiers (k=2), c indicates that there are c distraction behavior categories (c=2), and n indicates the current test sample number.

[0106] S420: The target recognition probability matrix is ​​fused by calculation to obtain the distraction behavior recognition result.

[0107] Specifically, the formula for calculating the fusion is as follows:

[0108]

[0109] Among them, Value_p ij (n) is the target recognition probability matrix, min() is the minimum value function, Mid_p j (n) is the intermediate matrix, New_p j (n) represents the result of distraction behavior identification.

[0110] Reference Figure 3 An example is given to illustrate the implementation of a driver distraction behavior recognition method for smart cockpits according to the present invention.

[0111] 1. First, initial distraction behavior images were obtained, including driver images of 7 buses, 2 public buses, 2 trucks, and 4 taxis. The road types involved included national highways, provincial highways, expressways, and urban roads. The environmental conditions covered different weather conditions such as sunny and cloudy days. Five types of driving behaviors were identified: safe driving behavior, using a mobile phone, drinking water, talking to the passenger, and tidying up one's appearance. A total of 3025 driving behavior images were used as initial distraction behavior images.

[0112] 2. Input the initial distracted behavior image into the MobileViT-CA model to obtain the driver's posture recognition result in the image; input the initial distracted behavior image into the YOLOv5 model to obtain the key object recognition result;

[0113] 3. Finally, the posture recognition results and key object recognition results are calculated to obtain the distraction behavior recognition results.

[0114] In summary, the driver distraction behavior recognition method for smart cockpits according to embodiments of the present invention has the following advantages:

[0115] 1. In this embodiment of the invention, by embedding the CA attention module into the backbone network of the MobileViT model, a MobileViT-CA model is constructed, which enhances the accurate localization and feature extraction of key information in the shallow semantics of images, and can also improve the accuracy of the classification network while ensuring the lightweight network model.

[0116] 2. In this embodiment of the invention, the posture features identified by the MobileViT-CA model are fused with the object recognition results extracted by the YOLOv5 model to obtain the final recognition result, which reduces the probability of similar distracted driving behaviors being misclassified and improves the accuracy and generalization of the model.

[0117] On the other hand, refer to Figure 4 This invention also provides a driver distraction behavior recognition device for smart cockpits, comprising:

[0118] Module 401 is used to construct the first model;

[0119] The second module 402 is used to input the initial distraction behavior image into the first model and extract the driver posture recognition result from the initial distraction behavior image;

[0120] The third module 403 is used to input the initial distraction behavior image into the second model for training and extract the key object recognition results in the initial distraction behavior image.

[0121] The fourth module 404 is used to calculate the distraction behavior recognition result based on the driver posture recognition result and the key object recognition result.

[0122] On the other hand, embodiments of the present invention also provide an electronic device, including a processor and a memory; the memory is used to store a program; the processor executes the program to implement the aforementioned driver distraction behavior recognition method for smart cockpits.

[0123] On the other hand, embodiments of the present invention also provide a computer-readable storage medium storing a program that is executed by a processor to implement the aforementioned driver distraction behavior recognition method for smart cockpits.

[0124] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0125] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0126] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0127] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0129] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0130] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0131] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0132] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0133] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for recognizing driver distraction behavior in smart cockpits, characterized in that, include: Construct the first model; Input the initial distraction behavior image into the first model, and extract the driver posture recognition result from the initial distraction behavior image; The initial distraction behavior image is input into the second model for training, and the key object recognition results in the initial distraction behavior image are extracted. Based on the driver posture recognition result and the key object recognition result, the distraction behavior recognition result is calculated; The step of constructing the first model includes: Construct the first convolutional layer, the second convolutional layer, and the fully connected layer; Construct a CA attention module, which includes two average pooling layers and two 1×1 convolutional layers; Construct an inverse residual module, which includes two 1×1 convolutional layers and one 3×3 depth-separable convolutional layer; A feature extraction module is constructed, which includes two 3×3 convolutional layers, two 1×1 convolutional layers, and several transformer models; A first model is constructed based on the first convolutional layer, the second convolutional layer, the fully connected layer, the CA attention module, the inverted residual module, and the feature extraction module; The step of inputting the initial distraction behavior image into the first model and extracting the driver posture recognition result from the initial distraction behavior image includes: The initial distraction behavior image is downsampled by the first convolutional layer to obtain a first feature map; The first feature map is encoded along the horizontal and vertical directions by the CA attention module to obtain the second feature map; The second feature map is downsampled using the inverted residual module to obtain the third feature map; The feature extraction module performs feature extraction and feature fusion on the third feature map to obtain the fourth feature map; The second convolutional layer adjusts the number of output channels of the fourth feature map to the number of channels of the initial distraction behavior image; The fourth feature map is predicted and classified using the fully connected layer to obtain the driver posture recognition result in the initial distraction behavior image; In the step of inputting the initial distraction behavior image into the second model for training and extracting the key object recognition results in the initial distraction behavior image, the second model is a YOLOv5 model, which includes an input module, a backbone module and an output module. The input module performs data augmentation on the initial distraction behavior image using the Mosaic data augmentation algorithm, and then adjusts the image size to the target size to obtain a standard image; The standard is sliced ​​using the backbone module, and then spliced ​​and convolved to obtain the initial feature map. The initial feature map is classified and predicted by the output module to obtain the key object recognition result; The step of calculating the distraction behavior recognition result based on the driver posture recognition result and the key object recognition result includes: The probability matrices for the two distraction behaviors of using a mobile phone and drinking water in the driver posture recognition results are concatenated with the probability matrices for the key object recognition results to obtain the target recognition probability matrix. The target recognition probability matrix is ​​fused through calculation to obtain the distraction behavior recognition result.

2. The method for recognizing driver distraction behavior in a smart cockpit according to claim 1, characterized in that, The step of encoding the first feature map along the horizontal and vertical directions using the CA attention module to obtain the second feature map includes: The first feature map is average pooled along the horizontal and vertical directions to obtain feature information in the horizontal and vertical directions. Based on the feature information of each channel in the horizontal and vertical directions, we obtain the horizontal feature set and the vertical feature set; The horizontal feature set and the vertical feature set are concatenated and convolved to obtain an intermediate feature map. After normalization and nonlinear operations, the intermediate feature maps are used to obtain horizontally independent feature maps and vertically independent feature maps. The horizontal independent feature map and the vertical independent feature map are passed through the second convolutional layer and the sigmoid activation function to obtain the horizontal attention weight and the vertical attention weight; The first feature map is weighted based on the horizontal attention weight and the vertical attention weight to obtain the second feature map.

3. The method for recognizing driver distraction behavior in a smart cockpit according to claim 1, characterized in that, In the step of calculating the distraction behavior recognition result based on the driver posture recognition result and the key object recognition result using the minimum value rule, the driver posture recognition result includes safe driving behavior, using a mobile phone, drinking water, talking to the passenger and tidying up one's appearance; the key object recognition result includes communication tools and drinking tools.

4. An apparatus for implementing the driver distraction behavior recognition method for smart cockpits as described in any one of claims 1-3, characterized in that, include: The first module is used to build the first model; The second module is used to input the initial distraction behavior image into the first model and extract the driver posture recognition result from the initial distraction behavior image. The third module is used to input the initial distraction behavior image into the second model for training and extract the key object recognition results in the initial distraction behavior image. The fourth module is used to calculate the distraction behavior recognition result based on the driver posture recognition result and the key object recognition result.

5. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 3.

Images