A driving behavior detection method and a training method for a driving behavior detection system

Abstract

This disclosure presents a driving behavior detection method and a training method for a driving behavior detection system. The method includes: acquiring an image dataset of the user to be detected; acquiring a driving behavior detection type; determining the data to be detected based on the image dataset and the driving behavior detection type; inputting the data to be detected into a joint multi-task driving behavior detection system, where the corresponding model performs behavior detection and outputs the driving behavior detection results for the user. This disclosure enables efficient and accurate joint multi-task detection of two typical non-standard driving behaviors: distracted driving and fatigued driving, laying a solid foundation for improving driving safety. Furthermore, it supports single-task detection, either focusing solely on distracted driving or fatigued driving, improving the flexibility and intelligence of the driving behavior detection process.

Description

Technical Field

[0001] This disclosure generally relates to vehicle control technology, and more particularly to a driving behavior detection method and a training method for a driving behavior detection system. Background Technology

[0002] With the rapid development of the global automotive industry, traffic accidents caused by improper driving behavior are also gradually increasing. Among them, distracted driving and fatigued driving are two particularly prominent improper driving behaviors, and the serious harm they pose to driving safety is becoming increasingly severe.

[0003] However, there is currently no comprehensive method for detecting distracted driving and fatigued driving. Therefore, how to efficiently and accurately detect driving behavior has become a pressing issue. Summary of the Invention

[0004] This disclosure addresses some of the shortcomings mentioned in the background art by providing a driving behavior detection method, device, and electronic device.

[0005] In a first aspect, embodiments of this disclosure provide a driving behavior detection method, comprising: acquiring an image dataset of a user to be detected; acquiring driving behavior detection types for the user to be detected, wherein the driving behavior detection types include distracted driving behavior detection, fatigued driving behavior detection, and joint detection of distracted driving behavior detection and fatigued driving behavior detection; determining data to be detected based on the image dataset and the driving behavior detection types; inputting the data to be detected into a joint multi-task driving behavior detection system, wherein the corresponding model in the joint multi-task driving behavior detection system performs behavior detection, and outputting the driving behavior detection result of the user to be detected, wherein the joint multi-task driving behavior detection system includes a distracted driving behavior detection model and a fatigued driving behavior detection model.

[0006] In a second aspect, embodiments of this disclosure provide a training method for a driving behavior detection system, comprising: acquiring training samples, wherein the training samples include multi-view detection data and target view detection data, and reference detection results corresponding to the multi-view detection data and target view detection data; inputting the training samples into a joint multi-task driving behavior detection system to be trained, wherein the corresponding model in the joint multi-task driving behavior detection system to be trained performs behavior detection, and outputs detection results corresponding to the training samples, wherein the joint multi-task driving behavior detection system to be trained includes a distracted driving behavior detection model and a fatigued driving behavior detection model; acquiring a first loss function corresponding to the distracted driving behavior detection model and a second loss function corresponding to the fatigued driving behavior detection model based on the detection results and reference detection results; acquiring a total loss function of the joint multi-task driving behavior detection system to be trained based on the first loss function and the second loss function, and adjusting the parameters of the joint multi-task driving behavior detection system to be trained based on the total loss function until the training termination condition is met, and determining the joint multi-task driving behavior detection system to be trained after the last parameter adjustment as the joint multi-task driving behavior detection system.

[0007] In a third aspect, embodiments of this disclosure provide a driving behavior detection device, comprising: a first acquisition unit for acquiring an image dataset of a user to be detected; a second acquisition unit for acquiring driving behavior detection types for the user to be detected, wherein the driving behavior detection types include distracted driving behavior detection, fatigued driving behavior detection, and joint detection of distracted driving behavior detection and fatigued driving behavior detection; a determination unit for determining data to be detected based on the image dataset and the driving behavior detection types; and a detection unit for inputting the data to be detected into a joint multi-task driving behavior detection system, whereby the corresponding model in the joint multi-task driving behavior detection system performs behavior detection and outputs the driving behavior detection result of the user to be detected, wherein the joint multi-task driving behavior detection system includes a distracted driving behavior detection model and a fatigued driving behavior detection model.

[0008] In a fourth aspect, embodiments of this disclosure provide a training apparatus for a driving behavior detection system, comprising: a first acquisition unit for acquiring training samples, wherein the training samples include multi-view detection data and target view detection data, and reference detection results corresponding to the multi-view detection data and target view detection data; a detection unit for inputting the training samples into a joint multi-task driving behavior detection system to be trained, wherein the corresponding model in the joint multi-task driving behavior detection system to be trained performs behavior detection, and outputs detection results corresponding to the training samples, wherein the joint multi-task driving behavior detection system to be trained includes a distracted driving behavior detection model and a fatigued driving behavior detection model; a second acquisition unit for acquiring a first loss function corresponding to the distracted driving behavior detection model and a second loss function corresponding to the fatigued driving behavior detection model based on the detection results and the reference detection results; and a determination unit for acquiring a total loss function of the joint multi-task driving behavior detection system to be trained based on the first loss function and the second loss function, and adjusting the parameters of the joint multi-task driving behavior detection system to be trained based on the total loss function until the training termination condition is met, and determining the joint multi-task driving behavior detection system to be trained after the last parameter adjustment as the joint multi-task driving behavior detection system.

[0009] In a fifth aspect, embodiments of this disclosure provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the methods described in the first and second aspects.

[0010] In a sixth aspect, embodiments of this disclosure provide a processor-readable storage medium storing a computer program for causing a processor to perform the methods described in the first and second aspects.

[0011] In a seventh aspect, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the methods described in the first and second aspects.

[0012] The embodiments provided in this disclosure have at least the following beneficial technical effects: A driving behavior detection method according to an embodiment of this disclosure can acquire an image dataset of a user to be detected and driving behavior detection types for that user. The driving behavior detection types include distracted driving behavior detection, fatigued driving behavior detection, and joint detection of distracted driving behavior detection and fatigued driving behavior detection. Based on the image dataset and driving behavior detection types, data to be detected is determined, and then this data is input into a joint multi-task driving behavior detection system. The corresponding model in the joint multi-task driving behavior detection system performs behavior detection and outputs the driving behavior detection results for the user to be detected. The joint multi-task driving behavior detection system includes a distracted driving behavior detection model and a fatigued driving behavior detection model. Therefore, this embodiment of the disclosure can efficiently and accurately perform joint multi-task detection of two typical non-standard driving behaviors—distracted driving behavior and fatigued driving behavior—simultaneously, laying a solid foundation for improving driving safety. Furthermore, it can also support single-task detection of only distracted driving behavior detection or only fatigued driving behavior detection, improving the flexibility and intelligence of the driving behavior detection process.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart illustrating a driving behavior detection method; Figure 2 This is a schematic diagram of the acquisition position of a multi-viewpoint. Figure 3 This is a flowchart illustrating another method for detecting driving behavior; Figure 4 This is a schematic diagram of a joint multi-task driving behavior detection system. Figure 5 This is a schematic diagram of a feature extraction and mapping process; Figure 6 This is a flowchart illustrating another method for detecting driving behavior; Figure 7 This is a schematic diagram of a feature fusion process; Figure 8 This is a schematic diagram of the classification layer structure of a distracted driving behavior detection model; Figure 9 This is a schematic diagram of a classification module for a fatigue driving behavior detection model; Figure 10This is a flowchart illustrating a training method for a driving behavior detection system; Figure 11 This is a schematic diagram of a RetinaFace face detector architecture; Figure 12 This is a flowchart illustrating another training method for a driving behavior detection system; Figure 13 This is a schematic diagram illustrating the design process of a driving behavior detection method; Figure 14 This is a schematic diagram of a driving behavior detection device; Figure 15 This is a schematic diagram of a training device for a driving behavior detection system. Figure 16 It is a block diagram of an electronic device. Detailed Implementation

[0015] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present disclosure and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the drawings, not the entire structure.

[0016] It should be noted that in this disclosure, distracted driving behavior mainly refers to inattentive driving behaviors, such as looking at a mobile phone, eating, or chatting; while fatigued driving behavior mainly refers to driving behavior when the driver is fatigued. Distracted driving and fatigued driving behaviors are mostly unconscious actions that occur while driving. Efficient and accurate detection can promptly remind drivers to regulate their driving behavior and ensure driving safety.

[0017] It's also important to note that driving behavior detection methods for distracted driving and those for fatigued driving often differ significantly. Firstly, the data differs: image-based methods for detecting distracted driving often focus on the driver's body movements, while methods for detecting fatigued driving tend to focus more on facial features. Therefore, the viewing angles from which images are collected differ. Secondly, the recognition methods differ. Distracted driving often relies on supervised learning for identification and classification, while fatigued driving uses features from the eyes and mouth for decision-making. Thus, detecting these two types of driving behaviors is essentially a separate task.

[0018] Based on the above characteristics, although some driving behavior detection methods already exist, the existing driving behavior detection methods are usually only for distracted driving behavior or only for fatigued driving behavior.

[0019] Therefore, this disclosure proposes a joint multi-task driving behavior detection method targeting distracted driving behavior and fatigued driving behavior, which can simultaneously perform efficient and accurate joint multi-task detection of these two typical non-standard driving behaviors.

[0020] Figure 1 This is a schematic diagram based on the first embodiment of this disclosure. (See diagram below.) Figure 1 As shown, a driving behavior detection method proposed in this disclosure embodiment will be explained and described, specifically including the following steps: S101. Obtain the image dataset of the user to be detected.

[0021] The user to be detected can be the driver of any vehicle; the image dataset of the user to be detected can be a set of image data of the user to be detected that includes each frame of image data collected during the driving process.

[0022] It should be noted that this disclosure does not limit the specific method for obtaining the image dataset of the user to be detected, and it can be set according to the actual situation.

[0023] One possible implementation is to predetermine the locations of image acquisition devices such as cameras, and then install multiple image acquisition devices at different locations within the driver's cab. Each image acquisition device acquires a video stream, and the acquired data undergoes necessary preprocessing. The locations include at least the following: Figure 2 The following four positions are shown: directly in front of the driver, to the right of the front passenger, to the left of the driver, and to the right front of the driver.

[0024] Furthermore, frame-taking operations can be performed on the video streams acquired by all image acquisition devices according to a pre-set cycle to obtain a multi-view image dataset. Then, the image acquisition device set at the location that can capture the clearest facial features (such as the driver's front - view 2) can be selected from all the image acquisition devices, and the video stream acquired by the image acquisition device can be frame-taking operations to obtain the target view detection data.

[0025] In this case, the target viewpoint data to be detected is a subset of the multi-view image dataset.

[0026] As another possible implementation, a first setting position of the image acquisition device for acquiring multi-view image datasets can be predetermined, and a second setting position of the image acquisition device for acquiring target view image datasets can be determined in advance, wherein the first setting position and the second setting position may overlap.

[0027] For example, four settings can be predetermined as the first settings: directly in front of the driver, to the right of the passenger, to the left of the driver, and to the right front of the driver. Three settings can be predetermined as the second settings: directly in front of the driver, to the left front of the driver, and to the right front of the driver. In this case, the setting position directly in front of the driver is the intersection of the first and second settings.

[0028] Furthermore, each image acquisition device positioned at the first location acquires a video stream, and frames are extracted from the video streams acquired by all the image acquisition devices at the first location according to a pre-set period to obtain a multi-view image dataset. Each image acquisition device positioned at the second location acquires a video stream, and the optimal video stream is selected from all the acquired video streams. Frames are then extracted from the optimal video stream according to a pre-set period to obtain the target viewpoint detection data. The optimal video stream can be the video stream with the clearest facial features.

[0029] In this case, the target viewpoint data to be detected is a subset of the multi-view image dataset, or the target viewpoint data to be detected is a union of the multi-view image dataset.

[0030] S102. Obtain the driving behavior detection type for the user to be detected, including distracted driving behavior detection, fatigued driving behavior detection, and joint detection of distracted driving behavior detection and fatigued driving behavior detection.

[0031] This disclosure supports multiple types of driving behavior detection, namely, it can detect distracted driving behavior only for distracted driving, it can detect fatigued driving behavior only for fatigued driving, and it can perform joint detection of distracted driving behavior and fatigued driving behavior.

[0032] S103. Determine the data to be detected based on the image dataset and the driving behavior detection type.

[0033] In this embodiment of the disclosure, different driving behavior detection types correspond to different data to be detected. For example, when the driving behavior detection type is to detect distracted driving behavior only, the data to be detected is multi-view data; when the driving behavior detection type is to detect fatigued driving behavior only, the data to be detected is target view data; and when the driving behavior detection type is to detect a combination of distracted driving behavior detection and fatigued driving behavior detection, the data to be detected is multi-view data and target view data.

[0034] S104. Input the data to be detected into the joint multi-task driving behavior detection system. The corresponding model in the joint multi-task driving behavior detection system performs behavior detection and outputs the driving behavior detection results of the user to be detected. The joint multi-task driving behavior detection system includes a distracted driving behavior detection model and a fatigued driving behavior detection model.

[0035] In this embodiment of the disclosure, different data to be detected correspond to different driving behavior detection results. For example, when the data to be detected is multi-view data, the driving behavior detection result is a distracted driving behavior detection result; when the data to be detected is target view data, the driving behavior detection result is a fatigued driving behavior detection result; and when the data to be detected is both multi-view data and target view data, the driving behavior detection result is a joint detection result for distracted driving behavior detection and fatigued driving behavior detection.

[0036] In other words, this disclosure allows for the detection of corresponding types of driving behavior based on at least a portion of image data from an image dataset composed of multiple image data sets. The input and output of the joint multi-task driving behavior detection system differ depending on the type of driving behavior detection. If the system input is only the data to be detected corresponding to distracted driving behavior detection, the system output is the distracted driving behavior detection result; if the system input is only the data to be detected corresponding to fatigued driving behavior detection, the system output is the fatigued driving behavior detection result; if the system input is the data to be detected corresponding to the joint detection of distracted driving behavior detection and fatigued driving behavior detection, the system output is the joint detection result for distracted driving behavior detection and fatigued driving behavior detection. Therefore, the joint multi-task driving behavior detection system in this disclosure supports both single-task processing and multi-task processing.

[0037] A driving behavior detection method according to an embodiment of this disclosure can acquire an image dataset of a user to be detected and driving behavior detection types for that user. The driving behavior detection types include distracted driving behavior detection, fatigued driving behavior detection, and joint detection of distracted driving behavior detection and fatigued driving behavior detection. Based on the image dataset and driving behavior detection types, data to be detected is determined, and then this data is input into a joint multi-task driving behavior detection system. The corresponding model in the joint multi-task driving behavior detection system performs behavior detection and outputs the driving behavior detection results for the user to be detected. The joint multi-task driving behavior detection system includes a distracted driving behavior detection model and a fatigued driving behavior detection model. Therefore, this embodiment of the disclosure can efficiently and accurately perform joint multi-task detection of two typical non-standard driving behaviors—distracted driving behavior and fatigued driving behavior—simultaneously, laying a solid foundation for improving driving safety. Furthermore, it can also support single-task detection of only distracted driving behavior detection or only fatigued driving behavior detection, improving the flexibility and intelligence of the driving behavior detection process.

[0038] It should be noted that some existing technologies exist that only target distracted driving behavior or fatigued driving behavior. The following will briefly explain the relevant detection methods.

[0039] Currently, the common methods for detecting distracted driving behavior include the following steps: 1. Use the State Farm driving behavior dataset or other open-source or self-built distracted driving behavior datasets to train a deep learning model to build a distracted driving behavior recognition model; 2. Fix the camera to the right side of the passenger seat to capture videos of distinctive driving behaviors; 3. The video is periodically frame-by-frame captured and preprocessed before being fed into a classification model for driving behavior recognition and classification; 4. Make relevant decisions and issue alerts based on the model's classification results over a period of time.

[0040] Currently, the common methods for detecting fatigued driving behavior include the following steps: 1. Fix the camera in front of the driver to collect video and periodically capture image frames; 2. Use a face detector to extract face images, and use a face landmark detection method to locate and detect faces and extract fatigue features related to the eyes and mouth; 3. Classify fatigue based on relevant features, such as using the PERCLOS (Percentage of Eyelid Closure) algorithm to calculate the driver's blinking frequency or mouth opening degree over a period of time to make a decision on whether it is fatigue.

[0041] However, the detection method proposed in this disclosure has been improved upon the prior art, even when dealing with distracted driving behavior detection and fatigued driving behavior detection. The following embodiments will explain in detail, and will not be repeated here.

[0042] The following example uses a combined detection method targeting distracted driving behavior and fatigued driving behavior to illustrate the specific process of "determining the data to be detected based on the image dataset and the driving behavior detection type" in step S103.

[0043] In response to the driving behavior detection type being joint detection of distracted driving behavior and fatigued driving behavior, as a possible implementation, a first selection strategy for distracted driving behavior detection and a second selection strategy for fatigued driving behavior detection can be obtained. Then, based on the first selection strategy, multi-view detection data can be obtained, and based on the second selection strategy, target view detection data can be obtained.

[0044] The first selection strategy can be to perform frame-taking operations on the video streams acquired by multiple image acquisition devices set at different locations in the cab according to a preset period, or to perform frame-taking operations on the video streams acquired by multiple image acquisition devices set at the first location according to a preset period. The specific process is as described above and will not be repeated here.

[0045] The second selection strategy can be to select the image acquisition device that can capture the clearest facial features from all the image acquisition devices set in different locations in the driver's cab according to a preset period, and then perform frame capture operation on the video stream acquired by the image acquisition device. Alternatively, it can perform frame capture operation on the video streams acquired by multiple image acquisition devices set in the second location according to a preset period. The specific process is as described above and will not be repeated here.

[0046] After determining the data to be detected based on the image dataset and the type of driving behavior detection, the cloud will send the input image to a trained multi-task model for classification and decision-making.

[0047] The following example uses a combined detection system for distracted driving behavior and fatigued driving behavior to illustrate the specific process of step S104, which involves "inputting the data to be detected into the combined multi-task driving behavior detection system, having the corresponding model in the combined multi-task driving behavior detection system perform behavior detection, and outputting the driving behavior detection results of the user to be detected."

[0048] As one possible implementation, such as Figure 3 As shown, the specific steps include: S301. Input the multi-view detection data and the target view detection data into the joint multi-task driving behavior detection system. The backbone network of the joint multi-task driving behavior detection system extracts features from the multi-view detection data and the target view detection data respectively to obtain the multi-view detection data features and the target view detection data features. The multi-view detection data features include a feature matrix and a vertex matrix. The feature matrix includes the features extracted from the vertices of the multi-view detection data. The vertex matrix is ​​designed according to the spatial position of the multi-view detection data.

[0049] It should be noted that the joint multi-task driving behavior detection system in this embodiment adopts a multi-task model architecture based on parameter sharing. It uses the proposed MVGD-Net model (a multi-view distracted driving behavior recognition model) and the FD-Net model (a model capable of recognizing closed-eye features and yawning features) to jointly perform multi-task recognition of distracted driving and fatigued driving. Its structure is as follows: Figure 4 As shown.

[0050] In this embodiment, the feature extraction modules of the distracted driving behavior detection model and the fatigue driving behavior detection model share parameters. A taskID flag is added to the model input layer to identify the subsequent module. A taskID of 0 indicates fatigue driving behavior detection, while a taskID of 1 indicates distracted driving behavior detection. In other words, the backbone network of the joint multi-task driving behavior detection system refers to the backbone network composed of the distracted driving behavior detection model backbone network and the fatigue driving behavior detection model backbone network, both using ResNet-18 backbone networks.

[0051] It should also be noted that when the backbone network (Convolutional Neural Network (CNN)) of the joint multi-task driving behavior detection system extracts features from the multi-view data to obtain the features of the multi-view data, such as... Figure 5As shown, the multi-view detection data features, including feature matrices and vertex matrices, will be obtained. These multi-view detection data features are graph-structured data.

[0052] S302. Input the multi-view data features to be detected into the distracted driving behavior detection model. The distracted driving behavior detection model performs distracted driving behavior detection based on the multi-view data features to obtain the first intermediate detection result. Input the target view data features to be detected into the fatigue driving behavior detection model. The fatigue driving behavior detection model performs fatigue driving behavior detection based on the target view data to be detected to obtain the second intermediate detection result.

[0053] One possible implementation method is to use a distracted driving behavior detection model to detect distracted driving behavior based on features of multi-view data, and obtain a first intermediate detection result. Figure 6 As shown, the specific steps include: S601. The feature fusion layer of the distracted driving behavior detection model performs feature fusion processing on the features of the multi-view data to be detected, obtains the aggregated new features, and inputs them into the first classification layer of the distracted driving behavior detection model.

[0054] S602. The first classification layer classifies the aggregated new features to obtain the first intermediate detection result.

[0055] It should be noted that the distracted driving behavior detection model proposed in this disclosure introduces the concept of multiple views. Based on GCN (Graph Convolutional Network), a multi-view distracted driving behavior recognition model MVGD-Net is constructed. Its structure consists of three main parts: a feature extraction layer (jointly connected to the backbone network of the multi-task driving behavior detection system), a feature fusion layer, and a classification layer.

[0056] In this embodiment of the disclosure, after obtaining the features of the multi-view data to be detected, all (in terms of) features in the multi-view data to be detected are used. Figure 2 For example, the features of four views are fused together, such as... Figure 7 As shown, firstly, the k-nearest neighbor algorithm is used for each view vertex to find the two closest view vertices and construct a comprehensive feature. V_f This comprehensive feature includes the features and location information of the current vertex's neighbors, then... V_f It will be sent into a specially designed R The network outputs a weight matrix. wThis weight matrix represents the influence of each vertex's neighbors on the current vertex. Then, the weights are multiplied by the original features to aggregate the features, and finally, the aggregated features are fed into a neural network layer for non-linear transformation to obtain the new aggregated features. The activation function can be Leaky ReLU (Leaky Rectified Linear Unit).

[0057] Furthermore, the aggregated new features will be input as follows: Figure 8 The first classification layer shown classifies the aggregated new features to obtain the first intermediate detection result. Regularization can be achieved using Dropout.

[0058] It should be noted that the fatigue driving behavior detection model proposed in this disclosure has a structure consisting of two main parts: a feature extraction layer (the backbone network in the joint multi-task driving behavior detection system) and a classification layer. Since fatigue driving behavior detection requires facial recognition, it only uses images from a single target perspective (such as images from the driver's direct viewpoint). Compared to the distracted driving behavior detection model, it eliminates the need for feature fusion, and therefore does not include a feature fusion layer, thus saving the feature fusion operation.

[0059] As one possible approach to detecting fatigue driving behavior using a fatigue driving behavior detection model based on data from the target perspective and obtaining a second intermediate detection result, the second classification layer of the fatigue driving behavior detection model can be used to classify the data from the target perspective to obtain the second intermediate detection result.

[0060] As an example, the second classification layer structure is as follows: Figure 9 As shown, the nonlinear activation function can be the h-swish function, which can effectively improve the accuracy of the neural network.

[0061] S303. Based on the first intermediate detection result and the second intermediate detection result, determine the driving behavior detection result and output it.

[0062] As one possible implementation, the shooting frequency of the multi-view image dataset can be obtained, and based on the shooting frequency, the joint judgment frequency for the first intermediate detection result and the second intermediate detection result can be determined. Further, based on the joint judgment frequency, the first number of times the first intermediate detection result is found to be abnormal and the second number of times the second intermediate detection result is found to be abnormal within any target period can be obtained; in response to the first number and / or the second number satisfying a preset condition, the driving behavior detection result is determined to be abnormal and output.

[0063] The preset conditions can be set according to the actual situation, such as any dynamically adjustable target threshold.

[0064] For example, the video stream is captured as a frame every 0.5 seconds (shooting frequency). After the camera video capture is turned on for a certain duration (e.g., 10 seconds), the classification results of the past 20 times (target period) are judged every 1 second (joint judgment frequency).

[0065] It should be noted that in this disclosure, the driving behavior detection result is determined to be abnormal when either the first or second intermediate detection result reaches the first target threshold. For example, if the number of times the first intermediate detection result or the second intermediate detection result is abnormal exceeds 10 times in the last 20 detections (the first target threshold), then the driving behavior detection result is determined to be abnormal.

[0066] Furthermore, to provide early warnings for potential non-standard driving behaviors, when neither the first nor the second count reaches the first target threshold, the sum of the first and second counts can be obtained. If the sum of the first and second counts reaches the second target threshold, the driving behavior detection result can be determined to be abnormal. The second target threshold can be the same as or different from the first target threshold. For example, if in the last 20 tests, the first intermediate detection result indicates abnormality more than 6 times and the second intermediate detection result indicates abnormality more than 7 times, then when the second target threshold is 12 times, the driving behavior detection result is determined to be abnormal.

[0067] It should also be noted that in this disclosure, when the driving behavior detection result indicates an abnormality, alarm processing can be performed. The alarm processing method can be at least one of the following: voice alarm message broadcast, vibration alarm, or indicator light flashing alarm.

[0068] Therefore, in this disclosure, multiple image acquisition devices are set up around the driver to collect video of the driver's driving behavior, and image frames are extracted from the video stream at a certain frequency and transmitted to a detection device to detect abnormal driving behavior. This detection device uploads the input multi-view data to the cloud, where the cloud feeds the input images into a trained multi-task model for classification, makes a decision on abnormal driving behavior, and alerts the driver via a device or terminal.

[0069] Furthermore, this disclosure constructs a multi-view distracted driving behavior recognition model MVGD-Net based on the concept of multi-view learning. It extracts features from the images of each view and constructs feature matrices and position matrices to build graph structure data. Then, it uses the k-nearest neighbor algorithm and GCN to aggregate features between views, thereby ensuring that the model can take into account driving behavior features from different perspectives and significantly improve the accuracy of model recognition.

[0070] Meanwhile, this disclosure addresses the significant difference between the methods for judging distracted driving and fatigued driving, proposing a joint multi-task driving behavior detection system using a multi-task architecture to jointly identify the two driving behavior modes. By sharing the parameters of the feature extraction layer among the sub-task models, it not only achieves the purpose of joint identification but also effectively reduces the problem of overfitting.

[0071] The following explanation uses examples of distracted driving behavior detection or fatigued driving behavior detection to illustrate the driving behavior detection process proposed in this disclosure.

[0072] As one possible implementation, optionally, the response driving behavior detection type is distracted driving behavior detection or fatigued driving behavior detection. Multi-view detection data or target view detection data can be acquired and input into the joint multi-task driving behavior detection system. The backbone network of the joint multi-task driving behavior detection system extracts features from the multi-view detection data or target view detection data to obtain multi-view detection data features or target view detection data features.

[0073] Furthermore, the multi-view data features to be detected can be input into the distracted driving behavior detection model, which will then perform distracted driving behavior detection based on the multi-view data features, and output the distracted driving behavior detection results. Alternatively, the target view data features can be input into the fatigue driving behavior detection model, which will then perform fatigue driving behavior detection based on the target view data, and output the fatigue driving behavior detection results.

[0074] Therefore, this disclosure improves upon existing technologies even when dealing with distracted driving behavior detection and fatigued driving behavior detection. Specifically, for distracted driving behavior detection, this disclosure employs a graph convolutional architecture-based model. By treating each view as a separate vertex, a graph structure is constructed, and graph convolution is performed to fuse features between different views, resulting in higher recognition accuracy. For fatigued driving behavior detection, this disclosure uses a supervised learning method for recognition, offering superior predictive performance and controllability.

[0075] like Figure 10 As shown, the training method for a driving behavior detection system proposed in this disclosure embodiment will be explained, specifically including the following steps: S1001. Obtain training samples, wherein the training samples include multi-view detection data and target view detection data, as well as reference detection results corresponding to the multi-view detection data and target view detection data.

[0076] S1002. Input the training samples into the joint multi-task driving behavior detection system to be trained. The corresponding model in the joint multi-task driving behavior detection system to be trained performs behavior detection and outputs the detection results corresponding to the training samples. The joint multi-task driving behavior detection system to be trained includes a distracted driving behavior detection model and a fatigued driving behavior detection model.

[0077] S1003. Based on the detection results and reference detection results, obtain the first loss function corresponding to the distracted driving behavior detection model and the second loss function corresponding to the fatigued driving behavior detection model.

[0078] S1004. Based on the first loss function and the second loss function, obtain the total loss function of the joint multi-task driving behavior detection system to be trained, and adjust the parameters of the joint multi-task driving behavior detection system to be trained according to the total loss function until the training termination condition is met. Then, determine the joint multi-task driving behavior detection system to be trained after the last parameter adjustment as the joint multi-task driving behavior detection system.

[0079] Among them, the multi-view data to be detected refers to the multi-view dataset used for training the distracted driving behavior detection model.

[0080] For example, in a simulated driving environment, four camera viewpoints can be set up to simultaneously collect multi-view driving data. These viewpoints are directly in front of the driver, to the right of the passenger, to the left of the driver, and to the right front of the driver. Distracted driving behaviors can be pre-defined into categories such as normal driving, looking at a mobile phone, answering a phone call, talking to the passenger, eating or drinking, operating a terminal screen, and looking down to pick up items. Each action is captured for half a minute, and different volunteers are selected as drivers to collect data. Furthermore, after data collection, the video data from different perspectives needs to be framed according to certain rules, and images with indistinct features need to be filtered out. It is important to note that the images from the four perspectives must correspond one-to-one. Finally, a multi-view distracted driving behavior dataset, i.e., multi-view detection data, is constructed through data annotation.

[0081] Among them, the target view data to be detected refers to the single-view dataset used for training the fatigue driving behavior detection model.

[0082] For example, fatigue driving behavior can be pre-defined into three categories: normal, yawning, and frequent blinking. Further, a dataset can be built by capturing eye and mouth images from facial images using a trained face detector; then, a model FD-Net capable of recognizing closed-eye and yawning features is trained through supervised learning. RetinaFace (a single-stage multi-task convolutional neural network face detection model) is chosen as the face detector for extracting eye and mouth images, and WIDER Face is used for training and testing the face detector. RetinaFace is a single-stage algorithm with a relatively fast detection speed. Its main architecture is as follows... Figure 11 As shown, feature extraction is first performed using the backbone network, then feature fusion and enhancement are performed on the extracted features, and finally, three detection heads—class detection, frame detection, and keypoint detection—are used for multi-task output. Feature enhancement can be performed using the SSH (Single Stage Headless) feature enhancement method.

[0083] According to an embodiment of this disclosure, a training method for a driving behavior detection system involves acquiring training samples and inputting them into a joint multi-task driving behavior detection system to be trained. The corresponding model in the joint multi-task driving behavior detection system performs behavior detection and outputs detection results corresponding to the training samples. Then, based on the detection results and reference detection results, a first loss function corresponding to the distracted driving behavior detection model and a second loss function corresponding to the fatigued driving behavior detection model are obtained. Based on the first and second loss functions, the total loss function of the joint multi-task driving behavior detection system to be trained is obtained. The parameters of the joint multi-task driving behavior detection system to be trained are adjusted according to the total loss function until the training termination condition is met. The joint multi-task driving behavior detection system after the last parameter adjustment is determined as the joint multi-task driving behavior detection system. Therefore, this embodiment of the disclosure can obtain a well-trained joint multi-task driving behavior detection system with a stable convergence state based on an effective model training method, ensuring the accuracy of driving behavior detection results. This lays a solid foundation for efficient and accurate joint multi-task detection of two typical non-standard driving behaviors—distracted driving behavior and fatigued driving behavior—and improves driving safety. In addition, it can support single-task detection, such as detecting only distracted driving behavior or only fatigued driving behavior, which improves the flexibility and intelligence of the driving behavior detection process.

[0084] The following example provides a detailed explanation of the specific process in step S104, "obtaining the total loss function of the joint multi-task driving behavior detection system to be trained based on the first loss function and the second loss function."

[0085] As one possible implementation, such as Figure 12 As shown, the specific steps include: S1201. For any loss function, use the Dynamic Weighted Average (DWA) algorithm to obtain the relative rate of change of loss for at least two consecutive periods, and perform exponential normalization on the relative rate of change of loss to obtain the weight of any loss function.

[0086] S1202. Obtain the total loss function based on the weights corresponding to each loss function.

[0087] To enable better training of the multi-task model across various sub-tasks, this embodiment selects the DWA (Dynamic Weight Average) algorithm to dynamically balance the weights of the loss functions of the two sub-tasks for calculating the total loss. The core formula of the DWA algorithm is as follows:

[0088]

[0089]

[0090] In this embodiment of the disclosure, the relative rate of change of loss for two consecutive periods can be calculated first. r t Then, the weights of each subtask are obtained through exponential normalization. w t Finally, the final loss function is obtained by combining the loss functions based on the weights of the sub-tasks. L .

[0091] Therefore, the subtask loss combination strategy using the DWA algorithm as the loss function in this disclosure can adjust the weights of subtasks according to the specific training situation, thus ensuring effective training of the model.

[0092] The driving behavior detection process and the training process of the driving behavior detection system of this disclosure will be explained below with reference to embodiments.

[0093] It should be noted that, in order to more clearly explain the important processing stages involved in this disclosure, the following explanations will be provided from three perspectives: distracted driving behavior detection, fatigued driving behavior detection, and a combination of distracted driving behavior detection and fatigued driving behavior detection.

[0094] The process for detecting and processing distracted driving behavior mainly involves four stages: image data acquisition, data preprocessing, multi-view model classification, and obtaining classification results. Optionally, firstly, video streams from multiple perspectives are acquired using cameras at different locations; then, multi-view driving images are obtained by periodically capturing frames from the video streams; finally, the data is preprocessed and fed into a multi-view classification model to obtain classification results.

[0095] It should be noted that since the current distracted driving dataset consists of single-view images, if a multi-view model is to be built, a multi-view dataset must first be built for model training. The specific construction process is as described above and will not be repeated here.

[0096] For the fatigue driving behavior detection process, unlike the conventional method of fatigue feature recognition by calculating PERCLOS, fatigue detection is also performed using supervised learning methods because distracted driving and fatigue driving will be identified simultaneously. The main process can be summarized as follows: First, a dataset is established by capturing eye and mouth images of faces using a trained face detector; then, a model FD-Net capable of recognizing closed-eye features and yawning features is trained through supervised learning.

[0097] To address the combined detection of distracted driving and fatigued driving, a multi-task model architecture based on parameter sharing is adopted. The proposed MVGD-Net and FD-Net models are used for multi-task joint recognition of distracted driving and fatigued driving. The main process is as follows: Figure 13 As shown.

[0098] It should be noted that the training of the distracted driving behavior detection model and the fatigued driving behavior detection model in the joint multi-task driving behavior detection system is treated as two sub-tasks. In order to perform better training on each sub-task, the DWA algorithm is selected to dynamically balance the weights of the loss functions of the two sub-tasks to calculate the total loss.

[0099] Corresponding to the driving behavior detection method provided in the above embodiments, an embodiment of this disclosure also provides a driving behavior detection device. Since the driving behavior detection device provided in this disclosure corresponds to the driving behavior detection method provided in the above embodiments, the implementation of a driving behavior detection method is also applicable to the driving behavior detection device provided in this embodiment, and will not be described in detail in this embodiment.

[0100] Figure 14 This is a schematic diagram of the structure of a driving behavior detection device according to an embodiment of the present disclosure.

[0101] like Figure 14As shown, the driving behavior detection device 2000 includes: a first acquisition unit 1410, a second acquisition unit 1420, a determination unit 1430, and a detection unit 1440.

[0102] The system includes a first acquisition unit 1410 for acquiring an image dataset of the user to be detected; a second acquisition unit 1420 for acquiring the driving behavior detection type for the user to be detected, wherein the driving behavior detection type includes distracted driving behavior detection, fatigued driving behavior detection, and joint detection of distracted driving behavior detection and fatigued driving behavior detection; a determination unit 1430 for determining the data to be detected based on the image dataset and the driving behavior detection type; and a detection unit 1440 for inputting the data to be detected into a joint multi-task driving behavior detection system, whereby the corresponding model in the joint multi-task driving behavior detection system performs behavior detection and outputs the driving behavior detection result of the user to be detected, wherein the joint multi-task driving behavior detection system includes a distracted driving behavior detection model and a fatigued driving behavior detection model.

[0103] According to one embodiment of this disclosure, the determining unit 1430 is further configured to: acquire a first selection strategy for distracted driving behavior detection and a second selection strategy for fatigued driving behavior detection; acquire multi-view detection data according to the first selection strategy, and acquire target view detection data according to the second selection strategy.

[0104] According to one embodiment of this disclosure, the detection unit 1440 is further configured to: input multi-view detection data and target view detection data into a joint multi-task driving behavior detection system; extract features from the multi-view detection data and target view detection data respectively through the backbone network of the joint multi-task driving behavior detection system to obtain multi-view detection data features and target view detection data features, wherein the multi-view detection data features include a feature matrix and a vertex matrix, the feature matrix includes features extracted from the vertices of the multi-view detection data, and the vertex matrix is ​​designed according to the spatial position of the multi-view detection data; input the multi-view detection data features into a distracted driving behavior detection model; perform distracted driving behavior detection based on the multi-view detection data features to obtain a first intermediate detection result; input the target view detection data features into a fatigue driving behavior detection model; perform fatigue driving behavior detection based on the target view detection data to obtain a second intermediate detection result; and determine and output the driving behavior detection result based on the first and second intermediate detection results.

[0105] According to one embodiment of this disclosure, the detection unit 1440 is further configured to: acquire the shooting frequency of the multi-view image dataset, and determine the joint judgment frequency for the first intermediate detection result and the second intermediate detection result based on the shooting frequency; acquire, based on the joint judgment frequency, the first number of times the first intermediate detection result is abnormal and the second number of times the second intermediate detection result is abnormal within any target period; and, in response to the first number and / or the second number satisfying a preset condition, determine that the driving behavior detection result is abnormal and output it.

[0106] According to one embodiment of this disclosure, the detection unit 1440 is further configured to: perform feature fusion processing on the multi-view data features to be detected by the feature fusion layer of the distracted driving behavior detection model to obtain new aggregated features and input them into the first classification layer of the distracted driving behavior detection model; and perform classification processing on the new aggregated features by the first classification layer to obtain a first intermediate detection result.

[0107] According to one embodiment of this disclosure, the detection unit 1440 is further configured to: classify the target viewpoint data to be detected by the second classification layer of the fatigue driving behavior detection model to obtain a second intermediate detection result.

[0108] A driving behavior detection device according to an embodiment of this disclosure can acquire an image dataset of a user to be detected and driving behavior detection types for that user. The driving behavior detection types include distracted driving behavior detection, fatigued driving behavior detection, and joint detection of distracted driving behavior detection and fatigued driving behavior detection. Based on the image dataset and driving behavior detection types, the device determines the data to be detected and then inputs it into a joint multi-task driving behavior detection system. The corresponding model in the joint multi-task driving behavior detection system performs behavior detection and outputs the driving behavior detection results for the user to be detected. The joint multi-task driving behavior detection system includes a distracted driving behavior detection model and a fatigued driving behavior detection model. Therefore, this embodiment of the disclosure can simultaneously perform efficient and accurate joint multi-task detection of two typical non-standard driving behaviors: distracted driving behavior and fatigued driving behavior, laying a solid foundation for improving driving safety. Furthermore, it can also support single-task detection of only distracted driving behavior detection or only fatigued driving behavior detection, improving the flexibility and intelligence of the driving behavior detection process.

[0109] Corresponding to the training method of the driving behavior detection system provided in the above embodiments, an embodiment of this disclosure also provides a training device for the driving behavior detection system. Since the training device for the driving behavior detection system provided in this embodiment corresponds to the training method of the driving behavior detection system provided in the above embodiments, the implementation method of the training method for the driving behavior detection system is also applicable to the training device for the driving behavior detection system provided in this embodiment, and will not be described in detail in this embodiment.

[0110] Figure 15 This is a schematic diagram of the structure of a training device for a driving behavior detection system according to an embodiment of the present disclosure.

[0111] like Figure 15 As shown, the training device 3000 of the driving behavior detection system includes: a first acquisition unit 1510, a detection unit 1520, a second acquisition unit 1530, and a determination unit 1540.

[0112] The system comprises the following components: a first acquisition unit 1510, used to acquire training samples, including multi-view detection data and target view detection data, as well as reference detection results corresponding to the multi-view detection data and target view detection data; a detection unit 1520, used to input the training samples into the joint multi-task driving behavior detection system to be trained, whereby the corresponding model in the joint multi-task driving behavior detection system to be trained performs behavior detection and outputs the detection results corresponding to the training samples, wherein the joint multi-task driving behavior detection system to be trained includes a distracted driving behavior detection model and a fatigued driving behavior detection model; a second acquisition unit 1530, used to acquire a first loss function corresponding to the distracted driving behavior detection model and a second loss function corresponding to the fatigued driving behavior detection model based on the detection results and reference detection results; and a determination unit 1540, used to acquire the total loss function of the joint multi-task driving behavior detection system to be trained based on the first and second loss functions, and adjust the parameters of the joint multi-task driving behavior detection system to be trained based on the total loss function until the training termination condition is met, and determine the joint multi-task driving behavior detection system to be trained after the last parameter adjustment as the joint multi-task driving behavior detection system.

[0113] According to an embodiment of this disclosure, the determining unit 1540 is further configured to: for any loss function, use the Dynamic Weighted Average (DWA) algorithm to obtain the relative rate of change of loss for at least two consecutive periods, and perform exponential normalization on the relative rate of change of loss to obtain the weight of any loss function; and obtain the total loss function based on the weight corresponding to each loss function.

[0114] According to an embodiment of this disclosure, a training apparatus for a driving behavior detection system can acquire training samples and input them into a joint multi-task driving behavior detection system to be trained. The corresponding model in the joint multi-task driving behavior detection system performs behavior detection and outputs detection results corresponding to the training samples. Then, based on the detection results and reference detection results, a first loss function corresponding to the distracted driving behavior detection model and a second loss function corresponding to the fatigued driving behavior detection model are obtained. Based on the first and second loss functions, the total loss function of the joint multi-task driving behavior detection system to be trained is obtained. The parameters of the joint multi-task driving behavior detection system to be trained are adjusted according to the total loss function until the training termination condition is met. The joint multi-task driving behavior detection system after the last parameter adjustment is determined as the joint multi-task driving behavior detection system. Therefore, this embodiment of the disclosure can obtain a well-trained joint multi-task driving behavior detection system with a stable convergence state based on an effective model training method, ensuring the accuracy of driving behavior detection results. This lays a solid foundation for efficient and accurate joint multi-task detection of two typical non-standard driving behaviors—distracted driving behavior and fatigued driving behavior—and improves driving safety. In addition, it can support single-task detection, such as detecting only distracted driving behavior or only fatigued driving behavior, which improves the flexibility and intelligence of the driving behavior detection process.

[0115] The method and apparatus are based on the same concept of the application. Since the methods and apparatus solve problems in similar ways, the implementation of the apparatus and methods can refer to each other, and the repeated parts will not be described again.

[0116] It should be noted that the division of units in the embodiments of this disclosure is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0117] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] According to embodiments of this disclosure, this disclosure also provides an electronic device 4000, such as... Figure 16 As shown, it includes a memory 400, a processor 500, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the aforementioned driving behavior detection method and driving behavior detection system training method.

[0119] According to embodiments of this disclosure, a processor-readable storage medium is also provided. This processor-readable storage medium stores a computer program that causes the processor to execute the aforementioned driving behavior detection method and driving behavior detection system training method.

[0120] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0121] According to embodiments of this disclosure, this disclosure also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the aforementioned driving behavior detection method and driving behavior detection system training method.

[0122] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0123] The specific embodiments described herein do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for detecting driving behavior, characterized in that, include: Obtain the image dataset of the user to be detected; Obtain the driving behavior detection type for the user to be detected, wherein the driving behavior detection type includes distracted driving behavior detection, fatigued driving behavior detection, and joint detection of distracted driving behavior detection and fatigued driving behavior detection; Based on the image dataset and the driving behavior detection type, determine the data to be detected; The data to be detected is input into the joint multi-task driving behavior detection system, and the corresponding model in the joint multi-task driving behavior detection system performs behavior detection and outputs the driving behavior detection results of the user to be detected. The joint multi-task driving behavior detection system includes a distracted driving behavior detection model and a fatigued driving behavior detection model.

2. The method according to claim 1, characterized in that, In response to the driving behavior detection type being the joint detection of distracted driving behavior and fatigued driving behavior, the step of determining the data to be detected based on the image dataset and the driving behavior detection type includes: Obtain a first selection strategy for detecting distracted driving behavior and a second selection strategy for detecting fatigued driving behavior; According to the first selection strategy, multi-view detection data is obtained, and according to the second selection strategy, target view detection data is obtained.

3. The method according to claim 2, characterized in that, The step of inputting the data to be detected into the joint multi-task driving behavior detection system, whereby the corresponding model in the joint multi-task driving behavior detection system performs behavior detection and outputs the driving behavior detection result of the user to be detected, includes: The multi-view detection data and the target view detection data are input into the joint multi-task driving behavior detection system. The backbone network of the joint multi-task driving behavior detection system extracts features from the multi-view detection data and the target view detection data respectively to obtain multi-view detection data features and target view detection data features. The multi-view detection data features include a feature matrix and a vertex matrix. The feature matrix includes the features extracted from the vertices of the multi-view detection data. The vertex matrix is ​​designed according to the spatial position of the multi-view detection data. The multi-view data features to be detected are input into the distracted driving behavior detection model, which performs distracted driving behavior detection based on the multi-view data features to obtain a first intermediate detection result. The target view data features to be detected are input into the fatigue driving behavior detection model, which performs fatigue driving behavior detection based on the target view data to be detected to obtain a second intermediate detection result. Based on the first intermediate detection result and the second intermediate detection result, the driving behavior detection result is determined and output.

4. The method according to claim 3, characterized in that, The step of determining and outputting the driving behavior detection result based on the first intermediate detection result and the second intermediate detection result includes: The shooting frequency of the multi-view image dataset is obtained, and the joint judgment frequency for the first intermediate detection result and the second intermediate detection result is determined based on the shooting frequency. Based on the joint judgment frequency, obtain the first number of times the first intermediate detection result shows an anomaly and the second number of times the second intermediate detection result shows an anomaly within any target period; In response to the first count and / or the second count meeting a preset condition, the driving behavior detection result is determined to be abnormal and output.

5. The method according to claim 3, characterized in that, The process of detecting distracted driving behavior by the distracted driving behavior detection model based on the features of the multi-view data to be detected, and obtaining a first intermediate detection result, includes: The feature fusion layer of the distracted driving behavior detection model performs feature fusion processing on the multi-view data to be detected, obtains new aggregated features, and inputs them into the first classification layer of the distracted driving behavior detection model. The first classification layer classifies the aggregated new features to obtain the first intermediate detection result.

6. The method according to claim 3, characterized in that, The process of detecting fatigue driving behavior by the fatigue driving behavior detection model based on the target viewpoint data to obtain a second intermediate detection result includes: The second classification layer of the fatigue driving behavior detection model classifies the data to be detected from the target perspective to obtain the second intermediate detection result.

7. A training method for a driving behavior detection system, characterized in that, include: Obtain training samples, wherein the training samples include multi-view detection data and target view detection data, as well as reference detection results corresponding to the multi-view detection data and the target view detection data; The training samples are input into the joint multi-task driving behavior detection system to be trained, and the corresponding model in the joint multi-task driving behavior detection system to be trained performs behavior detection and outputs the detection results corresponding to the training samples. The joint multi-task driving behavior detection system to be trained includes a distracted driving behavior detection model and a fatigued driving behavior detection model. Based on the detection results and the reference detection results, obtain the first loss function corresponding to the distracted driving behavior detection model and the second loss function corresponding to the fatigued driving behavior detection model; Based on the first loss function and the second loss function, the total loss function of the joint multi-task driving behavior detection system to be trained is obtained, and the parameters of the joint multi-task driving behavior detection system to be trained are adjusted according to the total loss function until the training termination condition is met. The joint multi-task driving behavior detection system to be trained after the last adjustment of the parameters is determined as the joint multi-task driving behavior detection system.

8. The method according to claim 7, characterized in that, The step of obtaining the total loss function of the joint multi-task driving behavior detection system to be trained based on the first loss function and the second loss function includes: For any loss function, the Dynamic Weighted Average (DWA) algorithm is used to obtain the relative rate of change of loss for at least two consecutive periods, and the relative rate of change of loss is exponentially normalized to obtain the weight of any loss function. The total loss function is obtained based on the weights corresponding to each loss function.

9. A driving behavior detection device, characterized in that, include: The first acquisition unit is used to acquire the image dataset of the user to be detected; The second acquisition unit is used to acquire the driving behavior detection type for the user to be detected, wherein the driving behavior detection type includes distracted driving behavior detection, fatigued driving behavior detection, and joint detection of distracted driving behavior detection and fatigued driving behavior detection; The determining unit is configured to determine the data to be detected based on the image dataset and the driving behavior detection type; The detection unit is used to input the data to be detected into the joint multi-task driving behavior detection system, where the corresponding model in the joint multi-task driving behavior detection system performs behavior detection and outputs the driving behavior detection result of the user to be detected. The joint multi-task driving behavior detection system includes a distracted driving behavior detection model and a fatigued driving behavior detection model.

10. A training device for a driving behavior detection system, characterized in that, include: The first acquisition unit is used to acquire training samples, wherein the training samples include multi-view detection data and target view detection data, as well as reference detection results corresponding to the multi-view detection data and the target view detection data. The detection unit is used to input the training samples into the joint multi-task driving behavior detection system to be trained, and the corresponding model in the joint multi-task driving behavior detection system to be trained performs behavior detection and outputs the detection results corresponding to the training samples. The joint multi-task driving behavior detection system to be trained includes a distracted driving behavior detection model and a fatigued driving behavior detection model. The second acquisition unit is used to acquire, based on the detection results and the reference detection results, the first loss function corresponding to the distracted driving behavior detection model and the second loss function corresponding to the fatigued driving behavior detection model; The determining unit is configured to obtain the total loss function of the joint multi-task driving behavior detection system to be trained based on the first loss function and the second loss function, and adjust the parameters of the joint multi-task driving behavior detection system to be trained based on the total loss function until the training termination condition is met, and determine the joint multi-task driving behavior detection system to be trained after the last adjustment of the parameters as the joint multi-task driving behavior detection system.

Images