An abnormal driving detection method, device, electronic equipment and medium
By dynamically adjusting the detection strategy and multi-layer detection mechanism, and combining super-resolution networks and various neural networks, the problem of abnormal behavior recognition in driver monitoring systems under lighting changes and small object detection scenarios has been solved, achieving highly accurate and stable abnormal driving detection.
Patent Information
- Application Number
- CN202411403621.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-10-09
AI Technical Summary
Existing driver monitoring systems cannot accurately identify abnormal driver behavior in scenarios involving changes in lighting and the detection of small objects, resulting in missed detections.
The target detection strategy is dynamically adjusted based on environmental information. A multi-layer detection mechanism is adopted, including preliminary detection, detail enhancement, and final detection. Super-resolution networks are used to enhance image details, and different neural networks are combined for multi-stage processing to improve detection accuracy and stability.
It improves the accuracy and stability of abnormal driving detection in scenarios with significant changes in lighting, reduces false alarms and false negatives, and can quickly identify abnormal driver behavior and respond promptly.
Smart Images

Figure CN119478902B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and in particular to an abnormal driving detection method, device, electronic device, and medium. Background Technology
[0002] Driver monitoring systems (DMS) play a crucial role in the field of intelligent driving, improving driving safety and reducing traffic accidents by detecting driver misbehavior in real time.
[0003] Current DMS (Driver Monitoring System) technology relies on computer vision and deep learning models to analyze images captured by in-vehicle cameras and identify the driver's facial features and behavioral state. However, the anomaly detection methods of these technologies have limitations, failing to accurately identify abnormal driver behavior in scenarios with changing lighting or small object detection. Summary of the Invention
[0004] To overcome the problems existing in related technologies, this disclosure provides an abnormal driving detection method, device, electronic equipment, and medium. The technical solution of this disclosure is as follows: According to a first aspect of the present disclosure, an abnormal driving detection method is provided, comprising: Based on the environmental information corresponding to the target vehicle, a target detection strategy is determined for the preliminary detection stage of the image to be detected; the environmental information includes at least illumination information and camera accuracy information; According to the target detection strategy, abnormal behavior detection is performed on the image to be detected to obtain preliminary anomaly detection results; the preliminary anomaly detection results include: abnormal behavior detection boxes and anomaly confidence scores; Determine whether the anomaly confidence level is within the anomaly detection interval; the anomaly detection interval indicates that abnormal driving cannot be determined by the anomaly confidence level. When the anomaly confidence level is within the anomaly detection range, the anomaly behavior detection box is supplemented with details to obtain a supplemented region; Abnormal behavior detection is performed on the supplementary area to obtain the final abnormal detection result.
[0005] Optionally, based on the environmental information corresponding to the target vehicle, a target detection strategy is determined for the preliminary detection stage of the image to be detected, including: When the illumination information is lower than the illumination threshold, and / or when the camera accuracy information is lower than the accuracy threshold, the first detection strategy is determined as the target detection strategy; the first detection strategy is used to detect faces and abnormal behaviors sequentially for low-resolution images to be detected. When the illumination information is higher than or equal to the illumination threshold, and when the camera accuracy information is higher than or equal to the accuracy threshold, the second detection strategy is determined as the target detection strategy; the second detection strategy is used to detect faces and abnormal behaviors simultaneously for high-resolution images to be detected.
[0006] Optionally, according to the first detection strategy, abnormal behavior detection is performed on the image to be detected to obtain preliminary abnormal detection results, including: The driver's face is detected from the image to be detected using a first neural network, resulting in a face detection box and a face confidence score; the first neural network is used to detect faces. When the face confidence score is greater than or equal to the face threshold, the face detection box is dynamically expanded to obtain an expanded region; The extended region is subjected to abnormal behavior detection using a second neural network, the abnormal behavior detection box is output, and the abnormal confidence level corresponding to the abnormal behavior detection box is determined; the second neural network is used to detect different abnormal behaviors.
[0007] Optionally, according to the second detection strategy, abnormal behavior detection is performed on the image to be detected to obtain preliminary abnormal detection results, including: The third neural network is used to detect the driver's face and abnormal behavior from the image to be detected, and to obtain the face detection box and face confidence, and the abnormal behavior detection box and abnormal confidence. The third neural network is used to detect faces and detect different abnormal behaviors. When the face confidence score is greater than or equal to the face threshold, the face detection box is dynamically expanded to obtain an expanded region; Determine the geometric center of the abnormal behavior detection box; When the geometric center of the abnormal behavior detection box is located within the extended region, the abnormal confidence level corresponding to the abnormal behavior detection box is determined.
[0008] Optionally, it also includes: If the geometric center of the abnormal behavior detection box is not located within the extended area, it is determined to be normal driving.
[0009] Optionally, abnormal behavior detection is performed on the supplementary region to obtain the final abnormal detection result, including: The supplementary region is detected by a fourth neural network to determine the target confidence level of the supplementary region as abnormal behavior; the fourth neural network is used to determine whether the category corresponding to the abnormal behavior detection box is correct. If the target confidence level is greater than or equal to the target threshold, it is determined to be abnormal driving; If the target confidence level is less than the target threshold, it is determined to be normal driving.
[0010] Optionally, the target detection strategy includes a first detection strategy and a second detection strategy, performing abnormal behavior detection on the supplementary region to obtain a final abnormal detection result, and further includes: If all images acquired at each time point are determined to indicate abnormal driving, a first target confidence level corresponding to the first detection strategy and a second target confidence level corresponding to the second detection strategy are determined respectively; the first target confidence level and the second target confidence level respectively represent different probabilities of abnormal behavior; Assign corresponding weights to the first target confidence level and the second target confidence level; The total confidence score obtained by weighted summation of the first target confidence score and the second target confidence score; Based on the total confidence level, determine whether the driving was abnormal.
[0011] Optionally, it also includes: If the anomaly confidence level is greater than the maximum value of the anomaly detection interval, abnormal driving is determined; If the anomaly confidence level is less than the minimum value of the anomaly detection interval, normal driving is determined.
[0012] According to a second aspect of the present disclosure, an abnormal driving detection device is provided, comprising: The strategy determination module is used to determine the target detection strategy for the preliminary detection stage of the image to be detected based on the environmental information corresponding to the target vehicle; the environmental information includes at least illumination information and camera accuracy information. The first detection module is used to perform abnormal behavior detection on the image to be detected according to the target detection strategy, and obtain preliminary abnormal detection results; the preliminary abnormal detection results include: abnormal behavior detection boxes and abnormal confidence scores; The judgment module is used to determine whether the anomaly confidence level is within the anomaly detection interval; the anomaly detection interval indicates that it is impossible to determine whether abnormal driving is occurring based on the anomaly confidence level. The supplementary module is used to supplement the abnormal behavior detection box with details when the anomaly confidence level is within the anomaly detection range, so as to obtain a supplementary area; The second detection module is used to detect abnormal behavior in the supplementary area and obtain the final abnormal detection result.
[0013] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when executed by the processor, the computer program implements the steps of the abnormal driving detection method as described in the first aspect.
[0014] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the abnormal driving detection method as described in the first aspect.
[0015] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the abnormal driving detection method as described in the first aspect.
[0016] This disclosure determines a target detection strategy based on environmental information, enabling it to cope with different driving environments, especially in scenarios with significant lighting variations, thereby improving detection accuracy and stability. For behaviors with anomaly confidence levels within the detection range, the system further supplements the details to obtain supplementary regions, and then re-detects them, which can improve the ability to capture subtle abnormal behaviors, especially in small object detection or complex action recognition, significantly improving detection accuracy and precision. This disclosure can quickly identify abnormal driver behavior in real-time video streams and respond promptly. Attached Figure Description
[0017] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments of this disclosure will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a schematic diagram illustrating the steps of an abnormal driving detection method according to an embodiment of this disclosure; Figure 2 This is a flowchart illustrating an abnormal driving detection method according to an embodiment of this disclosure; Figure 3 This is a flowchart illustrating an abnormal driving detection process using a first detection strategy, as shown in an embodiment of this disclosure. Figure 4 This is a flowchart illustrating an abnormal driving detection process employing a second detection strategy, as shown in an embodiment of this disclosure. Figure 5 This is a block diagram illustrating an abnormal driving detection device according to an embodiment of this disclosure; Figure 6This is a schematic diagram of an electronic device shown in an embodiment of this disclosure. Detailed Implementation
[0019] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0020] The terms "first," "second," etc., used in this disclosure and in the claims are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this disclosure can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0021] Driver monitoring systems play a crucial role in the field of intelligent driving, improving driving safety and reducing traffic accidents by detecting undesirable driver behaviors such as smoking and using mobile phones in real time. Driver monitoring systems are becoming increasingly important.
[0022] In related technologies, driver monitoring systems primarily rely on computer vision and deep learning models to identify the driver's facial features and behavioral state through images captured by in-vehicle cameras. However, the anomaly detection methods in these technologies have limitations, failing to accurately identify abnormal driver behavior in scenarios with changing lighting or small object detection.
[0023] Specifically, DMS systems in related technologies typically employ convolutional neural networks for face detection and behavior recognition, combined with pose estimation and gesture recognition techniques to determine whether a driver is smoking, making a phone call, or engaging in other behaviors. When detecting small objects such as cigarettes and mobile phones, some DMS systems rely on multi-scale feature extraction or contextual information to improve detection accuracy. However, in real-world driving scenarios, small objects like cigarettes or mobile phones can be obstructed, have low resolution, or exhibit angular variations, all of which can affect detection accuracy and lead to missed detections.
[0024] To address the aforementioned technical problems, this disclosure proposes an abnormal driving detection method. This method can effectively improve the detection accuracy of small objects in situations involving long distances and complex backgrounds, and avoid missing detections of abnormal driver behavior.
[0025] Figure 1 This is a schematic diagram illustrating the steps of an abnormal driving detection method according to an embodiment of this disclosure. Figure 1 As shown, the abnormal driving detection method may specifically include the following steps: Step S11: Determine the target detection strategy for the preliminary detection stage of the image to be detected based on the environmental information corresponding to the target vehicle; the environmental information includes at least illumination information and camera accuracy information.
[0026] Environmental information includes lighting conditions and camera resolution, enabling the acquisition of images at different resolutions under varying environmental conditions. To adapt to images of varying resolutions, different target detection strategies can be used for detection.
[0027] Preliminary image detection can be performed using target detection strategies adapted to image sharpness. Different detection strategies may correspond to different detection network complexities, enabling efficient use of computational resources.
[0028] Step S12: According to the target detection strategy, perform abnormal behavior detection on the image to be detected to obtain preliminary abnormal detection results; the preliminary abnormal detection results include: abnormal behavior detection boxes and abnormal confidence scores.
[0029] The images to be detected are captured by an in-vehicle camera. Based on a target detection strategy, it is determined whether any abnormal driver behavior exists in the images. Abnormal behavior includes actions such as answering a phone call or smoking. Abnormal driver behavior can be detected by analyzing the positional relationship between the driver and the corresponding unusual objects.
[0030] The determination of whether a driver is behaving abnormally is based on the anomaly confidence level of the abnormal behavior. The anomaly confidence level represents the probability of abnormal behavior in the image to be detected. When the probability is too high, it indicates that there is a high probability of abnormal behavior, and when the probability is too low, it indicates that there is a high probability of normal behavior.
[0031] Step S13: Determine whether the anomaly confidence level is within the anomaly detection interval; the anomaly detection interval indicates that it is impossible to determine whether abnormal driving is occurring based on the anomaly confidence level.
[0032] An anomaly detection interval is determined. If the anomaly confidence level falls within this interval, it indicates that the presence of abnormal behavior in the image to be detected cannot be confirmed. The anomaly detection interval is preset; it can be determined by checking whether the anomaly confidence level falls within the anomaly detection interval when determining the anomaly confidence level for each abnormal behavior.
[0033] Step S14: When the anomaly confidence level is within the anomaly detection range, the anomaly behavior detection box is supplemented with details to obtain a supplemented area.
[0034] If the anomaly confidence level falls within the anomaly detection range, further analysis of the anomalous behavior is required. To reduce computational load, anomaly detection boxes corresponding to the anomalous behavior are extracted, and these boxes are then enhanced with more details. Specifically, the image information corresponding to the anomaly detection boxes can be input into a super-resolution network, which generates a corresponding high-resolution image to enhance the details. The super-resolution network can be an SRCNN (Super-Resolution Convolutional Neural Network).
[0035] The supplementary area is the abnormal behavior detection box after detailed supplementation.
[0036] Step S15: Perform abnormal behavior detection on the supplementary area to obtain the final abnormal detection result.
[0037] Further anomaly detection is performed on the supplementary region to determine the target confidence level for anomalous behavior. The target confidence level represents the probability that anomalous behavior exists in the supplementary region.
[0038] Typically, an anomaly threshold can be set. If the target confidence level is greater than or equal to the anomaly threshold, it indicates the presence of abnormal behavior, thus determining that the driver in the image is driving abnormally. If the target confidence level is less than the anomaly threshold, it indicates the absence of abnormal behavior, thus determining that the driver in the image is driving normally.
[0039] By employing embodiments of this disclosure, the system can dynamically adjust its target detection strategy through real-time analysis of environmental information to address challenges in different scenarios, maintaining high detection accuracy under varying conditions. Based on advanced algorithms such as deep learning, the system can calculate the anomaly confidence level of each abnormal behavior in the image to be detected, distinguishing between normal driving behavior and potentially dangerous behavior. When the driver's current behavior cannot be determined through anomaly confidence level, the abnormal behavior detection box can be supplemented with details, further amplifying and enhancing the features of key regions for more in-depth detection. The multi-layer detection mechanism proposed in this disclosure helps reduce false positives and false negatives, improving the accuracy and reliability of detection.
[0040] In one optional embodiment, the method further includes: determining abnormal driving when the anomaly confidence level is greater than the maximum value of the anomaly detection interval; and determining normal driving when the anomaly confidence level is less than the minimum value of the anomaly detection interval.
[0041] When the anomaly confidence level is on both sides of the anomaly detection interval, it indicates that the detection result of the abnormal behavior has been determined and the detection result is accurate.
[0042] If the anomaly confidence level is less than the minimum value of the anomaly detection interval, the current driving behavior is within the normal range, and the system should determine it as normal driving. If the anomaly confidence level is greater than the maximum value of the anomaly detection interval, the current driving behavior is highly abnormal, and the system should determine it as abnormal driving. Specifically, when the anomaly detection interval is [0.3, 0.4], an anomaly confidence level less than the minimum value of the anomaly detection interval means the anomaly confidence level is less than 0.3, in which case it is directly determined as normal driving; an anomaly confidence level greater than the maximum value of the anomaly detection interval means the anomaly confidence level is greater than 0.4, in which case it is directly determined as abnormal driving.
[0043] By employing the embodiments of this disclosure, when the initial detection shows that the confidence level of two anomalies is not within the anomaly detection range, the driver's driving behavior in the image to be detected can be directly determined, which can reduce the amount of computation, improve detection efficiency, and save computing resources.
[0044] Based on environmental information, target detection strategies can be divided into a first detection strategy and a second detection strategy. The first and second detection strategies require different levels of computing resources and have different advantages. For example, the first detection strategy is suitable for complex environments and situations with low camera accuracy; the second detection strategy is suitable for situations with limited onboard computing resources and high real-time requirements. In other words, the first detection strategy is highly accurate but resource-intensive, while the second detection strategy is lightweight and fast but requires the image to meet certain clarity requirements.
[0045] In real-world vehicle usage scenarios, the target detection strategy can be selected based on the time period. For example, in summer, the second detection strategy is used from 7:00 AM to 7:00 PM, and the first detection strategy is used for the remaining time; in winter, the second detection strategy is used from 9:00 AM to 5:00 PM, and the first detection strategy is used for the remaining time.
[0046] In one optional embodiment, a target detection strategy is determined for the preliminary detection stage of the image to be detected based on the environmental information corresponding to the target vehicle. This includes: determining a first detection strategy as the target detection strategy when the illumination information is lower than an illumination threshold and / or when the camera accuracy information is lower than an accuracy threshold; the first detection strategy is used to detect faces and abnormal behaviors sequentially for low-resolution images to be detected; and determining a second detection strategy as the target detection strategy when the illumination information is higher than or equal to the illumination threshold and when the camera accuracy information is higher than or equal to the accuracy threshold; the second detection strategy is used to detect faces and abnormal behaviors simultaneously for high-resolution images to be detected.
[0047] The illumination threshold and accuracy threshold can be preset. Illumination information below the preset illumination threshold indicates that the environment is too dark and may affect image quality, while camera accuracy information below the preset accuracy threshold indicates that camera performance is limited.
[0048] When the illumination information is below the illumination threshold, and / or when the camera accuracy information is below the accuracy threshold, the clarity of the acquired image to be detected is low. Therefore, a first detection strategy with higher accuracy is adopted as the target detection strategy. The first detection strategy sequentially checks for faces and abnormal behaviors in the image to be detected. It first detects whether a face exists in the image to be detected, and then detects abnormal behaviors after the condition is met. This strategy can well adapt to the detection needs of low-resolution images to be detected. Two different neural networks can be used to perform the detection operation. The first neural network detects faces, and when the condition is met, it transmits its output to the second neural network for abnormal behavior detection.
[0049] When the illumination information is higher than or equal to the illumination threshold, and when the camera accuracy information is higher than or equal to the accuracy threshold, it indicates that the image to be detected acquired under these conditions has high clarity. Therefore, the second detection strategy is selected for preliminary detection. The second detection strategy can simultaneously detect faces and abnormal behaviors from the image to be detected. A single neural network can be used to perform the detection operation, simultaneously identifying faces and abnormal behaviors in the image to be detected.
[0050] Using embodiments of this disclosure, when image quality is low, employing a first detection strategy as the target detection strategy can improve detection capabilities in low-light or low-quality images. When both illumination and camera accuracy meet or exceed preset thresholds, employing a second detection strategy can improve detection accuracy and efficiency.
[0051] Figure 2 This is a flowchart illustrating an abnormal driving detection method according to an embodiment of this disclosure. Figure 2 As shown, the environmental information corresponding to the target vehicle when the image to be detected is acquired is determined; based on the environmental information, different detection strategies are used to detect whether the driver in the image to be detected exhibits abnormal behavior; if a first condition is met, the first detection strategy is used to perform abnormal behavior detection in the preliminary detection stage; if a second condition is met, the second detection strategy is used to perform abnormal behavior detection in the preliminary detection stage; through the first or second detection strategy, a preliminary abnormal detection result is obtained; the preliminary abnormal detection result includes: an abnormal behavior detection box and an abnormal confidence score; it is determined whether the abnormal confidence score is within the abnormal detection interval; if the abnormal confidence score is within the abnormal detection interval, the abnormal behavior detection box is supplemented with details to obtain a supplemented area, and abnormal behavior detection is performed on the supplemented area to obtain the final abnormal detection result; if the abnormal confidence score is not within the abnormal detection interval, it is determined whether the abnormal confidence score is greater than the maximum value of the abnormal detection interval; if it is greater than the maximum value of the abnormal detection interval, it is determined that the driver in the image to be detected is driving abnormally; if it is not greater than the maximum value of the abnormal detection interval, it is determined that the driver in the image to be detected is driving normally.
[0052] (1) The following is the process of preliminary detection using the first detection strategy.
[0053] In one optional embodiment, according to the first detection strategy, abnormal behavior detection is performed on the image to be detected to obtain preliminary abnormal detection results, including: detecting the driver's face from the image to be detected using a first neural network to obtain a face detection box and a face confidence score; the first neural network is used to detect faces; when the face confidence score is greater than or equal to a face threshold, the face detection box is dynamically expanded to obtain an expanded region; abnormal behavior detection is performed on the expanded region using a second neural network to output the abnormal behavior detection box and determine the abnormal confidence score corresponding to the abnormal behavior detection box; the second neural network is used to detect different abnormal behaviors.
[0054] When employing a first detection strategy as the target detection strategy, a first neural network detects the driver's face from the image to be detected. The first neural network can be a convolutional neural network (CNN), specifically the classic face detection model MTCNN (Multi-task Cascaded Convolutional Networks), to obtain the bounding box of the face. The first neural network for face detection has two heads: a regression head to output the detection box and a classification head to output the class confidence score; this classification head is binary.
[0055] Face confidence is used to characterize the probability that a face detection box contains a face. If the face confidence is greater than or equal to the face threshold, it means that the face detection box contains a face; if the face confidence is less than the face threshold, it means that the face detection box does not contain a face.
[0056] When the face confidence score is greater than or equal to the face threshold (i.e., when the face detection bounding box contains a face), the face detection bounding box is dynamically expanded to obtain an expanded region. Specifically, based on the center point of the face detection bounding box, it can be dynamically expanded by a factor of 1.5 to obtain the expanded region. This expanded region is then used as the key monitoring area. The expanded region needs to include areas where abnormal behavior may occur; specifically, it can cover areas where the driver might be holding a cigarette or a phone.
[0057] A second neural network is used to further detect abnormal behavior in the extended region. This second neural network can be YOLOv5 (You Only Look Once version 5), which determines the abnormal behavior detection boxes in the extended region and the corresponding anomaly confidence scores. The anomaly confidence score represents the probability that an abnormal behavior exists within the abnormal behavior detection box.
[0058] The second neural network can output various detection boxes, each corresponding to a different category and confidence level. Specifically, these include categories like smoking, making a phone call, normal behavior, and smoking and making a phone call. When a detection box is classified as abnormal behavior (e.g., smoking, making a phone call, smoking and making a phone call), the box is identified as an abnormal behavior detection box, and its corresponding confidence level is defined as an abnormal confidence level. The abnormal confidence level is then used to determine whether further detection is needed.
[0059] By employing embodiments of this disclosure, combining multi-stage processing for face detection and abnormal behavior detection, and utilizing a deep learning-based neural network model, the system can accurately identify abnormal driver behavior. It can be executed rapidly within a real-time video stream and can handle a certain degree of image quality variation.
[0060] Figure 3 This is a flowchart illustrating an abnormal driving detection process employing a first detection strategy, as shown in an embodiment of this disclosure. Figure 3As shown, when the illumination information is lower than the illumination threshold, and / or when the camera accuracy information is lower than the accuracy threshold, the first detection strategy is determined as the target detection strategy; the driver's face is detected from the image to be detected by the first neural network to obtain a face detection box and a face confidence score; the driver's face is detected from the image to be detected by the first neural network to obtain a face detection box and a face confidence score; it is determined whether the face confidence score is greater than or equal to the face threshold; when the face confidence score is greater than or equal to the face threshold, the face detection box is dynamically expanded to obtain an expanded region; abnormal behavior detection is performed on the expanded region by the second neural network, the abnormal behavior detection box is output, and the abnormal confidence score corresponding to the abnormal behavior detection box is determined; when the face confidence score is less than the face threshold, it is determined that the image to be detected is normal driving and there is no abnormal behavior.
[0061] (2) The following is the process of preliminary detection using the second detection strategy.
[0062] In one optional embodiment, according to the second detection strategy, abnormal behavior detection is performed on the image to be detected to obtain preliminary abnormal detection results, including: detecting driver's face and abnormal behavior from the image to be detected using a third neural network to obtain a face detection box and face confidence, an abnormal behavior detection box and an abnormal confidence; the third neural network is used to detect faces and detect different abnormal behaviors; when the face confidence is greater than or equal to a face threshold, the face detection box is dynamically expanded to obtain an expanded region; the geometric center of the abnormal behavior detection box is determined; when the geometric center of the abnormal behavior detection box is located within the expanded region, the abnormal confidence corresponding to the abnormal behavior detection box is determined.
[0063] The third neural network can be YOLOv5 (You Only Look Once version 5), which outputs various bounding boxes. The categories corresponding to the bounding boxes can be faces and abnormal behaviors. When outputting each bounding box, the confidence score of the corresponding category for each bounding box is also output simultaneously. Specifically, it can output face bounding boxes, face confidence scores, abnormal behavior bounding boxes, and abnormal behavior confidence scores.
[0064] If the face confidence score corresponding to the face detection box is greater than or equal to the face threshold, the face detection box is dynamically expanded to obtain the expanded region.
[0065] Next, the positional relationship between the extended region and the abnormal behavior detection box is determined. Specifically, the geometric center of the abnormal behavior detection box can be determined first, and then the positional relationship between this geometric center and the extended region can be determined. If the geometric center is located within the extended region, the abnormal behavior detection box is considered to be inside the extended region; if the geometric center is located outside the extended region, the abnormal behavior detection box is considered to be outside the extended region.
[0066] If the geometric center of the abnormal behavior detection box is located within the extended region, the abnormal confidence level corresponding to the abnormal behavior detection box is determined, and the abnormal confidence level is used to determine whether further detection is needed.
[0067] By employing embodiments of this disclosure, under favorable lighting and camera conditions, the second detection strategy can more effectively utilize computing resources, reducing computation time or lowering hardware resource requirements without sacrificing detection performance.
[0068] Figure 4 This is a flowchart illustrating an abnormal driving detection process employing a second detection strategy, as shown in an embodiment of this disclosure. Figure 4 As shown. When the illumination information is higher than or equal to the illumination threshold, and when the camera accuracy information is higher than or equal to the accuracy threshold, the second detection strategy is determined as the target detection strategy; the driver's face and abnormal behavior are detected from the image to be detected by the third neural network to obtain a face detection box and face confidence, an abnormal behavior detection box and anomaly confidence; it is determined whether the face confidence is greater than or equal to the face threshold; if the face confidence is less than the face threshold, it is determined to be normal driving; if the face confidence is greater than or equal to the face threshold, the face detection box is dynamically expanded to obtain an expanded region, and the geometric center of the abnormal behavior detection box is determined; it is determined whether the geometric center of the abnormal behavior detection box is located within the expanded region; if the geometric center of the abnormal behavior detection box is not located within the expanded region, it is determined to be normal driving; if the geometric center of the abnormal behavior detection box is located within the expanded region, the anomaly confidence corresponding to the abnormal behavior detection box is determined.
[0069] In one optional embodiment, the method further includes determining normal driving if the geometric center of the abnormal behavior detection box is not located within the extended area.
[0070] The extended region includes areas where the driver might exhibit abnormal behavior. Therefore, if the geometric center of the abnormal behavior detection box is not located within the extended region, that is, if the abnormal behavior detection box is located outside the extended region, it can be directly determined that the driver in the image to be detected is driving normally.
[0071] By employing the embodiments of this disclosure, when the geometric center of the abnormal behavior detection box is not located within the extended area, it can be directly determined that the driver is driving normally, which can effectively improve detection efficiency, improve the real-time performance of detection, and save computing resources.
[0072] Both the first and second detection strategies require that subsequent operations be performed only if the face confidence score corresponding to the face detection box is greater than or equal to the face threshold. If the face confidence score corresponding to the face detection box is less than the face threshold, it is determined that there is no driver in the image to be detected, and it is determined that the current driving is normal.
[0073] In some cases, both the first and second detection strategies are used simultaneously to determine whether a driver in the image is driving abnormally. This is because different target detection strategies used in the initial detection can affect the final detection result, that is, the result of detecting abnormal driving.
[0074] In one optional embodiment, the target detection strategy includes a first detection strategy and a second detection strategy. Abnormal behavior detection is performed on the supplementary region to obtain a final abnormal detection result. The strategy further includes: if all images acquired at various times indicate abnormal driving, determining a first target confidence level corresponding to the first detection strategy and a second target confidence level corresponding to the second detection strategy; the first target confidence level and the second target confidence level respectively represent different probabilities of abnormal behavior; assigning corresponding weights to the first target confidence level and the second target confidence level; calculating a total confidence level by weighted summation of the first target confidence level and the second target confidence level; and determining whether abnormal driving is present based on the total confidence level.
[0075] Normally, when abnormal driving is detected, the driver will be alerted. If abnormal driving is detected repeatedly after alerting the driver using the first or second detection strategy, then there may be an error in using a single first or second detection strategy, leading to the repeated detection of abnormal behavior.
[0076] Therefore, for the same image to be detected, either the first detection strategy or the second detection strategy can be used simultaneously for preliminary detection.
[0077] The first detection strategy is used to perform preliminary detection on the image to be detected, and a first abnormal behavior detection box is obtained; the first abnormal behavior detection box is supplemented with details to obtain a first supplemented region; the first supplemented region is detected to obtain a first target confidence score.
[0078] The second detection strategy is used to perform preliminary detection on the image to be detected, and a second abnormal behavior detection box is obtained; the second abnormal behavior detection box is supplemented with details to obtain a second supplemented region; the second supplemented region is detected to obtain a second target confidence score.
[0079] Based on the first target confidence score and the second target confidence score, it is determined whether abnormal driving has occurred. This can be achieved by assigning weights to the results corresponding to the first and second detection strategies; subsequently, the first and second target confidence scores are weighted and summed to obtain the total confidence score.
[0080] Generally, the first detection strategy has higher accuracy, therefore a higher weight can be assigned to the first target confidence level corresponding to the first detection strategy. The sum of the weights assigned to the first detection strategy and the weights assigned to the second detection strategy is 1. Specifically, a weight of 0.6 can be assigned to the first target confidence level; a weight of 0.4 can be assigned to the second target confidence level; c = 0.6a + 0.4b, where the first target confidence level is denoted as a, the second target confidence level as b, and the total confidence level as c.
[0081] The overall confidence level is used to determine whether abnormal driving has occurred. Specifically, the overall confidence level is compared with a preset threshold. If the overall confidence level is greater than or equal to the threshold, it indicates abnormal driving; if it is less than the threshold, it indicates normal driving.
[0082] Using embodiments of this disclosure, a single detection strategy may lead to false positives or false negatives due to limitations in the algorithm itself or the complexity of specific scenarios. By combining two or more detection strategies, they can complement each other, reducing errors caused by the shortcomings of a single strategy and thus improving the overall detection accuracy. The combination of multiple detection strategies enables the system to cope with a wider range of driving scenarios and abnormal behavior types. Even if a certain strategy fails under specific circumstances, other strategies can still provide effective detection, thereby enhancing the stability and reliability of the system. Assigning different weights to different detection strategies can be adjusted according to actual needs to adapt to different driving environments and safety requirements.
[0083] In one optional embodiment, abnormal behavior detection is performed on the supplementary region to obtain a final abnormal detection result, including: detecting the supplementary region through a fourth neural network to determine the target confidence level of the supplementary region as abnormal behavior; the fourth neural network is used to determine whether the category corresponding to the abnormal behavior detection box is correct; if the target confidence level is greater than or equal to the target threshold, it is determined to be abnormal driving; if the target confidence level is less than the target threshold, it is determined to be normal driving.
[0084] The fourth neural network can be a classification network, specifically MobileNet v2 (MobileNet: Inverted Residuals and Linear Bottlenecks v2).
[0085] The supplementary area is an area that may include abnormal behavior, and the category of abnormal behavior has already been identified in the initial detection phase, such as smoking or making a phone call; therefore, the purpose of further detection in the supplementary area is to confirm the accuracy of the initial detection. Thus, it is only necessary to determine whether the behavior is normal or abnormal.
[0086] The fourth neural network can obtain two possible categories for the supplementary region: normal behavior and abnormal behavior. For the supplementary region classified as abnormal behavior, its corresponding category probability is determined as the target confidence level.
[0087] If the target confidence level is greater than or equal to the target threshold, the driving is determined to be abnormal; if the target confidence level is less than the target threshold, the driving is determined to be normal. Typically, the target threshold can be set to 0.5.
[0088] By employing embodiments of this disclosure, a single neural network is used to detect supplementary regions, which can verify the results of the preliminary detection stage, thereby increasing the robustness of the entire detection process, reducing false positives and false negatives, and improving system reliability. Determining the final result by comparing the target confidence level with a preset target threshold makes the entire decision-making process more intuitive and easier to understand.
[0089] Figure 5 This is a block diagram illustrating an abnormal driving detection device according to an embodiment of this disclosure. Figure 5 As shown, the abnormal driving detection device includes: The strategy determination module 510 is used to determine a target detection strategy for the preliminary detection stage of the image to be detected based on the environmental information corresponding to the target vehicle; the environmental information includes at least illumination information and camera accuracy information. The first detection module 520 is used to perform abnormal behavior detection on the image to be detected according to the target detection strategy, and obtain preliminary abnormal detection results; the preliminary abnormal detection results include: abnormal behavior detection boxes and abnormal confidence scores; The judgment module 530 is used to determine whether the anomaly confidence level is within the anomaly detection interval; the anomaly detection interval indicates that it is impossible to determine whether abnormal driving is occurring based on the anomaly confidence level. Supplement module 540 is used to supplement the abnormal behavior detection box with details when the anomaly confidence level is within the anomaly detection range, so as to obtain a supplementary area; The second detection module 550 is used to detect abnormal behavior in the supplementary area and obtain the final abnormal detection result.
[0090] Optionally, the strategy determination module is specifically used to perform: When the illumination information is lower than the illumination threshold, and / or when the camera accuracy information is lower than the accuracy threshold, the first detection strategy is determined as the target detection strategy; the first detection strategy is used to detect faces and abnormal behaviors sequentially for low-resolution images to be detected. When the illumination information is higher than or equal to the illumination threshold, and when the camera accuracy information is higher than or equal to the accuracy threshold, the second detection strategy is determined as the target detection strategy; the second detection strategy is used to detect faces and abnormal behaviors simultaneously for high-resolution images to be detected.
[0091] The first detection module is specifically used to perform: The driver's face is detected from the image to be detected using a first neural network, resulting in a face detection box and a face confidence score; the first neural network is used to detect faces. When the face confidence score is greater than or equal to the face threshold, the face detection box is dynamically expanded to obtain an expanded region; The extended region is subjected to abnormal behavior detection using a second neural network, the abnormal behavior detection box is output, and the abnormal confidence level corresponding to the abnormal behavior detection box is determined; the second neural network is used to detect different abnormal behaviors.
[0092] Optionally, the first detection module is specifically used to perform: The third neural network is used to detect the driver's face and abnormal behavior from the image to be detected, and to obtain the face detection box and face confidence, and the abnormal behavior detection box and abnormal confidence. The third neural network is used to detect faces and detect different abnormal behaviors. When the face confidence score is greater than or equal to the face threshold, the face detection box is dynamically expanded to obtain an expanded region; Determine the geometric center of the abnormal behavior detection box; When the geometric center of the abnormal behavior detection box is located within the extended region, the abnormal confidence level corresponding to the abnormal behavior detection box is determined.
[0093] Optionally, the abnormal driving detection device further includes: The first determination submodule is used to determine normal driving when the geometric center of the abnormal behavior detection box is not located within the extended area.
[0094] Optionally, the supplementary module is specifically used to perform: The supplementary region is detected by a fourth neural network to determine the target confidence level of the supplementary region as abnormal behavior; the fourth neural network is used to determine whether the category corresponding to the abnormal behavior detection box is correct. If the target confidence level is greater than or equal to the target threshold, it is determined to be abnormal driving; If the target confidence level is less than the target threshold, it is determined to be normal driving.
[0095] Optionally, the target detection strategy includes a first detection strategy and a second detection strategy, wherein the second detection module is specifically used to execute: If all images acquired at each time point are determined to indicate abnormal driving, a first target confidence level corresponding to the first detection strategy and a second target confidence level corresponding to the second detection strategy are determined respectively; the first target confidence level and the second target confidence level respectively represent different probabilities of abnormal behavior; Assign corresponding weights to the first target confidence level and the second target confidence level; The total confidence score obtained by weighted summation of the first target confidence score and the second target confidence score; Based on the total confidence level, determine whether the driving was abnormal.
[0096] Optionally, the abnormal driving detection device further includes: The second determination submodule is used to determine abnormal driving when the anomaly confidence level is greater than the maximum value of the anomaly detection interval; The third determination submodule is used to determine normal driving when the anomaly confidence level is less than the minimum value of the anomaly detection interval.
[0097] It should be noted that the device embodiments are similar to the method embodiments, so the description is relatively simple. For relevant details, please refer to the method embodiments.
[0098] This disclosure also provides an electronic device, with reference to... Figure 6 , Figure 6 This is a schematic diagram of an electronic device according to an embodiment of this disclosure. For example... Figure 6 As shown, the electronic device 600 includes a memory 610 and a processor 620. The memory 610 and the processor 620 are connected via a bus for communication. The memory 610 stores a computer program that can run on the processor 620 to implement the steps in the abnormal driving detection method disclosed in this embodiment.
[0099] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the abnormal driving detection method disclosed in this disclosure.
[0100] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the abnormal driving detection method disclosed in this disclosure.
[0101] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0102] Those skilled in the art will understand that the embodiments disclosed herein can be provided as methods, apparatus, or computer program products. Therefore, the embodiments disclosed herein can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments disclosed herein can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0103] This disclosure describes embodiments of methods, apparatus, electronic devices, and computer program products according to embodiments of this disclosure with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0104] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0106] Although some embodiments of the present disclosure have been described, those skilled in the art, upon learning the basic inventive concept, can make further changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the present disclosure.
[0107] The above provides a detailed description of an abnormal driving detection method, device, electronic device, and storage medium provided by this disclosure. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.
Claims
1. An abnormal driving detection method characterized by comprising: include: Based on the environmental information corresponding to the target vehicle, determine the target detection strategy for the preliminary detection stage of the image to be detected; The environmental information includes at least lighting information and camera accuracy information; Based on the target detection strategy, abnormal behavior detection is performed on the image to be detected to obtain preliminary abnormal detection results; The preliminary anomaly detection results include: anomaly behavior detection boxes and anomaly confidence scores; Determine whether the anomaly confidence level is within the anomaly detection interval; the anomaly detection interval indicates that abnormal driving cannot be determined by the anomaly confidence level. When the anomaly confidence level is within the anomaly detection range, the anomaly behavior detection box is supplemented with details to obtain a supplemented region; Abnormal behavior detection is performed on the supplementary area to obtain the final abnormal detection result; The step of determining the target detection strategy for the preliminary detection stage of the image to be detected based on the environmental information corresponding to the target vehicle includes: When the illumination information is lower than the illumination threshold, and / or when the camera accuracy information is lower than the accuracy threshold, the first detection strategy is determined as the target detection strategy; the first detection strategy is used to detect faces and abnormal behaviors sequentially for low-resolution images to be detected. When the illumination information is higher than or equal to the illumination threshold, and when the camera accuracy information is higher than or equal to the accuracy threshold, the second detection strategy is determined as the target detection strategy; the second detection strategy is used to detect faces and abnormal behaviors simultaneously for high-resolution images to be detected.
2. The method according to claim 1, characterized in that, Based on the first detection strategy, abnormal behavior detection is performed on the image to be detected to obtain preliminary abnormal detection results, including: The driver's face is detected from the image to be detected using a first neural network, resulting in a face detection box and a face confidence score; the first neural network is used to detect faces. When the face confidence score is greater than or equal to the face threshold, the face detection box is dynamically expanded to obtain an expanded region; The extended region is subjected to abnormal behavior detection using a second neural network, the abnormal behavior detection box is output, and the abnormal confidence level corresponding to the abnormal behavior detection box is determined; the second neural network is used to detect different abnormal behaviors.
3. The method according to claim 1, characterized in that, According to the second detection strategy, abnormal behavior detection is performed on the image to be detected to obtain preliminary abnormal detection results, including: The third neural network is used to detect the driver's face and abnormal behavior from the image to be detected, and to obtain the face detection box and face confidence, and the abnormal behavior detection box and abnormal confidence. The third neural network is used to detect faces and detect different abnormal behaviors. When the face confidence score is greater than or equal to the face threshold, the face detection box is dynamically expanded to obtain an expanded region; Determine the geometric center of the abnormal behavior detection box; When the geometric center of the abnormal behavior detection box is located within the extended region, the abnormal confidence level corresponding to the abnormal behavior detection box is determined.
4. The method according to claim 3, characterized in that, Also includes: If the geometric center of the abnormal behavior detection box is not located within the extended area, it is determined to be normal driving.
5. The method according to claim 1, characterized in that, Anomaly detection is performed on the supplementary region to obtain the final anomaly detection result, including: The supplementary region is detected by a fourth neural network to determine the target confidence level of the supplementary region as abnormal behavior; the fourth neural network is used to determine whether the category corresponding to the abnormal behavior detection box is correct. If the target confidence level is greater than or equal to the target threshold, it is determined to be abnormal driving; If the target confidence level is less than the target threshold, it is determined to be normal driving.
6. The method according to claim 5, characterized in that, The target detection strategy includes a first detection strategy and a second detection strategy, which perform abnormal behavior detection on the supplementary region to obtain the final abnormal detection result, including: If all images acquired at each time point are determined to indicate abnormal driving, a first target confidence level corresponding to the first detection strategy and a second target confidence level corresponding to the second detection strategy are determined respectively; the first target confidence level and the second target confidence level respectively represent different probabilities of abnormal behavior; Assign corresponding weights to the first target confidence level and the second target confidence level; The total confidence score obtained by weighted summation of the first target confidence score and the second target confidence score; Based on the total confidence level, determine whether the driving was abnormal.
7. The method according to any one of claims 1-6, characterized in that, Also includes: If the anomaly confidence level is greater than the maximum value of the anomaly detection interval, abnormal driving is determined; If the anomaly confidence level is less than the minimum value of the anomaly detection interval, normal driving is determined.
8. An abnormal driving detection device, characterized in that, include: The strategy determination module is used to determine the target detection strategy for the preliminary detection stage of the image to be detected based on the environmental information corresponding to the target vehicle. The environmental information includes at least lighting information and camera accuracy information; Among them, based on the environmental information corresponding to the target vehicle, a target detection strategy is determined for the preliminary detection stage of the image to be detected, including: When the illumination information is lower than the illumination threshold, and / or when the camera accuracy information is lower than the accuracy threshold, the first detection strategy is determined as the target detection strategy; the first detection strategy is used to detect faces and abnormal behaviors sequentially for low-resolution images to be detected. When the illumination information is higher than or equal to the illumination threshold, and when the camera accuracy information is higher than or equal to the accuracy threshold, the second detection strategy is determined as the target detection strategy; the second detection strategy is used to detect faces and abnormal behaviors simultaneously for high-resolution images to be detected. The first detection module is used to perform abnormal behavior detection on the image to be detected according to the target detection strategy, and obtain preliminary abnormal detection results; the preliminary abnormal detection results include: abnormal behavior detection boxes and abnormal confidence scores; The judgment module is used to determine whether the anomaly confidence level is within the anomaly detection interval; the anomaly detection interval indicates that it is impossible to determine whether abnormal driving is occurring based on the anomaly confidence level. The supplementary module is used to supplement the abnormal behavior detection box with details when the anomaly confidence level is within the anomaly detection range, so as to obtain a supplementary area; The second detection module is used to detect abnormal behavior in the supplementary area and obtain the final abnormal detection result.
9. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein when executed by the processor, the computer program implements the steps of the abnormal driving detection method as described in any one of claims 1-7.
Citation Information
Patent Citations
Driving behavior detection method and device, electronic equipment and readable storage medium
CN115171082A
Target detection method and device for automatic driving, electronic equipment and medium
CN115311522A
Abnormal behavior detection method and device, storage medium and electronic equipment
CN116645731A
