Abnormal driving detection method and device, electronic equipment and medium

Abstract

The invention provides an abnormal driving detection method and device, electronic equipment and a medium, belongs to the technical field of computers, and aims to solve the problem that an abnormal behavior of a driver cannot be accurately recognized in related technologies. The method comprises the steps of determining a target detection strategy for a preliminary detection stage of a to-be-detected image according to environment information corresponding to a target vehicle; performing abnormal behavior detection on a to-be-detected image according to the target detection strategy to obtain a preliminary abnormal detection result; judging whether the abnormal confidence is in an abnormal detection interval or not; the abnormal detection interval represents that whether abnormal driving cannot be judged through abnormal confidence; under the condition that the abnormal confidence is in an abnormal detection interval, performing detail supplementation on the abnormal behavior detection box to obtain a supplementation area; and performing abnormal behavior detection on the supplementary region to obtain a final abnormal detection result.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an abnormal driving detection method, device, electronic equipment and medium. Background Art

[0002] Driver Monitoring System (DMS) plays a vital role in the field of intelligent driving. It can improve driving safety and reduce traffic accidents, and can detect driver's bad behavior in real time.

[0003] Current DMS technology relies on computer vision and deep learning models to analyze images captured by in-car cameras to identify the driver's facial features and behavior. However, the abnormality recognition method of related technologies cannot accurately identify the driver's abnormal behavior in scenes with changing lighting or small object detection scenes, and has limitations. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides an abnormal driving detection method, device, electronic device and medium. The technical solution of the present disclosure is as follows: According to a first aspect of an embodiment of the present disclosure, there is provided an abnormal driving detection method, comprising: Determine the target detection strategy for the preliminary detection stage of the image to be detected according to the environmental information corresponding to the target vehicle; the environmental information includes at least lighting information and camera accuracy information; According to the target detection strategy, abnormal behavior detection is performed on the image to be detected to obtain a preliminary abnormal detection result; the preliminary abnormal detection result includes: an abnormal behavior detection frame and an abnormal confidence; Determining whether the abnormal confidence level is in an abnormal detection interval; the abnormal detection interval indicates that it is impossible to determine whether the driving is abnormal by the abnormal confidence level; When the abnormality confidence is within the abnormality detection interval, supplementing the abnormal behavior detection frame with details to obtain a supplemented area; Abnormal behavior detection is performed on the supplementary area to obtain a final abnormality detection result.

[0005] Optionally, according to the environmental information corresponding to the target vehicle, determining a target detection strategy for the preliminary detection stage of the image to be detected includes: When the illumination information is lower than the illumination threshold, and / or when the camera accuracy information is lower than the accuracy threshold, the first detection strategy is determined as the target detection strategy; the first detection strategy is used to detect faces and abnormal behaviors in sequence for low-resolution images to be detected; When the illumination information is higher than or equal to the illumination threshold, and when the camera accuracy information is higher than or equal to the accuracy threshold, the second detection strategy is determined as the target detection strategy; the second detection strategy is used to detect faces and abnormal behaviors simultaneously for high-resolution images to be detected.

[0006] Optionally, according to the first detection strategy, abnormal behavior detection is performed on the image to be detected to obtain a preliminary abnormality detection result, including: Detecting the driver's face from the image to be detected by a first neural network to obtain a face detection frame and a face confidence level; the first neural network is used to detect faces; When the face confidence level is greater than or equal to the face threshold, dynamically expanding the face detection frame to obtain an expanded area; The extended area is subjected to abnormal behavior detection through a second neural network, the abnormal behavior detection frame is output, and the abnormal confidence corresponding to the abnormal behavior detection frame is determined; the second neural network is used to detect different abnormal behaviors.

[0007] Optionally, according to the second detection strategy, abnormal behavior detection is performed on the image to be detected to obtain a preliminary abnormality detection result, including: Detecting the driver's face and abnormal behavior from the image to be detected by a third neural network, and obtaining a face detection frame and a face confidence, an abnormal behavior detection frame and an abnormal confidence; the third neural network is used to detect faces and detect different abnormal behaviors; When the face confidence level is greater than or equal to the face threshold, dynamically expanding the face detection frame to obtain an expanded area; Determining the geometric center of the abnormal behavior detection frame; In a case where the geometric center of the abnormal behavior detection frame is located within the extended area, an abnormality confidence corresponding to the abnormal behavior detection frame is determined.

[0008] Optionally, it also includes: When the geometric center of the abnormal behavior detection frame is not located within the extended area, it is determined to be normal driving.

[0009] Optionally, abnormal behavior detection is performed on the supplementary area to obtain a final abnormality detection result, including: The supplementary area is detected by a fourth neural network to determine the target confidence that the supplementary area is abnormal behavior; the fourth neural network is used to determine whether the category corresponding to the abnormal behavior detection box is correct; When the target confidence is greater than or equal to the target threshold, determining it as abnormal driving; When the target confidence level is less than the target threshold, it is determined to be normal driving.

[0010] Optionally, the target detection strategy includes a first detection strategy and a second detection strategy, and abnormal behavior detection is performed on the supplementary area to obtain a final abnormality detection result, and further includes: In the case where all the images to be detected acquired at each moment are judged to be abnormal driving, respectively determining a first target confidence corresponding to the first detection strategy and a second target confidence corresponding to the second detection strategy; the first target confidence and the second target confidence respectively represent different abnormal behavior probabilities; assigning corresponding weights to the first target confidence and the second target confidence; A total confidence obtained by weighted summing the first target confidence and the second target confidence; Whether the driving is abnormal is determined based on the total confidence level.

[0011] Optionally, it also includes: In a case where the abnormality confidence is greater than a maximum value of the abnormality detection interval, determining abnormal driving; In a case where the abnormality confidence is less than the minimum value of the abnormality detection interval, normal driving is determined.

[0012] According to a second aspect of an embodiment of the present disclosure, there is provided an abnormal driving detection device, comprising: A strategy determination module, used to determine the target detection strategy for the preliminary detection stage of the image to be detected according to the environmental information corresponding to the target vehicle; the environmental information at least includes illumination information and camera accuracy information; A first detection module is used to perform abnormal behavior detection on the image to be detected according to the target detection strategy to obtain a preliminary abnormal detection result; the preliminary abnormal detection result includes: an abnormal behavior detection frame and an abnormal confidence level; A judgment module, used to judge whether the abnormal confidence level is in an abnormal detection interval; the abnormal detection interval indicates that it is impossible to judge whether the driving is abnormal by the abnormal confidence level; A supplementing module, used for supplementing the abnormal behavior detection frame with details to obtain a supplemented area when the abnormal confidence level is within the abnormal detection interval; The second detection module is used to perform abnormal behavior detection on the supplementary area to obtain a final abnormality detection result.

[0013] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the abnormal driving detection method described in the first aspect are implemented.

[0014] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the abnormal driving detection method described in the first aspect are implemented.

[0015] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the abnormal driving detection method as described in the first aspect are implemented.

[0016] The present disclosure determines the target detection strategy based on environmental information, and can cope with different driving environments, especially in scenes with large changes in lighting, and can improve the accuracy and stability of detection. For behaviors with abnormal confidence levels in the detection interval, the system further supplements details, obtains supplementary areas, and re-detects them, which can improve the ability to capture subtle abnormal behaviors, especially in small object detection or complex action recognition, and can significantly improve the accuracy and precision of detection. The present disclosure can quickly identify abnormal behaviors of drivers in real-time video streams and respond in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the description of the embodiments of the present disclosure will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 is a schematic diagram of the steps of an abnormal driving detection method shown in an embodiment of the present disclosure; Figure 2 is a flow chart of abnormal driving detection shown in an embodiment of the present disclosure; Figure 3 is a flowchart of abnormal driving detection using a first detection strategy shown in an embodiment of the present disclosure; Figure 4 is a flowchart of abnormal driving detection using a second detection strategy shown in an embodiment of the present disclosure; Figure 5 is a block diagram of an abnormal driving detection device shown in an embodiment of the present disclosure; Figure 6It is a schematic diagram of an electronic device shown in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0020] The terms "first", "second", etc. in the specification and claims of the present disclosure are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable when appropriate, so that the embodiments of the present disclosure can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0021] Driver monitoring system plays a vital role in the field of intelligent driving, which can improve driving safety and reduce traffic accidents, and can detect the driver's bad behavior in real time, such as smoking, making phone calls, etc. Driver monitoring system is becoming more and more important.

[0022] In the related art, the driver monitoring system mainly relies on computer vision and deep learning models to identify the driver's facial features and behavior status through images taken by the in-car camera. However, the abnormality recognition method of the related art cannot accurately identify the driver's abnormal behavior in scenes with changing lighting or small object detection scenes, and has limitations.

[0023] Specifically, the DMS system in the related technology usually uses convolutional neural networks for face detection and behavior recognition, combined with posture estimation and gesture recognition technology to determine whether the driver is smoking, making phone calls, etc. Among them, when detecting small objects such as cigarettes and mobile phones, some DMS systems rely on multi-scale feature extraction or contextual information to improve detection accuracy. However, in actual driving scenarios, small objects such as cigarettes or mobile phones may have problems such as occlusion, low resolution, and angle changes, which will affect the accuracy of detection and lead to missed detection problems.

[0024] In order to solve the above technical problems, the present disclosure proposes an abnormal driving detection method, which can effectively improve the detection accuracy of small objects in scenarios with long distances and complex backgrounds, and avoid missing detection of abnormal driver behavior.

[0025] Figure 1 Schematic diagram of the steps of an abnormal driving detection method shown in an embodiment of the present disclosure. Figure 1 As shown, the abnormal driving detection method may specifically include the following steps: Step S11: Determine a target detection strategy for the preliminary detection phase of the image to be detected according to the environmental information corresponding to the target vehicle; the environmental information at least includes illumination information and camera accuracy information.

[0026] Environmental information includes information such as lighting conditions and camera accuracy. Images of different resolutions can be collected under different environmental conditions. In order to adapt to images of various clarity, different target detection strategies can be used for detection.

[0027] The image can be preliminarily detected by using a target detection strategy that is adapted to the image clarity. Different detection strategies may correspond to different detection network complexities, which can effectively utilize computing resources.

[0028] Step S12: According to the target detection strategy, abnormal behavior detection is performed on the image to be detected to obtain a preliminary abnormality detection result; the preliminary abnormality detection result includes: an abnormal behavior detection frame and an abnormality confidence.

[0029] The image to be detected is captured by the camera in the car. According to the target detection strategy, it is determined whether there is abnormal behavior of the driver in the image to be detected, and abnormal behavior includes answering the phone, smoking, etc. The abnormal behavior of the driver can be detected by detecting the position relationship between the abnormal objects corresponding to each abnormal behavior and the driver.

[0030] Whether the driver has abnormal behavior is determined by the abnormal confidence of the abnormal behavior. The abnormal confidence indicates the probability of abnormal behavior in the image to be detected. When the probability is too high, it indicates that there is a high probability of abnormal behavior. When the probability is too low, it indicates that there is a high probability of normal behavior.

[0031] Step S13: Determine whether the abnormal confidence level is in an abnormal detection interval; the abnormal detection interval indicates that it is impossible to determine whether the driving is abnormal through the abnormal confidence level.

[0032] An abnormality detection interval is determined. When the abnormality confidence is within the abnormality detection interval, it indicates that it is not possible to confirm whether the image to be detected has abnormal behavior. The abnormality detection interval is preset, and it can be that when determining the abnormality confidence of each abnormal behavior, it is determined whether the abnormality confidence is within the abnormality detection interval.

[0033] Step S14: When the abnormality confidence is within the abnormality detection interval, the abnormal behavior detection frame is supplemented with details to obtain a supplemented area.

[0034] When the abnormal confidence is in the abnormal detection interval, further abnormal behavior needs to be performed. In order to reduce the amount of calculation, the abnormal behavior detection frame corresponding to the abnormal behavior is extracted, and the abnormal behavior detection frame is supplemented with details. Specifically, the image information corresponding to the abnormal behavior detection frame can be input into a super-resolution network, and the corresponding high-resolution image is generated by the super-resolution network to enhance the details. The super-resolution network can be SRCNN (Super-Resolution Convolutional Neural Network).

[0035] The supplemented area is the abnormal behavior detection frame after being supplemented with details.

[0036] Step S15: performing abnormal behavior detection on the supplementary area to obtain a final abnormality detection result.

[0037] The supplementary area is further subjected to abnormality detection to determine the target confidence level of the supplementary area being abnormal behavior. The target confidence level indicates the probability of abnormal behavior in the supplementary area.

[0038] Usually, an abnormal threshold can be set. When the target confidence is greater than or equal to the abnormal threshold, it indicates that abnormal behavior exists, thereby determining that the driver in the image to be detected is driving abnormally. When the target confidence is less than the abnormal threshold, it indicates that there is no abnormal behavior, thereby determining that the driver in the image to be detected is driving normally.

[0039] By adopting the embodiments of the present disclosure, through real-time analysis of environmental information, the system can dynamically adjust the target detection strategy to cope with challenges in different scenarios, and can maintain high detection accuracy in different environments. Based on advanced algorithms such as deep learning, the abnormal confidence of each abnormal behavior in the image to be detected can be calculated, and normal driving behavior and potentially dangerous behavior can be distinguished. In the case where the driver's current behavior cannot be determined by the abnormal confidence, the abnormal behavior detection frame can be supplemented with details to further amplify and enhance the features of key areas for more in-depth detection. The multi-layer detection mechanism proposed in the present disclosure helps to reduce false positives and false negatives, and improve the accuracy and reliability of detection.

[0040] Among them, in an optional embodiment, it also includes: when the abnormality confidence is greater than the maximum value of the abnormality detection interval, determining abnormal driving; when the abnormality confidence is less than the minimum value of the abnormality detection interval, determining normal driving.

[0041] When the anomaly confidence is on both sides of the anomaly detection interval, it indicates that the detection result of the abnormal behavior has been determined and the detection result is accurate.

[0042] When the abnormality confidence is less than the minimum value of the abnormality detection interval, it indicates that the current driving behavior is within the normal range, and the system should determine it as normal driving. When the abnormality confidence is greater than the maximum value of the abnormality detection interval, it indicates that the current driving behavior is highly abnormal, and the system should determine it as abnormal driving. Specifically, when the abnormality detection interval is [0.3, 0.4], the abnormality confidence is less than the minimum value of the abnormality detection interval, indicating that the abnormality confidence is less than 0.3. In this case, it is directly judged as normal driving; the abnormality confidence is greater than the maximum value of the abnormality detection interval, indicating that the abnormality confidence is greater than 0.4. In this case, it is directly judged as abnormal.

[0043] By adopting the embodiments of the present disclosure, when the initial detection shows that the two-point anomaly confidence is not within the anomaly detection interval, the driving behavior of the driver in the image to be detected is directly determined, which can reduce the amount of calculation, improve detection efficiency, and save computing resources.

[0044] Among them, through environmental information, the target detection strategy can be divided into a first detection strategy and a second detection strategy. The first detection strategy and the second detection strategy require different levels of computing resources and have different advantages. For example, the first detection strategy can be applicable to situations where the detection environment is complex and the camera accuracy is low; the second detection strategy can be applicable to situations where the on-board computing resources are limited and high real-time performance is required. In other words, the first detection strategy has high accuracy but consumes more resources, while the second detection strategy is lightweight and fast, but requires the clarity of the image to be detected to meet the requirements.

[0045] In actual vehicle use scenarios, you can select the target detection strategy to use based on the time period. For example, in summer, the second detection strategy is used from 7 a.m. to 7 p.m., and the first detection strategy is used for the rest of the time; in winter, the second detection strategy is used from 9 a.m. to 5 p.m., and the first detection strategy is used for the rest of the time.

[0046] Among them, in an optional embodiment, according to the environmental information corresponding to the target vehicle, a target detection strategy for the preliminary detection stage of the image to be detected is determined, including: when the lighting information is lower than the lighting threshold and / or when the camera accuracy information is lower than the accuracy threshold, the first detection strategy is determined as the target detection strategy; the first detection strategy is used to detect faces and abnormal behaviors in sequence for low-resolution images to be detected; when the lighting information is higher than or equal to the lighting threshold, and when the camera accuracy information is higher than or equal to the accuracy threshold, the second detection strategy is determined as the target detection strategy; the second detection strategy is used to detect faces and abnormal behaviors at the same time for high-resolution images to be detected.

[0047] The illumination threshold and the accuracy threshold may be preset in advance. If the illumination information is lower than the preset illumination threshold, it indicates that the environment is too dark and may affect the image quality. If the camera accuracy information is lower than the preset accuracy threshold, it indicates that the camera performance is limited.

[0048] When the illumination information is lower than the illumination threshold, and / or when the camera accuracy information is lower than the accuracy threshold, the clarity of the acquired image to be detected is low, so the first detection strategy with higher accuracy is adopted as the target detection strategy. The first detection strategy sequentially checks the face and abnormal behavior in the image to be detected, first detects whether there is a face in the image to be detected, and then detects abnormal behavior in the image to be detected after the conditions are met, which can well adapt to the detection needs of low-resolution images to be detected. Two different neural networks can be used to perform the detection operation. The first neural network detects the face, and when the conditions are met, transmits its own output to the second neural network for abnormal behavior detection.

[0049] When the illumination information is higher than or equal to the illumination threshold, and when the camera accuracy information is higher than or equal to the accuracy threshold, it means that the clarity of the image to be detected collected in this case is high, so the second detection strategy is selected for preliminary detection. The second detection strategy can detect faces and abnormal behaviors from the image to be detected at the same time. A separate neural network can be used to perform the detection operation and simultaneously determine faces and abnormal behaviors in the image to be detected.

[0050] By adopting the embodiments of the present disclosure, when the image quality is low, the first detection strategy is adopted as the target detection strategy, which can improve the detection capability in low-light or low-quality images. When both the illumination and camera accuracy meet or exceed the preset threshold, the second detection strategy is adopted to improve the accuracy and efficiency of detection.

[0051] Figure 2 is a flow chart of abnormal driving detection shown in an embodiment of the present disclosure. Figure 2 As shown, the environmental information corresponding to the target vehicle when the image to be detected is collected is determined; according to the environmental information, different detection strategies are used to detect whether the driver in the image to be detected has abnormal behavior; when the first condition is met, the first detection strategy is used to perform abnormal behavior detection in the preliminary detection stage; when the second condition is met, the second detection strategy is used to perform abnormal behavior detection in the preliminary detection stage; a preliminary abnormal detection result is obtained through the first detection strategy or the second detection strategy; the preliminary abnormal detection result includes: an abnormal behavior detection frame and an abnormal confidence; it is determined whether the abnormal confidence is in the abnormal detection interval; when the abnormal confidence is in the abnormal detection interval, the abnormal behavior detection frame is supplemented with details to obtain a supplemented area, and abnormal behavior detection is performed on the supplemented area to obtain a final abnormal detection result; when the abnormal confidence is not in the abnormal detection interval, it is determined whether the abnormal confidence is greater than the maximum value of the abnormal detection interval; when it is greater than the maximum value of the abnormal detection interval, it is determined that the driver in the image to be detected is driving abnormally; when it is not greater than the maximum value of the abnormal detection interval, it is determined that the driver in the image to be detected is driving normally.

[0052] (1) The following is the process of conducting preliminary testing using the first testing strategy.

[0053] Among them, in an optional embodiment, according to the first detection strategy, abnormal behavior detection is performed on the image to be detected to obtain preliminary abnormal detection results, including: detecting the driver's face from the image to be detected through a first neural network to obtain a face detection frame and a face confidence; the first neural network is used to detect the face; when the face confidence is greater than or equal to the face threshold, the face detection frame is dynamically expanded to obtain an extended area; abnormal behavior detection is performed on the extended area through a second neural network, the abnormal behavior detection frame is output, and the abnormal confidence corresponding to the abnormal behavior detection frame is determined; the second neural network is used to detect different abnormal behaviors.

[0054] When the first detection strategy is adopted as the target detection strategy, the driver's face is detected from the image to be detected by the first neural network. The first neural network can be a convolutional neural network (CNN), and specifically, the classic face detection model MTCNN (Multi-task Cascaded Convolutional Networks) can be used to obtain the bounding box of the face. The first neural network for detecting faces has two heads, one is a regression head, which is used to output the detection box, and the other is a classification head, which is used to output the category confidence. The classification head is a 2-class classification.

[0055] The face confidence is used to characterize the probability that a face is included in a face detection frame. When the face confidence is greater than or equal to the face threshold, it means that the face detection frame contains a face; when the face confidence is less than the face threshold, it means that the face detection frame does not contain a face.

[0056] When the face confidence is greater than or equal to the face threshold, that is, when the face detection frame contains the face, the face detection frame is dynamically expanded to obtain an extended area. Specifically, the face detection frame can be dynamically expanded by 1.5 times based on the center point of the face detection frame to obtain an extended area. The extended area is used as a key monitoring area. The extended area needs to include areas where abnormal behavior may occur, specifically, it can cover areas where the driver may hold cigarettes or phones.

[0057] The second neural network is used to further detect the extended area. The second neural network can be YOLOv5 (You Only Look Once version 5), which determines the abnormal behavior detection box in the extended area and the corresponding abnormal confidence. The abnormal confidence represents the probability of abnormal behavior in the abnormal behavior detection box.

[0058] The second neural network can output various detection frames, and the detection frames can correspond to different categories and confidence levels, specifically, including smoking, making phone calls, normal behavior, smoking and making phone calls. In the case where the category of the detection frame is abnormal behavior, for example, smoking, making phone calls, smoking and making phone calls, the detection frame is determined as an abnormal behavior detection frame, and its corresponding confidence level is determined as an abnormal confidence level. Whether further detection is needed is determined by the abnormal confidence level.

[0059] By using the embodiments of the present disclosure, combined with multi-stage processing of face detection and abnormal behavior detection, and a neural network model based on deep learning, the system can accurately identify abnormal driver behavior. It can be performed quickly in real-time video streams, while being able to cope with a certain degree of image quality variation.

[0060] Figure 3 FIG. 1 is a flowchart of abnormal driving detection using a first detection strategy shown in an embodiment of the present disclosure. Figure 3As shown, when the illumination information is lower than the illumination threshold, and / or when the camera accuracy information is lower than the accuracy threshold, the first detection strategy is determined as the target detection strategy; the driver's face is detected from the image to be detected by the first neural network to obtain a face detection frame and a face confidence; the driver's face is detected from the image to be detected by the first neural network to obtain a face detection frame and a face confidence; it is determined whether the face confidence is greater than or equal to the face threshold; when the face confidence is greater than or equal to the face threshold, the face detection frame is dynamically expanded to obtain an extended area, abnormal behavior detection is performed on the extended area by the second neural network, the abnormal behavior detection frame is output, and the abnormal confidence corresponding to the abnormal behavior detection frame is determined; when the face confidence is less than the face threshold, it is determined that the image to be detected is normal driving without abnormal behavior.

[0061] (2) The following is the process of conducting preliminary testing using the second testing strategy.

[0062] Among them, in an optional embodiment, according to the second detection strategy, abnormal behavior detection is performed on the image to be detected to obtain preliminary abnormal detection results, including: detecting the driver's face and abnormal behavior from the image to be detected through a third neural network to obtain a face detection frame and face confidence, an abnormal behavior detection frame and abnormal confidence; the third neural network is used to detect faces and detect different abnormal behaviors; when the face confidence is greater than or equal to the face threshold, the face detection frame is dynamically expanded to obtain an extended area; the geometric center of the abnormal behavior detection frame is determined; when the geometric center of the abnormal behavior detection frame is located in the extended area, the abnormal confidence corresponding to the abnormal behavior detection frame is determined.

[0063] The third neural network may be YOLOv5 (You Only Look Once version 5), which outputs various detection frames, and the categories corresponding to the detection frames may be faces and abnormal behaviors. When outputting various detection frames, the confidence of the categories corresponding to the various detection frames is simultaneously output. Specifically, the face detection frame, face confidence, abnormal behavior detection frame, and abnormal confidence may be output.

[0064] When the face confidence level corresponding to the face detection frame is greater than or equal to the face threshold, the face detection frame is dynamically expanded to obtain an expanded area.

[0065] Then determine the positional relationship between the extended area and the abnormal behavior detection frame. Specifically, the geometric center of the abnormal behavior detection frame can be determined first, and then the positional relationship between the geometric center and the extended area can be determined. When the geometric center is within the extended area, the abnormal behavior detection frame is considered to be within the extended area; when the geometric center is outside the extended area, the abnormal behavior detection frame is considered to be outside the extended area.

[0066] When the geometric center of the abnormal behavior detection frame is located within the extended area, the abnormality confidence corresponding to the abnormal behavior detection frame is determined, and whether further detection is required is determined based on the abnormality confidence.

[0067] By adopting the embodiments of the present disclosure, when the lighting and camera conditions are good, the second detection strategy can more effectively utilize computing resources, and can reduce computing time or reduce the demand for hardware resources without sacrificing detection performance.

[0068] Figure 4 FIG. 1 is a flowchart of abnormal driving detection using the second detection strategy shown in an embodiment of the present disclosure. Figure 4 As shown. In the case where the illumination information is higher than or equal to the illumination threshold, and in the case where the camera accuracy information is higher than or equal to the accuracy threshold, the second detection strategy is determined as the target detection strategy; the driver's face and abnormal behavior are detected from the image to be detected by a third neural network to obtain a face detection frame and a face confidence, an abnormal behavior detection frame and an abnormal confidence; it is determined whether the face confidence is greater than or equal to the face threshold; when the face confidence is less than the face threshold, it is determined to be normal driving; when the face confidence is greater than or equal to the face threshold, the face detection frame is dynamically expanded to obtain an extended area, and the geometric center of the abnormal behavior detection frame is determined; it is determined whether the geometric center of the abnormal behavior detection frame is located in the extended area; when the geometric center of the abnormal behavior detection frame is not located in the extended area, it is determined to be normal driving; when the geometric center of the abnormal behavior detection frame is located in the extended area, the abnormal confidence corresponding to the abnormal behavior detection frame is determined.

[0069] Among them, in an optional embodiment, it also includes: when the geometric center of the abnormal behavior detection box is not located in the extended area, determining it as normal driving.

[0070] The extended area includes an area where the driver may have abnormal behavior. Therefore, when the geometric center of the abnormal behavior detection frame is not located in the extended area, that is, when the abnormal behavior detection frame is located outside the extended area, it is directly determined that the driver in the image to be detected is driving normally.

[0071] By adopting the embodiments of the present disclosure, when the geometric center of the abnormal behavior detection frame is not located in the extended area, it is directly determined that the driver is driving normally, which can effectively improve the detection efficiency, improve the real-time performance of the detection, and save computing resources.

[0072] Both the first detection strategy and the second detection strategy require that subsequent operations be performed only when the face confidence corresponding to the face detection frame is greater than or equal to the face threshold. When the face confidence corresponding to the face detection frame is less than the face threshold, it is determined that there is no driver in the image to be detected, and it is determined that the current driving is normal.

[0073] In some cases, the first detection strategy and the second detection strategy are used simultaneously to determine whether the driver in the image to be detected is driving abnormally. This is because different target detection strategies used in the preliminary detection will affect the final detection result, that is, the result of detecting abnormal driving.

[0074] Among them, in an optional embodiment, the target detection strategy includes a first detection strategy and a second detection strategy, performs abnormal behavior detection on the supplementary area to obtain a final abnormal detection result, and also includes: when the images to be detected acquired at each moment are judged to be abnormal driving, respectively determine the first target confidence corresponding to the first detection strategy and the second target confidence corresponding to the second detection strategy; the first target confidence and the second target confidence respectively represent different abnormal behavior probabilities; assign corresponding weights to the first target confidence and the second target confidence; obtain the total confidence after the weighted sum of the first target confidence and the second target confidence; and judge whether it is abnormal driving based on the total confidence.

[0075] Normally, when abnormal driving is detected, the driver will be prompted. If abnormal driving is detected by the first detection strategy or the second detection strategy, and prompts are given, but the driver is frequently detected to be driving abnormally, it is possible that the use of a single first detection strategy or the second detection strategy has errors, resulting in multiple detections of abnormal behavior.

[0076] Therefore, for the same image to be detected, the first detection strategy or the second detection strategy can be used simultaneously to perform preliminary detection.

[0077] A first detection strategy is used to perform preliminary detection on the image to be detected to obtain a first abnormal behavior detection frame; details are supplemented on the first abnormal behavior detection frame to obtain a first supplemented area; and the first supplemented area is detected to obtain a first target confidence.

[0078] The image to be detected is preliminarily detected using the second detection strategy to obtain a second abnormal behavior detection frame; the second abnormal behavior detection frame is supplemented with details to obtain a second supplemented area; the second supplemented area is detected to obtain a second target confidence.

[0079] According to the first target confidence and the second target confidence, it is determined whether the driving is abnormal. The results corresponding to the first detection strategy and the second detection strategy can be weighted, and then the first target confidence and the second target confidence are weighted and summed to obtain the total confidence.

[0080] Usually, the first detection strategy has a higher accuracy, so a higher weight can be assigned to the first target confidence corresponding to the first detection strategy. The sum of the weight assigned to the first detection strategy and the weight assigned to the second detection strategy is 1. Specifically, a weight of 0.6 can be assigned to the first target confidence; a weight of 0.4 can be assigned to the second target confidence; c=0.6a+0.4b, the first target confidence is recorded as a, the second target confidence is recorded as b, and the total confidence is recorded as c.

[0081] The total confidence level is used to determine whether the driving is abnormal. Specifically, the total confidence level can be compared with a preset threshold. If the total confidence level is greater than or equal to the threshold, it indicates abnormal driving; if the total confidence level is less than the threshold, it indicates normal driving.

[0082] With the embodiments of the present disclosure, a single detection strategy may lead to false positives or missed positives due to the limitations of the algorithm itself or the complexity of a specific scenario. By combining two or more detection strategies, they can complement each other and reduce errors caused by the defects of a single strategy, thereby improving the accuracy of the overall detection. The combination of multiple detection strategies enables the system to cope with a wider range of driving scenarios and abnormal behavior types. Even if a certain strategy fails in a specific situation, other strategies can still provide effective detection, thereby enhancing the stability and reliability of the system. Different weights are assigned to different detection strategies, which can be adjusted according to actual needs to adapt to different driving environments and safety requirements.

[0083] Among them, in an optional embodiment, abnormal behavior detection is performed on the supplementary area to obtain a final abnormal detection result, including: detecting the supplementary area through a fourth neural network to determine the target confidence that the supplementary area is abnormal behavior; the fourth neural network is used to determine whether the category corresponding to the abnormal behavior detection box is correct; when the target confidence is greater than or equal to the target threshold, it is determined to be abnormal driving; when the target confidence is less than the target threshold, it is determined to be normal driving.

[0084] The fourth neural network may be a classification network, specifically MobileNet v2 (MobileNet: Inverted Residuals and Linear Bottlenecks v2).

[0085] The supplementary area is an area that may contain abnormal behaviors, and the category of abnormal behaviors has been determined in the preliminary detection stage, such as smoking or making phone calls; therefore, the purpose of further testing the supplementary area is to determine the correctness of the preliminary detection. Therefore, it is only necessary to determine whether it is normal behavior or abnormal behavior.

[0086] The fourth neural network can obtain two possible categories of the supplementary area, including normal behavior and abnormal behavior. For the supplementary area with abnormal behavior as the category, the corresponding category probability is determined as the target confidence.

[0087] When the target confidence is greater than or equal to the target threshold, it is determined to be abnormal driving; when the target confidence is less than the target threshold, it is determined to be normal driving. Usually, the target threshold can be set to 0.5.

[0088] By using the embodiments of the present disclosure, a separate neural network is used to detect the supplementary area, which can verify the results of the preliminary detection stage, thereby increasing the robustness of the entire detection process, reducing false positives and false negatives, and improving the reliability of the system. By comparing the target confidence with the preset target threshold, determining the final result can make the entire decision-making process more intuitive and easy to understand.

[0089] Figure 5 is a block diagram of an abnormal driving detection device shown in an embodiment of the present disclosure. Figure 5 As shown, the abnormal driving detection device includes: A strategy determination module 510 is used to determine a target detection strategy for the preliminary detection stage of the image to be detected according to the environmental information corresponding to the target vehicle; the environmental information at least includes illumination information and camera accuracy information; The first detection module 520 is used to perform abnormal behavior detection on the image to be detected according to the target detection strategy to obtain a preliminary abnormal detection result; the preliminary abnormal detection result includes: an abnormal behavior detection frame and an abnormal confidence level; A judgment module 530 is used to judge whether the abnormal confidence level is in an abnormal detection interval; the abnormal detection interval indicates that it is impossible to judge whether the driving is abnormal by the abnormal confidence level; A supplementing module 540 is used to supplement the abnormal behavior detection frame with details to obtain a supplemented area when the abnormality confidence is within the abnormality detection interval; The second detection module 550 is used to perform abnormal behavior detection on the supplementary area to obtain a final abnormality detection result.

[0090] Optionally, the strategy determination module is specifically used to execute: When the illumination information is lower than the illumination threshold, and / or when the camera accuracy information is lower than the accuracy threshold, the first detection strategy is determined as the target detection strategy; the first detection strategy is used to detect faces and abnormal behaviors in sequence for low-resolution images to be detected; When the illumination information is higher than or equal to the illumination threshold, and when the camera accuracy information is higher than or equal to the accuracy threshold, the second detection strategy is determined as the target detection strategy; the second detection strategy is used to detect faces and abnormal behaviors simultaneously for high-resolution images to be detected.

[0091] The first detection module is specifically used to perform: Detecting the driver's face from the image to be detected by a first neural network to obtain a face detection frame and a face confidence level; the first neural network is used to detect faces; When the face confidence level is greater than or equal to the face threshold, dynamically expanding the face detection frame to obtain an expanded area; The extended area is subjected to abnormal behavior detection through a second neural network, the abnormal behavior detection frame is output, and the abnormal confidence corresponding to the abnormal behavior detection frame is determined; the second neural network is used to detect different abnormal behaviors.

[0092] Optionally, the first detection module is specifically configured to execute: Detecting the driver's face and abnormal behavior from the image to be detected by a third neural network, and obtaining a face detection frame and a face confidence, an abnormal behavior detection frame and an abnormal confidence; the third neural network is used to detect faces and detect different abnormal behaviors; When the face confidence level is greater than or equal to the face threshold, dynamically expanding the face detection frame to obtain an expanded area; Determining the geometric center of the abnormal behavior detection frame; In a case where the geometric center of the abnormal behavior detection frame is located within the extended area, an abnormality confidence corresponding to the abnormal behavior detection frame is determined.

[0093] Optionally, the abnormal driving detection device further includes: The first determination submodule is configured to determine that the driving is normal when the geometric center of the abnormal behavior detection frame is not located within the extended area.

[0094] Optionally, the supplementary module is specifically used to execute: The supplementary area is detected by a fourth neural network to determine the target confidence that the supplementary area is abnormal behavior; the fourth neural network is used to determine whether the category corresponding to the abnormal behavior detection box is correct; When the target confidence is greater than or equal to the target threshold, determining it as abnormal driving; When the target confidence level is less than the target threshold, it is determined to be normal driving.

[0095] Optionally, the target detection strategy includes a first detection strategy and a second detection strategy, and the second detection module is specifically configured to execute: In the case where all the images to be detected acquired at each moment are judged to be abnormal driving, respectively determining a first target confidence corresponding to the first detection strategy and a second target confidence corresponding to the second detection strategy; the first target confidence and the second target confidence respectively represent different abnormal behavior probabilities; assigning corresponding weights to the first target confidence and the second target confidence; A total confidence obtained by weighted summing the first target confidence and the second target confidence; Whether the driving is abnormal is determined based on the total confidence level.

[0096] Optionally, the abnormal driving detection device further includes: A second determination submodule, configured to determine abnormal driving when the abnormality confidence is greater than a maximum value of the abnormality detection interval; The third determination submodule is used to determine normal driving when the abnormality confidence is less than the minimum value of the abnormality detection interval.

[0097] It should be noted that the device embodiment is similar to the method embodiment, so the description is relatively simple, and the relevant parts can be referred to the method embodiment.

[0098] The present disclosure also provides an electronic device, referring to Figure 6 , Figure 6 is a schematic diagram of an electronic device shown in an embodiment of the present disclosure. Figure 6 As shown, the electronic device 600 includes: a memory 610 and a processor 620. The memory 610 and the processor 620 are connected via a bus communication. A computer program is stored in the memory 610. The computer program can be run on the processor 620 to implement the steps in the abnormal driving detection method disclosed in the embodiment of the present disclosure.

[0099] The embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the abnormal driving detection method disclosed in the embodiment of the present disclosure are implemented.

[0100] The embodiment of the present disclosure also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps in the abnormal driving detection method disclosed in the embodiment of the present disclosure are implemented.

[0101] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0102] It should be understood by those skilled in the art that the embodiments of the present disclosure may be provided as methods, devices or computer program products. Therefore, the embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] The embodiments of the present disclosure are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic devices, and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0104] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0106] Although some embodiments of the present disclosure have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the present disclosure.

[0107] The above is a detailed introduction to an abnormal driving detection method, device, electronic device and storage medium provided by the present disclosure. Specific examples are used in this article to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method of the present disclosure and its core idea. At the same time, for a person skilled in the art, according to the idea of ​​the present disclosure, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present disclosure.

Claims

1. An abnormal driving detection method, characterized in that: include: Determine the target detection strategy for the preliminary detection stage of the image to be detected according to the environmental information corresponding to the target vehicle; The environmental information includes at least lighting information and camera accuracy information; According to the target detection strategy, abnormal behavior detection is performed on the image to be detected to obtain a preliminary abnormal detection result; The preliminary anomaly detection results include: an abnormal behavior detection frame and an abnormality confidence; Determining whether the abnormal confidence level is in an abnormal detection interval; the abnormal detection interval indicates that it is impossible to determine whether the driving is abnormal by the abnormal confidence level; When the abnormality confidence is within the abnormality detection interval, supplementing the abnormal behavior detection frame with details to obtain a supplemented area; Abnormal behavior detection is performed on the supplementary area to obtain a final abnormality detection result.

2. The method according to claim 1, characterized in that According to the environmental information corresponding to the target vehicle, determine the target detection strategy for the preliminary detection stage of the image to be detected, including: When the illumination information is lower than the illumination threshold, and / or when the camera accuracy information is lower than the accuracy threshold, the first detection strategy is determined as the target detection strategy; the first detection strategy is used to detect faces and abnormal behaviors in sequence for low-resolution images to be detected; When the illumination information is higher than or equal to the illumination threshold, and when the camera accuracy information is higher than or equal to the accuracy threshold, the second detection strategy is determined as the target detection strategy; the second detection strategy is used to detect faces and abnormal behaviors simultaneously for high-resolution images to be detected.

3. The method according to claim 2, characterized in that According to the first detection strategy, abnormal behavior detection is performed on the image to be detected to obtain preliminary abnormality detection results, including: Detecting the driver's face from the image to be detected by a first neural network to obtain a face detection frame and a face confidence level; the first neural network is used to detect faces; When the face confidence level is greater than or equal to the face threshold, dynamically expanding the face detection frame to obtain an expanded area; The extended area is subjected to abnormal behavior detection through a second neural network, the abnormal behavior detection frame is output, and the abnormal confidence corresponding to the abnormal behavior detection frame is determined; the second neural network is used to detect different abnormal behaviors.

4. The method according to claim 2, characterized in that: According to the second detection strategy, abnormal behavior detection is performed on the image to be detected to obtain preliminary abnormality detection results, including: Detecting the driver's face and abnormal behavior from the image to be detected by a third neural network, and obtaining a face detection frame and a face confidence, an abnormal behavior detection frame and an abnormal confidence; the third neural network is used to detect faces and detect different abnormal behaviors; When the face confidence level is greater than or equal to the face threshold, dynamically expanding the face detection frame to obtain an expanded area; Determining the geometric center of the abnormal behavior detection frame; In a case where the geometric center of the abnormal behavior detection frame is located within the extended area, an abnormality confidence corresponding to the abnormal behavior detection frame is determined.

5. The method according to claim 4, characterized in that Also includes: When the geometric center of the abnormal behavior detection frame is not located within the extended area, it is determined to be normal driving.

6. The method according to claim 1, characterized in that Perform abnormal behavior detection on the supplementary area to obtain a final abnormality detection result, including: The supplementary area is detected by a fourth neural network to determine the target confidence that the supplementary area is abnormal behavior; the fourth neural network is used to determine whether the category corresponding to the abnormal behavior detection box is correct; When the target confidence is greater than or equal to the target threshold, determining it as abnormal driving; When the target confidence level is less than the target threshold, it is determined to be normal driving.

7. The method according to claim 6, characterized in that The target detection strategy includes a first detection strategy and a second detection strategy, and abnormal behavior detection is performed on the supplementary area to obtain a final abnormality detection result, including: In the case where all the images to be detected acquired at each moment are judged to be abnormal driving, respectively determining a first target confidence corresponding to the first detection strategy and a second target confidence corresponding to the second detection strategy; the first target confidence and the second target confidence respectively represent different abnormal behavior probabilities; assigning corresponding weights to the first target confidence and the second target confidence; A total confidence obtained by weighted summing the first target confidence and the second target confidence; Whether the driving is abnormal is determined based on the total confidence level.

8. The method according to any one of claims 1 to 7, characterized in that: Also includes: In a case where the abnormality confidence is greater than a maximum value of the abnormality detection interval, determining abnormal driving; In a case where the abnormality confidence is less than the minimum value of the abnormality detection interval, normal driving is determined.

9. An abnormal driving detection device, characterized in that: include: A strategy determination module is used to determine the target detection strategy for the preliminary detection stage of the image to be detected according to the environmental information corresponding to the target vehicle; The environmental information includes at least lighting information and camera accuracy information; A first detection module is used to perform abnormal behavior detection on the image to be detected according to the target detection strategy to obtain a preliminary abnormality detection result; The preliminary anomaly detection results include: an abnormal behavior detection frame and an abnormality confidence; A judgment module, used to judge whether the abnormal confidence level is in an abnormal detection interval; the abnormal detection interval indicates that it is impossible to judge whether the driving is abnormal by the abnormal confidence level; A supplementing module, used for supplementing the abnormal behavior detection frame with details to obtain a supplemented area when the abnormal confidence level is within the abnormal detection interval; The second detection module is used to perform abnormal behavior detection on the supplementary area to obtain a final abnormality detection result.

10. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the abnormal driving detection method according to any one of claims 1 to 8 are implemented.

Images