Dangerous driving behavior detection method and device, equipment and storage medium

Abstract

The invention discloses a dangerous driving behavior detection method and device, equipment and a storage medium, and relates to the technical field of vehicle control. The method comprises the following steps: acquiring multi-modal data of a target vehicle, wherein the multi-modal data comprises data of at least two modals; according to the multi-modal data, the dangerous driving behavior is predicted, a prediction result is obtained, and the prediction result is used for indicating the behavior type of the predicted dangerous driving behavior; and under the condition that the prediction result meets the alarm condition, an alarm action is executed, and the alarm action is a prompt action aiming at the dangerous driving behavior. According to the method, the safety of public transportation is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of vehicle control, and particularly to a method, device, equipment and storage medium for detecting dangerous driving behaviors. Background Art

[0002] In recent years, dangerous driving behaviors such as deliberately driving a motor vehicle into pedestrians have occurred frequently. These dangerous driving behaviors have caused a large number of innocent casualties and serious social property losses, posing a great threat to social public traffic safety and significantly reducing the sense of security of the public when traveling.

[0003] In the related art, after a dangerous driving behavior occurs in a target vehicle, other vehicles that are harmed by the dangerous driving behavior trigger safety measures. Among them, the safety measures include: the airbag deploying, sending a help message to the rescue center, sending a help message to other vehicles, etc.

[0004] In the related art, it is impossible to effectively intervene before a dangerous driving behavior occurs, and there are obvious lags and limitations in ensuring public traffic safety. Summary of the Invention

[0005] The embodiments of the present application provide a method, device, equipment and storage medium for detecting dangerous driving behaviors. The technical solutions provided by the embodiments of the present application are as follows:

[0006] According to one aspect of the embodiments of the present application, a method for detecting dangerous driving behaviors is provided. The method includes:

[0007] Obtaining multimodal data of a target vehicle, where the multimodal data includes data of at least two modalities;

[0008] Predicting a dangerous driving behavior based on the multimodal data to obtain a prediction result, where the prediction result is used to indicate the behavior type of the predicted dangerous driving behavior;

[0009] Performing an alarm action based on the prediction result, where the alarm action is a prompt action for the dangerous driving behavior.

[0010] According to one aspect of the embodiments of the present application, a device for detecting dangerous driving behaviors is provided. The device includes:

[0011] An obtaining module, configured to obtain multimodal data of a target vehicle, where the multimodal data includes data of at least two modalities;

[0012] A predicting module, configured to predict a dangerous driving behavior based on the multimodal data to obtain a prediction result, where the prediction result is used to indicate the behavior type of the predicted dangerous driving behavior;

[0013] An alarm module, configured to perform an alarm action based on the prediction result, where the alarm action is a prompt action for the dangerous driving behavior.

[0014] According to one aspect of the embodiments of the present application, a terminal device is provided. The terminal device includes a processor and a memory. A computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned method for detecting dangerous driving behavior.

[0015] According to one aspect of the embodiments of the present application, a computer-readable storage medium is provided. A computer program is stored in the readable storage medium, and the computer program is loaded and executed by a processor to implement the above-mentioned method for detecting dangerous driving behavior.

[0016] According to one aspect of the embodiments of the present application, a computer program product is provided. The computer program product includes a computer program, and the computer program is loaded and executed by a processor to implement the above-mentioned method for detecting dangerous driving behavior.

[0017] The technical solutions provided by the embodiments of the present application at least include the following beneficial effects:

[0018] By fusing different modalities of data collected by the target vehicle and predicting dangerous driving behavior based on the fused multi-modal data, the accuracy and reliability of predicting dangerous driving behavior are improved, ensuring accurate identification of dangerous driving behavior in a complex traffic environment. By promptly performing an alarm action for the dangerous driving behavior, the safety of public transportation is guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a schematic diagram of the implementation environment of the solution provided by an embodiment of the present application;

[0020] Figure 2 is a flowchart of the method for detecting dangerous driving behavior provided by an embodiment of the present application;

[0021] Figure 3 is a block diagram of the device for detecting dangerous driving behavior provided by an embodiment of the present application;

[0022] Figure 4 is a block diagram of the structure of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] To make the objectives, technical solutions, and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings.

[0024] Please refer to Figure 1, which shows a schematic diagram of the solution implementation environment provided by an embodiment of the present application. The solution implementation environment may include: a target vehicle 10 and a server 20.

[0025] The target vehicle 10 is a vehicle capable of executing a detection method for dangerous driving behaviors. In some embodiments, the target vehicle 10 includes a public traffic safety controller 11. The public traffic safety controller 11 is used to execute a detection method for dangerous driving behaviors to detect whether there are dangerous driving behaviors in the target vehicle 10. The public traffic safety controller 11 is an electronic device with data calculation and processing functions. In some embodiments, the public traffic safety controller 11 includes at least one of the following: a danger identification module 11-1, an alarm module 11-2, and a self-check module 11-3. The danger identification module 11-1 is used to detect and identify dangerous driving behaviors. The alarm module 11-2 is used to execute an alarm action for the detected dangerous driving behaviors. The self-check module 11-3 is used to detect the target vehicle 10.

[0026] In some embodiments, the target vehicle 10 further includes a communication control module 12 and a vehicle gateway 13. The communication control module 12 is used for data transmission, communication, real-time positioning and tracking, etc. In some embodiments, the communication control module 12 can be implemented as a Telematics BOX (abbreviated as T-BOX). The T-BOX is a hardware module for implementing vehicle network services and is responsible for connecting the target vehicle 10 and the server 20. The vehicle gateway 13 is used to coordinate data exchange between different bus protocols in the target vehicle 10 and realize the interconnection and intercommunication between various components, functional domains, and external networks.

[0027] In some embodiments, the target vehicle 10 further includes at least one of the following: a power domain 14, a chassis domain 15, a body domain 16, a cockpit domain 17, an autonomous driving domain 18, etc., and may also include other components, which are not limited in the embodiments of the present application. The power domain 14 is responsible for driving the power system management of the target vehicle 10, including at least one of the following: an internal combustion engine, an electric motor, a battery, a transmission, etc., and may also include other components, which are not limited in the embodiments of the present application. The chassis domain 15 is responsible for managing the driving stability, steering, braking, and suspension systems of the target vehicle 10. The body domain 16 is responsible for controlling the body functions of the target vehicle 10, including at least one of the following: doors, windows, lights, air conditioners, etc. The cockpit domain 17 is responsible for managing human-machine interaction and integrating software and hardware such as instrument clusters, central controls, and voice. The autonomous driving domain 18 is used to realize environment perception, path planning, and vehicle control, etc.

[0028] The server 20 is used to provide vehicle networking services for the target vehicle 10. In some embodiments, the server 20 may be implemented as a public server for social vehicles. In some embodiments, the public server for social vehicles may be a server of a Telematics Service Provider (TSP). Exemplarily, the server mentioned above may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms, but is not limited thereto.

[0029] In some embodiments, the server 20 includes a public traffic safety module 21. The public traffic safety module 21 is used to receive and respond to information or requests related to public traffic safety sent by the target vehicle 10.

[0030] The target vehicle 10 and the server 20 can communicate with each other through a network. The network can be a wired network or a wireless network.

[0031] Please refer to Figure 2 , which shows a flowchart of a method for detecting dangerous driving behaviors provided by an embodiment of the present application. The execution subject of each step of this method can be a computer device. For example, this computer device can be Figure 1 the public traffic safety controller 11 in the solution implementation environment shown. This method may include at least one of the following steps (210 to 230):

[0032] Step 210, obtaining multi-modal data of the target vehicle, where the multi-modal data includes data of at least two modalities.

[0033] The multi-modal data of the target vehicle refers to a combination of various types of data obtained by the target vehicle through different modalities. A modality refers to the type or information of data. Different modalities represent different-dimensional data expression methods. In some embodiments, the modality includes at least one of the following: visual modality, auditory modality, text modality, sensor modality, etc., and may also include other modalities, which are not limited in the embodiments of the present application.

[0034] The visual modality refers to the image data formed by the object shape, color, and spatial information captured through optical signals. In some embodiments, the data forms of the visual modality include at least one of the following: images, videos, thermal imaging, etc., and may also include other data forms, which are not limited in the embodiments of the present application. In some embodiments, the data of the visual modality is obtained through at least one of the following data sources: monocular cameras, wide-angle cameras, binocular cameras, infrared cameras, etc., and may also include other data sources, which are not limited in the embodiments of the present application. In some embodiments, the above cameras may be disposed at at least one of the following positions in the target vehicle: inside the front windshield, at the four corners of the vehicle, the front bumper, the roof, the front grille, the roof, etc., and may also be disposed at other positions of the target vehicle, which are not limited in the embodiments of the present application.

[0035] The auditory modality is the audio information generated based on sound wave vibrations. In some embodiments, the data forms of the auditory modality include at least one of the following: voice signals, ambient sounds, voiceprints, etc., and may also include other data forms, which are not limited in the embodiments of the present application. In some embodiments, the data of the auditory modality is obtained through a microphone or a vibration sensor, and may also be obtained through other means, which are not limited in the embodiments of the present application.

[0036] The text modality is the structured or unstructured language information carried by symbols. In some embodiments, the data forms of the text modality include at least one of the following: natural languages, structured data, symbol sequences, etc., and may also include other data forms, which are not limited in the embodiments of the present application. In some embodiments, the sources of the data of the text modality may include at least one of the following: text messages, vehicle diagnostic logs, CAN (Controller Area Network) bus protocol messages, etc., and may also include other sources, which are not limited in the embodiments of the present application.

[0037] The sensor modality is the quantified data collected by sensors. In some embodiments, the data forms of the sensor modality include at least one of the following: time series, analog signals, digital signals, multi-dimensional signals, etc., and may also include other data forms, which are not limited in the embodiments of the present application. In some embodiments, the data of the sensor modality is obtained through at least one of the following data sources: inertial sensors, pressure sensors, optical sensors, magnetoresistive sensors, gas sensors, acoustic sensors, chemical sensors, biosensors, etc., and may also include other data sources, which are not limited in the embodiments of the present application.

[0038] In some embodiments, the multimodal data includes at least one of the following: camera data, radar data, lidar data, vehicle speed, throttle opening, Internet data, weather information, geographical location information, GPS (Global Positioning System) trajectory, vehicle acceleration, steering wheel angle, audio, etc. Other data may also be included, and the embodiments of the present application do not limit this.

[0039] In some embodiments, data of at least one modality is obtained from at least one data source to obtain multimodal data.

[0040] Step 220: Predict a dangerous driving behavior based on the multimodal data to obtain a prediction result, where the prediction result is used to indicate the behavior type of the predicted dangerous driving behavior.

[0041] A dangerous driving behavior refers to a driving behavior that threatens public traffic safety.

[0042] In some embodiments, a preprocessing operation is performed on the multimodal data to obtain preprocessed multimodal data. In some embodiments, the preprocessing operation includes at least one of the following: normalization processing, key frame extraction, acceleration curve plotting, GPS trajectory map matching, etc. Other preprocessing operations may also be included, and the embodiments of the present application do not limit this.

[0043] Normalization processing refers to scaling the data into a specific interval. For example, the specific interval is [0, 1] or [-1, 1]. For another example, the vehicle speed is scaled into a specific interval. For another example, the vehicle acceleration is scaled into a specific interval.

[0044] Key frame extraction refers to selecting representative or informative frames from a continuous video stream or image sequence. Acceleration curve plotting refers to presenting the change of vehicle acceleration over time in a graphical manner. GPS trajectory map matching refers to the process of comparing and matching the position points recorded by GPS with the actual road map.

[0045] In some embodiments, the prediction result includes at least one of the following: the behavior type of the dangerous driving behavior, a danger score, the occurrence time, the occurrence location, the prediction confidence level, etc. Other information may also be included, and the embodiments of the present application do not limit this.

[0046] The behavior types of dangerous driving behaviors are used to distinguish different driving behaviors that pose a threat to public traffic safety. The danger score is used to indicate the degree of danger of a dangerous driving behavior. The higher the danger score, the more dangerous the dangerous driving behavior. The lower the danger score, the safer the dangerous driving behavior. The occurrence time refers to the specific time when the dangerous driving behavior occurs. For example, the time is recorded using UTC (Universal Time Coordinated), accurate to milliseconds. The occurrence location refers to the exact location where the dangerous driving behavior occurs. For example, the geographical location information of the target vehicle when the dangerous driving behavior occurs is obtained through a map API (Application Programming Interface). The prediction confidence is used to indicate the degree of uncertainty of the occurrence of a dangerous driving behavior. The greater the prediction confidence, the smaller the likelihood of the occurrence of the dangerous driving behavior. The smaller the prediction confidence, the greater the likelihood of the occurrence of the dangerous driving behavior.

[0047] In some embodiments, the multimodal data includes at least one of the following: camera images, radar data, and vehicle speed.

[0048] The camera images are obtained by at least one camera provided on the target vehicle. In some embodiments, the camera images can be directly obtained from the at least one camera, or can be obtained from the video captured by the at least one camera.

[0049] The radar data is obtained by at least one radar provided on the target vehicle. In some embodiments, the radar includes at least one of the following: millimeter-wave radar, lidar, ultrasonic radar, etc., and may also include other radars, which are not limited in the embodiments of the present application. In some embodiments, the radar data can be implemented as point cloud data.

[0050] The vehicle speed refers to the driving speed of the target vehicle. In some embodiments, the vehicle speed can be obtained by at least one of the following: wheel speed sensors, GPS speed, inertial measurement units, etc., and can also be obtained by other means, which are not limited in the embodiments of the present application.

[0051] In some embodiments, feature extraction is performed on the camera images to obtain the image features of the camera images; the camera images and the data of other modalities in the multimodal data are fused to obtain the fused multimodal data; based on the fused multimodal data, dangerous driving behaviors are predicted to obtain a prediction result.

[0052] Image features are useful features extracted from camera images. In some embodiments, the image features include at least one of the following: global features, local features, depth features, etc., and may also include other features, which are not limited in the embodiments of this application. Global features are used to indicate the overall environmental perception of the entire camera image. For example, global features include weather features, lighting features, etc. Local features are used to describe specific regions in the camera image. For example, the specific region can be other vehicles, pedestrians, checkpoints, traffic lights, etc. Depth features refer to high-level features. In some embodiments, the image features in the camera image can be extracted by at least one of the following models: ResNet, YOLO (You Only Look Once, an object detection model that only detects once), Faster R-CNN (Faster Region-based Convolutional Neural Network, a faster region-based convolutional neural network), ViT (Vision Transformer, a vision transformer), etc., and other models can also be used to extract image features, which are not limited in the embodiments of this application.

[0053] In some embodiments, at least one modality data in the multimodal data is synchronized in space and time to obtain the multimodal data after space-time synchronization; feature extraction is performed on the camera image to obtain the image features of the camera image; through the feature fusion technology, the camera image and the data of other modalities in the multimodal data are fused to obtain the fused multimodal data.

[0054] Feature fusion refers to mapping the features of different modalities into the same vector space. In some embodiments, feature fusion is performed by means of weighted or attention mechanisms. In some embodiments, the feature fusion technology includes at least one of the following: data-level fusion, feature-level fusion, and decision-level fusion, etc., and may also include other feature fusion technologies, which are not limited in the embodiments of this application. Data-level fusion refers to directly splicing at least one modality data in the multimodal data, but strict alignment of space-time coordinates is required. Feature-level fusion refers to fusing the features extracted from each modality first. Decision-level fusion refers to fusing the prediction results of each modality after separately predicting each modality.

[0055] By the above method, fusing data of different modalities helps to improve the reliability and accuracy of the prediction results for dangerous driving behaviors.

[0056] In some embodiments, the fused multimodal data is input into a behavior prediction model, and the behavior prediction model outputs a prediction result based on the determination conditions of at least one behavior type. The prediction result includes the prediction probabilities of at least one behavior type, and the behavior prediction model is used to predict dangerous driving behaviors.

[0057] The behavior prediction model is an AI (Artificial Intelligence) model used to predict dangerous driving behaviors. In some embodiments, the behavior prediction model can be implemented using at least one of the following models: CNN (Convolutional Neural Network), LSTM (Long Short-Term Memory), MLP (Multilayer Perceptron), ViT, classifier, etc., and can also be implemented by other models, which is not limited in the embodiments of this application.

[0058] The determination condition of the behavior type refers to the rule or threshold for the model to judge whether a specific dangerous driving behavior occurs.

[0059] In some embodiments, the behavior type includes at least one of the following: ramming into a crowd, breaking through a checkpoint, speeding, and fleeing the scene of an accident.

[0060] Ramming into pedestrians refers to the dangerous driving behavior in which the target vehicle actively or out of control drives towards a crowded area. Breaking through a checkpoint refers to the dangerous driving behavior in which the target vehicle forcibly breaks through a checkpoint. A checkpoint refers to an obstacle that blocks the progress of a vehicle. In some embodiments, the checkpoint can include at least one of the following: a barrier gate, a railing, a door, a checkpoint sign, a water horse, a residential community access control, a toll booth, etc., and can also include other checkpoints, which is not limited in the embodiments of this application. Speeding refers to the dangerous driving behavior in which the target vehicle seriously exceeds the road speed limit. Fleeing the scene of an accident refers to the dangerous driving behavior in which the target vehicle does not stop to handle the accident after a traffic accident, but instead drives away from the accident scene.

[0061] In some embodiments, the behavior type can also include insufficient braking. Insufficient braking refers to the situation where the target vehicle is unable to effectively reduce its speed or stop when it encounters a situation that requires stopping or decelerating, resulting in the braking distance required by the target vehicle exceeding the actual achievable braking distance.

[0062] (1) Ramming into a crowd

[0063] In some embodiments, when the behavior type is ramming into a crowd, the behavior determination conditions include: the number of pedestrians is greater than the first threshold, the distance between the target vehicle and the crowd is greater than the braking distance threshold, and no braking signal is received. The number of pedestrians is obtained by recognizing pedestrians in the camera image. The distance between the target vehicle and the crowd is determined based on radar data. The braking distance threshold is determined based on the vehicle speed. The braking signal is used to indicate that the target vehicle is in a braking state.

[0064] The number of pedestrians refers to the number of pedestrians within the detection area. The detection area refers to the spatial range that needs to be monitored when determining a charging crowd. For example, the detection area is the front area of the target vehicle. In some embodiments, the detection area may be rectangular, may be fan-shaped, or may be other shapes, which are not limited in the embodiments of the present application. In some embodiments, based on the road width where the target vehicle is located, the lateral coverage width of the detection area is determined; based on the vehicle speed, the longitudinal coverage distance of the detection area is determined. For example, the lateral coverage width of the detection area is 1.5 times the road width where the target vehicle is located. For example, according to the current speed of the target vehicle, the system response time, and the buffer distance, the longitudinal coverage distance of the detection area is determined. In some embodiments, in the case where the behavior type is charging the crowd, the behavior determination condition may further include: the pedestrian density is greater than the sixth threshold, and the pedestrian density is determined based on the number of pedestrians and the area of the detection area.

[0065] In some embodiments, the camera image is input into a pedestrian detection model, and the pedestrian detection model outputs the number of pedestrians. The pedestrian detection model is used to detect and locate pedestrians in an image or video stream. In some embodiments, the pedestrian detection model may be an existing model or a model designed and trained by a person skilled in the art, which is not limited in the embodiments of the present application. In some embodiments, the first threshold is preset by a person skilled in the art, which is not limited in the embodiments of the present application. In some embodiments, the number of the first thresholds is multiple; according to the road type of the road where the target vehicle is located and the current time period, the target first threshold is determined from the multiple first thresholds, and the target first threshold is used to determine whether the number of pedestrians meets the behavior determination condition. In some embodiments, the road type includes at least one of the following: urban road, school section, highway, rural road, etc., and may also include other road types, which are not limited in the embodiments of the present application.

[0066] When the driver steps on the brake pedal, a braking signal is generated.

[0067] In some embodiments, based on the radar data, the distance between the target vehicle and at least one pedestrian is calculated; the distance between the nearest pedestrian and the target vehicle is determined as the distance between the target vehicle and the crowd. The braking distance threshold refers to the theoretical braking distance of the target vehicle. In some embodiments, according to the vehicle speed, the friction coefficient, and the acceleration due to gravity, the braking distance threshold is determined. For example, the braking distance threshold D = v 2 / (2μg), where μ is the friction coefficient and g is the acceleration due to gravity.

[0068] In some embodiments, when the behavior type is charging into a crowd, the behavior determination condition further includes: the vehicle acceleration continuously increases. In some embodiments, based on the throttle opening, it is determined whether the acceleration of the target vehicle continuously increases. The throttle opening is used to indicate the depth to which the driver depresses the accelerator pedal. The throttle opening can reflect the intensity of the driver's command to accelerate the target vehicle. That is to say, the throttle opening can be used to determine whether there is an intention of accelerating and charging. In some embodiments, the throttle opening and the vehicle speed are obtained through the CAN bus; based on the throttle opening and the vehicle speed, the vehicle acceleration of the target vehicle is determined.

[0069] (2) Behavior of running through a checkpoint

[0070] In some embodiments, when the behavior type is the behavior of running through a checkpoint, the behavior determination condition includes: the checkpoint is in a closed state, the distance between the target vehicle and the checkpoint is less than a third threshold, and the vehicle speed is greater than a fourth threshold. The checkpoint is obtained by performing target recognition on the camera image, and the distance between the target vehicle and the checkpoint is determined based on the radar data.

[0071] In some embodiments, target recognition is performed on the camera image to obtain a checkpoint recognition result, and the checkpoint recognition result is used to indicate whether there is a checkpoint and the position of the checkpoint in the camera image. In some embodiments, the camera image is input into a target detection model, and the checkpoint recognition result is output by the target detection model. The checkpoint position is used to indicate the position of the checkpoint in the camera image. In some embodiments, the checkpoint position includes the checkpoint angle; when the checkpoint angle is greater than a seventh threshold, it is determined that the checkpoint is in an open state; when the checkpoint angle is less than the seventh threshold, it is determined that the checkpoint is in a closed state. In some embodiments, the seventh threshold is preset by those skilled in the relevant art, and the embodiments of the present application do not limit this. In some embodiments, the checkpoint recognition result is also used to identify the checkpoint and the checkpoint identifier in the camera image.

[0072] In some embodiments, based on the radar data, the distance between the target vehicle and the recognized checkpoint is calculated; the distance between the nearest checkpoint and the target vehicle is determined as the distance between the target vehicle and the checkpoint.

[0073] In some embodiments, when the behavior type is the behavior of running through a barrier, the behavior determination condition further includes: the vehicle speed is greater than an eighth threshold, and the vehicle acceleration continuously increases. In some embodiments, based on the GPS trajectory data and the vehicle speed, the vehicle acceleration is determined. The GPS trajectory data refers to the position, speed, and time information of the target vehicle collected through the global positioning system. In some embodiments, the GPS trajectory data includes at least one of the following: timestamp, latitude, longitude, altitude, speed, heading, longitude, etc., and may also include other information, which is not limited in the embodiments of the present application.

[0074] Exemplarily, if the target vehicle continuously accelerates and approaches (speed > 5 km / h and distance < 10 m) when the railing is in the closed state, it is determined as a behavior of crashing through the barrier.

[0075] (III) Speeding

[0076] In some embodiments, when the behavior type is speeding, the behavior determination conditions include: the vehicle speed is greater than the road speed limit threshold, and the road speed limit threshold is determined based on the road type where the target vehicle is located.

[0077] The road speed limit threshold is the minimum threshold for determining the behavior type as speeding. In some embodiments, the road speed limit threshold is determined based on the speed limit and speeding ratio of the road type where the target vehicle is located. The speeding ratio defines the ratio of the speeding behavior, that is, the allowable speeding range.

[0078] Exemplarily, the speed limit on the highway is 120 km / h, the speed limit on the urban road is 50 km / h, the speed limit on the rural road is 30 km / h, and the speed limit in the school section is 30 km / h. For example, the speeding ratio can be 20% or 100%. In some embodiments, the speeding ratio is stipulated by relevant regulations, and the embodiments of the present application do not make any limitations.

[0079] (IV) Hit-and-run

[0080] In some embodiments, when the behavior type is hit-and-run, the behavior determination conditions include: there is a vehicle anomaly, the vehicle speed is greater than the fifth threshold after detecting a collision impact, and the vehicle anomaly includes at least one of the following: detecting scattered objects, abnormal positions of other vehicles, and abnormal body of the target vehicle.

[0081] In some embodiments, traffic accidents may include at least one of the following: rear-end collision, multi-vehicle chain collision, side collision, head-on collision, etc., and may also include other traffic accidents, which are not limited in the embodiments of the present application.

[0082] Scattered objects refer to the items, parts or fragments of the vehicle scattered on the road after a traffic accident. In some embodiments, scattered objects may include at least one of the following: body parts, mechanical parts, fragments, goods, personal items, leakage of vehicle oil, damage to road signs, damage to traffic facilities, etc., and may also include other scattered objects, which are not limited in the embodiments of the present application. In some embodiments, target recognition is performed on the camera image to obtain the scattered object recognition result, and the scattered object recognition result is used to indicate whether there are scattered objects in the camera image. In some embodiments, the camera image is input into the target detection model, and the target detection model outputs the scattered object recognition result.

[0083] An abnormal position of other vehicles means that the position of the vehicle collided by the target vehicle is abnormal. In some embodiments, the abnormal position of other vehicles includes at least one of the following: rollover, sudden stop, large-angle rotation, etc., and may also include other abnormal positions of other vehicles, which are not limited in the embodiments of the present application. In some embodiments, in some embodiments, target recognition is performed on the camera image to obtain a position abnormality recognition result, and the position abnormality recognition result is used to indicate whether there are other vehicles with abnormal positions in the camera image. In some embodiments, the camera image is input into a target detection model, and the target detection model outputs a position abnormality recognition result.

[0084] An abnormal body of the target vehicle means the damage condition of the target vehicle body after a collision impact. In some embodiments, the abnormal body of the target vehicle includes at least one of the following: body dent, body crack, body fracture, window breakage, headlight damage, wheel damage, etc., and may also include other abnormal bodies, which are not limited in the embodiments of the present application. In some embodiments, the abnormal body of the target vehicle can be detected by at least one of the following methods: camera, light sensor, laser, radar, etc., and may also include other methods, which are not limited in the embodiments of the present application.

[0085] In some embodiments, it can be determined whether the target vehicle has undergone a collision impact by at least one of the following: accelerometer, airbag sensor, wheel speed sensor, radar data, camera image, etc., and it can also be determined whether the target vehicle has undergone a collision impact by other methods, which are not limited in the embodiments of the present application.

[0086] In some embodiments, when the behavior type is hit-and-run, the behavior determination conditions further include: the vehicle acceleration is in an increasing state within a first time period, no braking signal is received, and the target vehicle moves away from the accident location. The first time period is used to detect whether the target vehicle continues to accelerate. In some embodiments, the first time period is set by those skilled in the art, which is not limited in the embodiments of the present application. In some embodiments, according to the GPS trajectory data, it is determined whether the target vehicle moves away from the accident location.

[0087] In some embodiments, for the judgment process of the same set of behavior determination conditions, performing target recognition on the camera image once can obtain multiple recognition results. For example, the multiple recognition results include at least one of the following: the number of pedestrians, checkpoint recognition result, debris recognition result, position abnormality recognition result, etc., and may also include other recognition results, which are not limited in the embodiments of the present application.

[0088] (V) Insufficient braking

[0089] In some embodiments, the behavior type includes insufficient braking; in the case where the behavior type is insufficient braking, the behavior determination condition includes: the predicted braking distance is greater than the maximum braking threshold, and the maximum braking threshold is related to at least one of the following: the road type of the road where the target vehicle is located, weather information, and OBD (On-Board Diagnostics) data. The predicted braking distance is determined based on the vehicle speed of the target vehicle. The maximum braking threshold is determined based on the road speed limit.

[0090] In some embodiments, the speed limit of the road where the target vehicle is located is determined according to the road type of the road where the target vehicle is located.

[0091] The weather information is used to indicate the weather conditions of the environment where the target vehicle is located. In some embodiments, the weather information can be obtained through at least one of the following: receiving weather data from a weather station, weather data shared by the vehicle, a windshield rain sensor, an infrared road surface temperature sensor, etc. The weather information can also be obtained through other means, and the embodiments of the present application do not limit this. In some embodiments, the friction coefficient is determined according to the weather information.

[0092] OBD (On-Board Diagnostics) is an automatic diagnostic and reporting system on the target vehicle, which is used to monitor and record various operating states and fault information of the vehicle. In some embodiments, the OBD data can include at least one of the following: brake pad wear, brake fluid pressure, and brake disc temperature, etc. The OBD data can also include other data, and the embodiments of the present application do not limit this.

[0093] In some embodiments, the predicted braking distance is determined according to the vehicle speed, friction coefficient, gravitational acceleration, and OBD data; the maximum braking threshold is determined according to the speed limit of the road where the target vehicle is located, friction coefficient, gravitational acceleration, and OBD data.

[0094] Through the above method, different behavior determination conditions are set for different dangerous driving behaviors. Based on multi-modal data and different behavior determination conditions, it is possible to more accurately determine the dangerous driving behaviors that the target vehicle may perform.

[0095] Step 230, based on the prediction result, perform an alarm action, and the alarm action is a prompt action for dangerous driving behaviors.

[0096] The alarm action is used to remind the drivers of other vehicles, pedestrians, or the monitoring system to pay attention to the possible dangerous driving behaviors of the target vehicle. In some embodiments, the alarm actions corresponding to different behavior types are different. In some embodiments, the alarm action is performed according to the prediction result and the priority of each of at least one behavior type.

[0097] In some embodiments, performing an alarm action includes at least one of the following.

[0098] (1) Behavior recording

[0099] In some embodiments, behavior information of a dangerous driving behavior is recorded, and the behavior information is information related to the dangerous driving behavior and the target vehicle.

[0100] In some embodiments, the behavior information corresponding to different behavior types of dangerous driving behaviors is different. In some embodiments, according to the behavior type of the dangerous driving behavior, at least one record data is obtained; the at least one record data is stored in the local memory of the target vehicle.

[0101] In some embodiments, the above at least one record data includes at least one of the following: timestamp, geographical location information, vehicle speed, vehicle acceleration, steering wheel angle, sensor data, camera image, radar data, laser data, GPS track data, weather environment information, etc., and other data may also be included, which is not limited in the embodiments of the present application.

[0102] (2) Information reporting

[0103] In some embodiments, the behavior information of the dangerous driving behavior is reported.

[0104] In some embodiments, the behavior information of the dangerous driving behavior is sent to an external system. In some embodiments, the external system includes at least one of the following: Server 20, nearby vehicles, traffic management platform, etc., and other external systems may also be included, which is not limited in the embodiments of the present application.

[0105] (3) Visual reminder

[0106] In some embodiments, the lights of the target vehicle are controlled to flash. The target vehicle reminds the drivers and pedestrians of other surrounding vehicles of the dangerous driving behavior of the target vehicle through the flashing of the lights. In some embodiments, controlling the lights of the target vehicle to flash includes at least one of the following: rapid flashing of the headlights, continuous flashing of the brake lights, continuous flashing of the turn signals, flashing of the dashboard warning lights, etc., and other flashing methods of the lights may also be included, which is not limited in the embodiments of the present application.

[0107] (4) Auditory reminder

[0108] In some embodiments, the target vehicle is controlled to emit a sound signal, which is used to indicate the existence of dangerous driving behavior. The target vehicle uses a sound alarm to remind the drivers of other surrounding vehicles and pedestrians to pay attention to the dangerous driving behavior of the target vehicle. In some embodiments, controlling the target vehicle to emit a sound signal includes at least one of the following: beeping, voice prompt, honking, external vehicle alarm, etc., and may also include other ways of emitting sound signals, which are not limited in the embodiments of the present application.

[0109] It should be noted that the above warning actions can be used alone or in combination, which is not limited in the embodiments of the present application.

[0110] Through the above method, according to the prediction result, a warning is sent to the drivers of other vehicles, pedestrians or relevant supervision systems in a timely manner to avoid or reduce the safety risks brought by dangerous driving behavior.

[0111] In summary, the technical solution provided by the embodiments of the present application improves the accuracy and reliability of predicting dangerous driving behavior by fusing different modal data collected by the target vehicle and predicting dangerous driving behavior based on the fused multi-modal data, ensures the accurate identification of dangerous driving behavior in a complex traffic environment, and guarantees the safety of public transportation by timely performing warning actions for dangerous driving behavior.

[0112] The self-check process of the target vehicle is introduced below.

[0113] In some embodiments, the target vehicle is detected according to multi-modal data to obtain a self-check abnormal result, which is used to indicate whether the target vehicle has an abnormality; when the target vehicle is in an unstarted state and the self-check abnormal result indicates that the target vehicle has an abnormality, a fault message is displayed, and the fault message is used to indicate the abnormal situation of the target vehicle; when the target vehicle is in a started state and the self-check abnormal result indicates that the target vehicle has an abnormality, the target vehicle is braked.

[0114] In some embodiments, the multi-modal data is analyzed to determine whether the target vehicle has an abnormality, and a self-check abnormal result is obtained. In some embodiments, the detection of the target vehicle includes at least one of the following: circuit integrity, sensor accuracy, communication status, etc., and may also include other detections, which are not limited in the embodiments of the present application.

[0115] In some embodiments, based on the self-check cycle, the target vehicle performs periodic self-checks, and the self-check cycle is used to indicate the time interval between two adjacent self-checks performed by the target vehicle. In some embodiments, the self-check cycle is preset by relevant technical personnel, which is not limited in the embodiments of the present application. For example, relevant technical personnel set the self-check cycle to 1 second, that is, the target vehicle performs a self-check every 1 second.

[0116] The fault information is used to prompt the driver to perform maintenance inspection on the target vehicle.

[0117] When the target vehicle is in an unstarted state and the self-check abnormal result shows an abnormality, cut off the power source for operating the target vehicle. The power source is used to drive the target vehicle to travel. When the target vehicle is a fuel vehicle, the power source of the target vehicle is the engine. When the target vehicle is an electric vehicle, the power source of the target vehicle is the motor. In some embodiments, prevent the high-voltage power for engine ignition from being powered on by cutting off the engine ignition circuit or the high-voltage power supply circuit.

[0118] In some embodiments, when the target vehicle is in a started state and the self-check abnormal result indicates that the target vehicle has an abnormality, braking the target vehicle includes at least one of the following: cutting off the power source for operating the target vehicle; starting the braking system of the target vehicle, where the braking system is used to forcibly decelerate the target vehicle; fixing the vehicle steering of the target vehicle, where the vehicle steering is used to indicate the degree to which the driving direction of the target vehicle deviates from the center line of the target vehicle. Cut off the engine fuel supply or the high-voltage power output, and then start the vehicle braking system to brake the target vehicle to a stop with the maximum braking force, and lock the steering system in the current position to prevent the target vehicle from steering.

[0119] By the above method, by detecting the target vehicle and processing the target vehicle when the self-check abnormal result indicates that the target vehicle has an abnormality, it is possible to prevent the target vehicle from operating with problems and ensure the reliability of the target vehicle itself.

[0120] The following is an embodiment of the device of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the method embodiment of the present application.

[0121] Please refer to Figure 3 which shows a block diagram of a dangerous driving behavior device provided by an embodiment of the present application. This device has the functions implemented in the above examples, and the functions can be implemented by hardware or by hardware executing corresponding software. This device can be the computer device introduced above or can be set in the computer device. As Figure 3 shown, the device 300 may include an acquisition module 310, a prediction module 320, and an alarm module 330.

[0122] The acquisition module 310 is used to acquire multi-modal data of the target vehicle, and the multi-modal data includes data of at least two modalities.

[0123] The prediction module 320 is used to predict dangerous driving behavior according to the multi-modal data to obtain a prediction result, and the prediction result is used to indicate the behavior type of the predicted dangerous driving behavior.

[0124] An alarm module 330, configured to perform an alarm action based on the prediction result, where the alarm action is a prompt action for the dangerous driving behavior.

[0125] In some embodiments, the multimodal data includes at least one of the following: camera images, radar data, and vehicle speed. The prediction module 320 is configured to extract features from the camera images to obtain image features of the camera images; fuse the camera images and data of other modalities in the multimodal data to obtain fused multimodal data; and predict the dangerous driving behavior based on the fused multimodal data to obtain the prediction result.

[0126] In some embodiments, the prediction module 320 is further configured to input the fused multimodal data into a behavior prediction model, and the behavior prediction model outputs the prediction result based on determination conditions of at least one behavior type. The prediction result includes prediction probabilities of at least one behavior type, and the behavior prediction model is used to predict the dangerous driving behavior.

[0127] In some embodiments, the behavior type includes at least one of the following: ramming a crowd, breaking through a checkpoint, speeding, and accident escape. When the behavior type is ramming a crowd, the behavior determination conditions include: the number of pedestrians is greater than a first threshold, the distance between the target vehicle and the crowd is greater than a braking distance threshold, and no braking signal is received. The number of pedestrians is obtained by performing pedestrian recognition on the camera images, the distance between the target vehicle and the crowd is determined based on the radar data, the braking distance threshold is determined based on the vehicle speed, and the braking signal is used to indicate that the target vehicle is in a braking state. When the behavior type is breaking through a checkpoint, the behavior determination conditions include: the checkpoint is in a closed state, the distance between the target vehicle and the checkpoint is less than a third threshold, and the vehicle speed is greater than a fourth threshold. The checkpoint is obtained by performing target recognition on the camera images, and the distance between the target vehicle and the checkpoint is determined based on the radar data. When the behavior type is speeding, the behavior determination conditions include: the vehicle speed is greater than a road speed limit threshold, and the road speed limit threshold is determined based on the road type where the target vehicle is located. When the behavior type is accident escape, the behavior determination conditions include: there is a vehicle anomaly, and the vehicle speed is greater than a fifth threshold after detecting a collision impact. The vehicle anomaly includes at least one of the following: detecting debris, abnormal positions of other vehicles, and abnormal body of the target vehicle.

[0128] In some embodiments, performing the warning action includes at least one of the following: recording behavior information of the dangerous driving behavior, where the behavior information is information related to the dangerous driving behavior and the target vehicle; reporting the behavior information of the dangerous driving behavior; controlling the lights of the target vehicle to flash; controlling the target vehicle to emit a sound signal, where the sound signal is used to indicate the existence of the dangerous driving behavior.

[0129] In some embodiments, the device 300 further includes a self-check module (not shown in Figure 3 ), which is used to detect the target vehicle according to the multimodal data to obtain a self-check abnormal result, where the self-check abnormal result is used to indicate whether the target vehicle has an abnormality; in the case where the target vehicle is in an unstarted state and the self-check abnormal result indicates that the target vehicle has an abnormality, display a fault information, where the fault information is used to indicate the abnormal situation of the target vehicle; in the case where the target vehicle is in a started state and the self-check abnormal result indicates that the target vehicle has an abnormality, brake the target vehicle.

[0130] In summary, the technical solution provided by the embodiments of the present application fuses different modal data collected by the target vehicle, and predicts dangerous driving behaviors based on the fused multimodal data, improving the accuracy and reliability of predicting dangerous driving behaviors, ensuring accurate identification of dangerous driving behaviors in a complex traffic environment, and guaranteeing the safety of public transportation by promptly performing warning actions for dangerous driving behaviors.

[0131] It should be noted that when the device provided in the above embodiments realizes its functions, only the above division of each functional module is used for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiments and the method embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments and will not be repeated here.

[0132] Please refer to Figure 4 , which shows a structural block diagram of a computer device 400 provided by an embodiment of the present application. The computer device 400 can be Figure 1 the public traffic safety controller 11 in the shown implementation environment, and is used to implement the method for detecting dangerous driving behaviors provided in the above embodiments. Specifically:

[0133] Generally, the computer device 400 includes: a processor 410 and a memory 420.

[0134] The processor 410 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. The processor 410 may be implemented in at least one of the hardware forms of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 410 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 410 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 410 may further include an AI processor, which is used to process computational operations related to machine learning.

[0135] The memory 420 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 420 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage media in the memory 420 are used to store a computer program, and the computer program is configured to be executed by one or more processors to implement the method for detecting dangerous driving behaviors.

[0136] Those skilled in the art can understand that Figure 4 the structure shown in does not constitute a limitation on the computer device 400, and it may include more or fewer components than shown in the figure, or combine certain components, or adopt different component arrangements.

[0137] In an exemplary embodiment, a computer-readable storage medium is further provided. A computer program is stored in the storage medium, and when the computer program is executed by a processor, the detection method of the above-mentioned dangerous driving behavior is implemented. Optionally, the computer-readable storage medium may include: Read-Only Memory (ROM), Random Access Memory (RAM), Solid State Drives (SSD), or optical discs, etc. Among them, the random access memory may include Resistance Random Access Memory (ReRAM) and Dynamic Random Access Memory (DRAM).

[0138] In an exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program, and the computer program is stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the detection method of the above-mentioned dangerous driving behavior.

[0139] It should be understood that "a plurality of" mentioned herein refers to two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after. In addition, the step numbers described herein only exemplarily show a possible execution sequence between steps. In some other embodiments, the above steps may not be executed in the order of the numbers. For example, two steps with different numbers are executed simultaneously, or two steps with different numbers are executed in the reverse order of the illustration. The embodiments of the present application do not limit this.

[0140] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting dangerous driving behavior, characterized in that: The method comprises: Acquire multimodal data of a target vehicle, wherein the multimodal data includes data of at least two modalities; Predicting the dangerous driving behavior according to the multimodal data to obtain a prediction result, wherein the prediction result is used to indicate the behavior type of the predicted dangerous driving behavior; Based on the prediction result, a warning action is performed, where the warning action is a prompt action for the dangerous driving behavior.

2. The method according to claim 1, characterized in that The multimodal data includes at least one of the following: camera images, radar data, and vehicle speed; The step of predicting dangerous driving behavior based on the multimodal data to obtain a prediction result includes: Performing feature extraction on the camera image to obtain image features of the camera image; Fusing the camera image with data of other modes in the multimodal data to obtain fused multimodal data; Based on the fused multimodal data, the dangerous driving behavior is predicted to obtain the prediction result.

3. The method according to claim 2, characterized in that The step of predicting the dangerous driving behavior based on the fused multimodal data to obtain the prediction result includes: The fused multimodal data is input into a behavior prediction model, and the behavior prediction model outputs the prediction result based on a determination condition of at least one behavior type, wherein the prediction result includes a prediction probability of at least one behavior type, and the behavior prediction model is used to predict the dangerous driving behavior.

4. The method according to claim 3, characterized in that: The behavior type includes at least one of the following: colliding with a crowd, breaking through a checkpoint, speeding, and escaping an accident; In the case where the behavior type is the collision with a crowd, the behavior determination conditions include: the number of pedestrians is greater than a first threshold, the distance between the target vehicle and the crowd is greater than a braking distance threshold, and no braking signal is received, the number of pedestrians is obtained by performing pedestrian recognition on the camera image, the distance between the target vehicle and the crowd is determined based on the radar data, the braking distance threshold is determined based on the vehicle speed, and the braking signal is used to indicate that the target vehicle is in a braking state; In the case where the behavior type is the checkpoint-breaking behavior, the behavior determination condition includes: the checkpoint is in a closed state, the distance between the target vehicle and the checkpoint is less than a third threshold, and the vehicle speed is greater than a fourth threshold, the checkpoint is obtained by performing target recognition on the camera image, and the distance between the target vehicle and the checkpoint is determined based on the radar data; In the case where the behavior type is speeding, the behavior determination condition includes: the vehicle speed is greater than a road speed limit threshold, and the road speed limit threshold is determined based on the type of road on which the target vehicle is located; In the case where the behavior type is the accident escape, the behavior judgment conditions include: the existence of vehicle abnormalities, the detection of the vehicle speed being greater than a fifth threshold after the collision impact, and the vehicle abnormalities including at least one of the following: the detection of scattered objects, abnormal positions of other vehicles, and abnormalities in the body of the target vehicle.

5. The method according to claim 1, characterized in that The execution of the alarm action includes at least one of the following: Recording behavior information of the dangerous driving behavior, wherein the behavior information is information related to the dangerous driving behavior and the target vehicle; Reporting behavioral information of the dangerous driving behavior; Controlling the lights of the target vehicle to flash; The target vehicle is controlled to emit a sound signal, where the sound signal is used to indicate the presence of the dangerous driving behavior.

6. The method according to claim 1, characterized in that The method further comprises: According to the multimodal data, the target vehicle is detected to obtain a self-detection abnormality result, wherein the self-detection abnormality result is used to indicate whether the target vehicle is abnormal; When the target vehicle is in an unstarted state and the abnormal self-test result indicates that the target vehicle is abnormal, displaying fault information, wherein the fault information is used to indicate the abnormal condition of the target vehicle; When the target vehicle is in a start-up state and the abnormal self-test result indicates that the target vehicle is abnormal, the target vehicle is braked.

7. A device for detecting dangerous driving behavior, characterized in that: The device comprises: An acquisition module, used to acquire multimodal data of a target vehicle, wherein the multimodal data includes data of at least two modalities; A prediction module, used to predict the dangerous driving behavior according to the multimodal data, and obtain a prediction result, wherein the prediction result is used to indicate the behavior type of the predicted dangerous driving behavior; The warning module is used to execute a warning action based on the prediction result, and the warning action is a prompt action for the dangerous driving behavior.

8. A terminal device, characterized in that: The terminal device comprises a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to be executed by a processor to implement the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises a computer program, which is loaded and executed by a processor to implement the method according to any one of claims 1 to 6.

Images