Driver behavior anomaly detection method and device
By obtaining and analyzing driver's behavior, attention and vehicle status information, combining preset behavior and weight coefficients, identifying and handling driver behavior abnormalities, the problem of insufficient detection accuracy in the existing technology is solved and the vehicle driving safety is improved sex.
Patent Information
- Application Number
- CN202310559584.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-18
- Publication Date
- 2024-11-22
AI Technical Summary
When detecting abnormal behavior of drivers, the accuracy and robustness of the prior art are insufficient, making it difficult to effectively identify the driver in a road rage state, resulting in the impact of vehicle driving safety.
By obtaining driver's behavior information, attention information and vehicle status information, combining preset behavior and weight coefficients, identifying the level of driver's behavior abnormality, and performing corresponding control operations to improve the accuracy and robustness of the detection results. sex.
It improves the accuracy and robustness of driver behavior abnormality detection, reduces the false detection rate, and ensures the safety performance of vehicle driving.
Smart Images

Figure CN119007164A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of terminal technology, and in particular to a method and device for detecting abnormal driver behavior. Background Art
[0002] "Road rage" refers to aggressive or angry behavior by drivers of cars or other motor vehicles. Such behavior may include: vulgar gestures, verbal insults, intentionally driving in an unsafe or unsafe manner, or making threats. If electronic devices can accurately identify abnormal driving behaviors such as road rage and give warnings, or even restrict the driver's driving behavior, it can effectively improve the safety of vehicle driving.
[0003] One way to detect road rage is to determine whether the driver is in a road rage state by detecting the driver's behavior in operating the vehicle. For example, if the driver honks the horn rapidly, ignites the double flashers, or brakes frequently, the driver is determined to be in a road rage state. However, the operation of the vehicle is only part of the manifestation of road rage. Judging road rage by these features is not robust. For example, the driver's double flashes and emergency honking of the horn are likely to remind the vehicle in front of the main safety, and do not contain anger, but are judged as road rage.
[0004] Another way to detect road rage is to determine whether the driver is in a road rage state by detecting the driver's emotions. For example, a camera is used to collect a photo of the driver's face, extract the features of the photo, and input the features into an emotion recognition model. If the anger classification probability output by the model reaches a certain threshold, it is determined that the driver is in a road rage state. However, the accuracy of facial emotion recognition based on visual models is poor. For example, some people with an angry face may always be identified as road rage.
[0005] How to improve the accuracy of detecting unsafe driving behaviors such as road rage to improve the safety performance of vehicle driving is an urgent problem to be solved in this application. Summary of the invention
[0006] The embodiments of the present application provide a method and device for detecting abnormal driver behavior, which can improve the accuracy of detecting unsafe driving behaviors such as road rage, thereby improving the safety performance of vehicle driving.
[0007] In a first aspect, a method for detecting abnormal driver behavior is provided, which can be applied to a vehicle or a chip of the vehicle or a cloud device of the vehicle. Taking the method applied to a first vehicle as an example, the method includes: obtaining behavior information of a first object, behavior information of a driver of the first vehicle, and attention information of the driver; wherein the behavior information of the first object includes first information and position information of the first object, the first information is used to indicate a first behavior performed by the first object, the behavior information of the driver includes second information, the second information is used to indicate a second behavior performed by the driver, and the attention information of the driver is used to determine the object that the driver pays attention to; determining that the driver's behavior is abnormal according to the behavior information of the first object, the behavior information of the driver, and the attention information of the driver.
[0008] In the embodiment of the present application, when the first object performs a first behavior, the driver of the first vehicle pays attention to the first object (or the driver pays attention to the first object performing the first behavior), and the driver of the first vehicle performs a second behavior, it is determined that the driver's behavior of the first vehicle is abnormal. When judging that the driver's behavior of the first vehicle is abnormal, this scheme not only identifies the driver's behavior of the first vehicle, but also identifies the triggering cause that causes the driver of the first vehicle to perform the second behavior (that is, the first object performs the first behavior), and identifies the driver's attention to the triggering cause. The judgment logic is close to the scene of abnormal driver behavior in daily life (such as the driver's road rage is generally triggered by a reason and the driver generally quickly develops road rage after paying attention to the triggering cause), so it can improve the accuracy and robustness of the abnormal driver behavior detection results and reduce the false detection rate.
[0009] In one possible design, the first vehicle may obtain behavior information of the first object, behavior information of the driver of the first vehicle, and attention information of the driver through one or more sensors.
[0010] Exemplarily, the first vehicle may detect the behavior information of the first object through an out-of-cabin sensor, where the out-of-cabin sensor is deployed outside the cabin of the first vehicle, and the out-of-cabin sensor includes one or more of a radar, a camera around the vehicle, or a microphone of a rear vehicle.
[0011] Exemplarily, the first vehicle may capture an image of the driver through a camera monitor system (CMS) and a driver monitor system (DMS); perform feature analysis on the image to obtain the driver's attention information.
[0012] Exemplarily, the first vehicle can detect behavioral information of the driver of the first vehicle through in-cabin sensors, and the in-cabin sensors are deployed in the cabin of the first vehicle; the in-cabin sensors include one or more of DMS, CMS, microphone, functional components for controlling the throttle, functional components for controlling the brakes, functional components for controlling the horn, functional components for controlling the headlights, and a steering wheel.
[0013] It can be understood that the above sensors are only examples and are not limited thereto.
[0014] In one possible design, the first vehicle determines that the driver's behavior is abnormal based on the behavior information of the first object, the driver's behavior information and the driver's attention information, which may include: determining that the first behavior belongs to a first preset behavior based on the first information; and determining that the driver is paying attention to the first object based on the position information of the first object and the driver's attention information; and determining that the second behavior belongs to a second preset behavior based on the second information.
[0015] In other words, when the first object performs a first preset behavior, the driver of the first vehicle pays attention to the first object and the driver of the first vehicle performs a second preset behavior, determining that the driver behavior of the first vehicle is abnormal helps to improve the accuracy and robustness of the abnormal driver behavior detection results.
[0016] In one possible design, the driver's attention information includes the driver's attention direction; when the first vehicle determines that the driver is paying attention to the first object based on the behavior information of the first object and the driver's attention information, the first vehicle determines that the first object is located in the driver's attention direction based on the position information of the first object and the driver's attention direction.
[0017] The focus direction may include one or more of the driver's line of sight, face orientation, finger direction, or arm direction, and this application does not impose any specific limitation.
[0018] Optionally, the first vehicle can determine, based on the position information of the first object and the driver's attention direction, that the first object is located in the driver's attention direction when the angle between the direction of the first object relative to the first vehicle and the driver's attention direction is within a preset angle range, and further determine that the driver is paying attention to the first object.
[0019] In this design, the first vehicle determines that the driver is paying attention to the first object based on the spatial position relationship between the first object and the first vehicle and the driver's attention direction, which helps to improve the accuracy and robustness of the driver behavior abnormality detection results.
[0020] In a possible design, the behavior information of the first object may further include the occurrence time of the first behavior, and the driver's attention information may further include the time corresponding to the attention direction. The first vehicle determines that the driver is paying attention to the first object based on the behavior information of the first object and the driver's attention information, and further determines that the first object is located in the driver's attention direction within a first time period after the first behavior occurs based on the occurrence time of the first behavior and the time corresponding to the attention direction.
[0021] In this design, when the first vehicle determines whether the driver of the first vehicle is paying attention to the first object, in addition to the spatial position relationship between the driver's attention information and the first object, it also determines the time relationship between the driver's attention information and the first behavior performed by the first object. Only when the driver quickly focuses his attention on the first object after the first behavior occurs can it be determined that the driver is paying attention to the first object, which helps to improve the accuracy and robustness of the abnormal driver behavior detection results.
[0022] In one possible design, the driver's behavior information may also include the time when the second behavior occurs; when the first vehicle determines that the driver's behavior is abnormal based on the behavior information of the first object, the driver's behavior information and the driver's attention information, it also determines the time when the driver pays attention to the first object; based on the time when the driver pays attention to the first object and the time when the second behavior occurs, it is determined that the driver performs the second behavior within a second time period after paying attention to the first object.
[0023] In this design, when the driver pays attention to the first object within a short period of time and performs the second behavior within a short period of time after the first object performs the first behavior, the driver's behavior is determined to be abnormal, which helps to further improve the accuracy and robustness of the driver's behavior abnormality detection results.
[0024] In a possible design, the first vehicle may also calculate the level of abnormal driver behavior based on the correspondence between the preset behavior and the weight coefficient; and perform a control operation corresponding to the level of abnormal driver behavior based on the correspondence between the level of abnormal driver behavior, the preset level and the control operation. The control operation may include, but is not limited to, voice reminders, limiting vehicle speed, etc.
[0025] Exemplarily, the behaviors in the correspondence between the preset behaviors and the weight coefficients include honking the horn, accelerating suddenly, and frequently turning on the hazard lights, where the weight coefficient corresponding to rapid acceleration ≥ the weight coefficient corresponding to honking the horn ≥ the weight coefficient corresponding to frequently turning on the hazard lights. The behaviors in the correspondence between the preset behaviors and the weight coefficients may also include sudden braking and sudden turning, where the weight coefficient corresponding to sudden braking ≥ the weight coefficient corresponding to honking the horn, and the weight coefficient corresponding to sudden turning ≤ the weight coefficient corresponding to frequently turning on the hazard lights.
[0026] In this design, the first vehicle can determine the level of abnormal driver behavior and perform control operations corresponding to the level, which can improve the safety performance of the vehicle.
[0027] In one possible design, the first vehicle can use the behavior information of the first object, the behavior information of the driver, and the status information of the first vehicle as feature data, input them into a trained model, run the model, and obtain output information, wherein the output information includes a behavior judgment result, and the behavior judgment result is used to indicate that the driver's behavior is abnormal; the first vehicle determines that the driver's behavior is abnormal based on the behavior judgment result.
[0028] Exemplarily, the model may include a first-level feature extraction module, a splicing module, a second-level feature extraction module and a classification module; the model is run to obtain output information, including: based on the first-level feature extraction module, feature extraction is performed on each of the behavior information of the first object, the driver's behavior information and the status information of the first vehicle to obtain multiple one-dimensional feature vectors; based on the splicing module, the multiple one-dimensional feature vectors are spliced into a multi-dimensional feature vector to obtain a first feature vector; based on the second-level feature extraction module, the dimension of the first feature vector is expanded to obtain a second feature vector, and the dimension of the second feature vector is higher than the dimension of the first feature vector; based on the classification module, the second feature is predicted to obtain output information, and the output information includes a behavior recognition result, and the behavior recognition result is used to indicate abnormal driver behavior.
[0029] In this design, the first vehicle uses a model to identify abnormal driver behavior, which can improve the efficiency of identifying abnormal driver behavior. In addition, it helps the vehicle to take timely countermeasures when the driver's behavior is abnormal, further improving the vehicle's safety performance.
[0030] In one possible design, the output information also includes a level identification result, which is used to indicate the level of abnormal driver behavior. The first vehicle can also perform a control operation corresponding to the level of abnormal driver behavior based on the correspondence between the level of abnormal driver behavior, a preset level and the control operation.
[0031] In this way, the first vehicle can directly obtain the level of abnormal driver behavior based on the model output, and then take corresponding measures in a timely manner (such as executing control operations corresponding to the level of abnormal driver behavior), which helps to improve vehicle safety performance.
[0032] In a possible design, the first vehicle can input a sample data set into an initial model for training to obtain a trained model; wherein the sample data set includes a number of labeled sample data, the sample data includes behavior information of the sample object, behavior information of the driver of the sample vehicle, and attention information of the driver of the sample vehicle, and the label of the sample data is used to indicate that the driver behavior of the sample vehicle is abnormal and / or the level of abnormal behavior of the driver of the sample vehicle. Optionally, the sample data may also include sample data whose sample label indicates that the driver behavior of the vehicle is normal.
[0033] It can be understood that in actual applications, after the model training is completed by other devices, the trained model is deployed to the first vehicle.
[0034] In a second aspect, a device for detecting abnormal driver behavior is provided. The device may be a vehicle or a chip of the vehicle or a cloud device of the vehicle, etc. The device includes a module or unit or technical means for executing the method described in the first aspect or any possible design of the first aspect.
[0035] Exemplarily, the device may include:
[0036] an acquisition module, configured to acquire behavior information of a first object, behavior information of a driver of a first vehicle, and attention information of the driver; wherein the behavior information of the first object includes first information and position information of the first object, the first information is used to indicate a first behavior performed by the first object, the behavior information of the driver includes second information, the second information is used to indicate a second behavior performed by the driver, and the attention information of the driver is used to determine an object that the driver is paying attention to;
[0037] The processing module is used to determine whether the driver's behavior is abnormal based on the behavior information of the first object, the behavior information of the driver, and the attention information of the driver.
[0038] In one possible design, the acquisition module is specifically used to: detect the behavior information of the first object through an out-of-cabin sensor, the out-of-cabin sensor is deployed outside the cabin of the first vehicle, and the out-of-cabin sensor includes one or more of a radar, a camera around the vehicle, or a microphone of a rear vehicle; collect the image of the driver through a DMS or a CMS; perform feature analysis on the image to obtain the driver's attention information; detect the behavior information of the driver of the first vehicle through an in-cabin sensor, the in-cabin sensor is deployed in the cabin of the first vehicle; the in-cabin sensor includes one or more of a DMS, a CMS, a microphone, a functional component for controlling the throttle, a functional component for controlling the brake, a functional component for controlling the horn, a functional component for controlling the headlights, and a steering wheel.
[0039] In one possible design, the processing module is specifically used to: determine that the first behavior belongs to the first preset behavior based on the first information; and determine that the driver is paying attention to the first object based on the position information of the first object and the driver's attention information; and determine that the second behavior belongs to the second preset behavior based on the second information.
[0040] In one possible design, the driver's attention information includes the driver's attention direction; when the processing module determines that the driver is paying attention to the first object based on the behavior information of the first object and the driver's attention information, it is specifically used to: determine that the first object is located in the driver's attention direction based on the position information of the first object and the driver's attention direction.
[0041] In a possible design, the attention direction includes one or more of the driver's sight direction, face orientation, finger direction, or arm direction.
[0042] In one possible design, the behavior information of the first object also includes the occurrence time of the first behavior, and the driver's attention information also includes the time corresponding to the attention direction; when the processing module determines that the driver is paying attention to the first object based on the behavior information of the first object and the driver's attention information, it is also used to: determine, based on the occurrence time of the first behavior and the time corresponding to the attention direction, that the first object is located in the driver's attention direction within a first time period after the first behavior occurs.
[0043] In one possible design, the driver's behavior information also includes the time when the second behavior occurs; when the processing module determines that the driver's behavior is abnormal based on the behavior information of the first object, the driver's behavior information and the driver's attention information, it is also used to: determine the time when the driver pays attention to the first object; determine that the driver performs the second behavior within a second time period after paying attention to the first object based on the time when the driver pays attention to the first object and the time when the second behavior occurs.
[0044] In one possible design, the processing module is also used to: calculate the level of abnormal driver behavior based on the correspondence between preset behavior and weight coefficient; and execute control operations corresponding to the level of abnormal driver behavior based on the correspondence between the level of abnormal driver behavior, the preset level and the control operation.
[0045] In one possible design, the behaviors in the correspondence relationship between the preset behaviors and the weight coefficients include honking the horn, sudden acceleration, and frequent hazard lights, where the weight coefficient corresponding to sudden acceleration ≥ the weight coefficient corresponding to honking the horn ≥ the weight coefficient corresponding to frequent hazard lights.
[0046] In one possible design, the behaviors in the correspondence relationship between the preset behaviors and the weight coefficients also include sudden braking and sudden turning, and the weight coefficient corresponding to sudden braking is ≥ the weight coefficient corresponding to honking the horn, and the weight coefficient corresponding to sudden turning is ≤ the weight coefficient corresponding to frequent hazard flashing.
[0047] In one possible design, the processing module is specifically used to: input the behavior information of the first object, the behavior information of the driver, and the state information of the first vehicle into a trained model, run the model, and obtain output information, wherein the output information includes a behavior judgment result, and the behavior judgment result is used to indicate that the driver's behavior is abnormal; determine that the driver's behavior is abnormal based on the behavior judgment result.
[0048] In one possible design, the model includes a first-level feature extraction module, a splicing module, a second-level feature extraction module and a classification module; when the processing module runs the model, it is specifically used to: based on the first-level feature extraction module, perform feature extraction on each of the behavior information of the first object, the driver's behavior information and the state information of the first vehicle, to obtain multiple one-dimensional feature vectors; based on the splicing module, splice the multiple one-dimensional feature vectors into a multi-dimensional feature vector to obtain a first feature vector; based on the second-level feature extraction module, perform dimension expansion on the first feature vector to obtain a second feature vector, and the dimension of the second feature vector is higher than the dimension of the first feature vector; based on the classification module, predict the second feature to obtain output information, and the output information includes a behavior recognition result, and the behavior recognition result is used to indicate abnormal driver behavior.
[0049] In one possible design, the output information also includes a level identification result, which is used to indicate the level of abnormal driver behavior; the processing module is also used to: execute control operations corresponding to the level of abnormal driver behavior based on the correspondence between the level of abnormal driver behavior, a preset level and the control operation.
[0050] In one possible design, the processing module is also used to: input the sample data set into the initial model for training to obtain a trained model; wherein the sample data set includes a number of labeled sample data, the sample data includes behavior information of the sample object, behavior information of the driver of the sample vehicle, and attention information of the driver of the sample vehicle, and the label of the sample data is used to indicate abnormal behavior of the driver of the sample vehicle and / or the level of abnormal behavior of the driver of the sample vehicle.
[0051] In a third aspect, a device for detecting abnormal driver behavior is provided, comprising at least one processor and at least one memory; the memory is used to store computer-executable instructions; the processor is used to execute the computer-executable instructions stored in the memory, so that the device performs the method described in the first aspect or any possible design of the first aspect.
[0052] In a fourth aspect, a vehicle is provided, comprising: at least one sensor, configured to obtain behavior information of a first object, behavior information of a driver of a first vehicle, and attention information of the driver; wherein the behavior information of the first object comprises first information and position information of the first object, the first information is used to indicate a first behavior performed by the first object, the behavior information of the driver comprises second information, the second information is used to indicate a second behavior performed by the driver, and the driver attention information is used to determine an object that the driver is paying attention to;
[0053] At least one processor is used to determine whether the driver's behavior is abnormal based on the behavior information of the first object, the behavior information of the driver, and the attention information of the driver.
[0054] The specific functional implementation of the sensor and the processor can refer to the specific implementation of the corresponding design in the first aspect.
[0055] In a fifth aspect, a terminal device is provided, which may include the device described in the second aspect or any possible design of the second aspect, or include the device described in the third aspect.
[0056] In a sixth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store instructions, and when the instructions are executed, the method described in the first aspect or any possible design of the first aspect is implemented.
[0057] In a seventh aspect, a computer program product is provided, wherein instructions are stored in the computer program product, and when the computer program product is run on a computer, the computer executes the method described in the first aspect or any possible design of the first aspect.
[0058] For the technical effects of the second to seventh aspects mentioned above, please refer to the technical effects that can be achieved by the corresponding design in the first aspect mentioned above, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] Figure 1 A specific scenario diagram is provided for the embodiment of the present application;
[0060] Figure 2 A schematic diagram of a vehicle architecture provided in an embodiment of the present application;
[0061] Figure 3 A flowchart of a method for detecting abnormal driver behavior provided in an embodiment of the present application;
[0062] Figure 4A to Figure 4D A schematic diagram of the driver's attention direction;
[0063] Figure 5A This is an example diagram of the deployment position of the in-cabin camera;
[0064] Figure 5B This is a schematic diagram of the shooting angle of the camera in the cabin;
[0065] Figure 5C , Figure 5D This is an example of an image taken by the camera inside the cabin;
[0066] Figure 6 A schematic diagram of a driving scenario;
[0067] Figure 7 A schematic diagram of a logic for determining abnormal driver behavior;
[0068] Figure 8 A schematic diagram of a possible model is shown below;
[0069] Fig. 9 A schematic diagram of a driver abnormal behavior detection device provided in an embodiment of the present application;
[0070] Fig.10 A schematic diagram of another device for detecting abnormal driver behavior provided in an embodiment of the present application;
[0071] Fig.11 A schematic diagram of another device for detecting abnormal driver behavior provided in an embodiment of the present application. DETAILED DESCRIPTION
[0072] In the description of this application, "plurality" refers to two or more than two. "At least one" refers to one or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. Words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order.
[0073] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.
[0074] The embodiments of the present application can be applied to driving scenarios, such as autonomous driving, assisted driving, or manual driving scenarios.
[0075] See also Figure 1, a specific scene diagram is provided for the embodiment of the present application, in which the driver of a first vehicle drives the first vehicle on a road. In addition to the first vehicle, the scene may also include other vehicles, such as a second vehicle traveling in the same direction as the first vehicle, a third vehicle traveling in the opposite direction of the first vehicle, etc. In addition to motor vehicles, the scene may also include other things, such as non-motor vehicles ( Figure 1 Taking bicycles as an example, it is not limited to this), pedestrians ( Figure 1 Only one pedestrian is shown in the figure, but it is not limited to this), road facilities (such as traffic lights, etc.) ( Figure 1 It is understood that Figure 1 This is just an example, and the actual scenario is not limited to this.
[0076] The technical solution provided in the embodiment of the present application can be applied to any vehicle. Figure 1 Taking the first vehicle shown in the figure as an example, the first vehicle can be called "self-vehicle", and other vehicles other than the first vehicle (such as the second vehicle, the third vehicle, etc.) can be called "other vehicles". However, in actual situations, the same technical solution can also be applied to other vehicles such as the second vehicle or the third vehicle.
[0077] See also Figure 2 , is a schematic diagram of a vehicle architecture provided in an embodiment of the present application. Taking the vehicle as a first vehicle as an example, the architecture includes in-cabin sensors, out-of-cabin sensors, and a processing device.
[0078] 1) The out-of-cabin sensor is deployed outside the cabin to detect objects outside the first vehicle (such as other vehicles, pedestrians, non-motor vehicles, road facilities, etc.), generate corresponding sensor data, and then send it to the processing device, or process the sensor data before sending it to the processing device. Out-of-cabin sensors include but are not limited to radars (such as lidar, millimeter-wave radar, etc.), perimeter cameras, rear vehicle microphones, etc. It can be understood that the out-of-cabin sensors may include sensors arranged outside the cabin of the first vehicle (such as perimeter cameras, radars, rear vehicle microphones, etc.). The object outside the first vehicle may be a person, an object (such as other motor vehicles or non-motor vehicles), or an animal (such as a dog, cat, etc.), and this application does not impose any restrictions.
[0079] 2) The in-cabin sensor is used to detect components of the first vehicle and / or objects in the cabin of the first vehicle, generate corresponding sensor data, and then send it to the processing device, or process the sensor data before sending it to the processing device. It can be understood that the in-cabin sensor may include a sensor arranged inside the cabin of the first vehicle, such as an in-cabin camera (specifically, a camera monitoring system (Camera Monitor System, CMS), a driver monitoring system (Driver Monitor System, DMS), etc.), a microphone, a functional part for controlling the throttle, a functional part for controlling the brakes, a functional part for controlling the horn, a functional part for controlling the lights, a steering wheel, etc. The object in the cabin of the first vehicle may be a person (such as a driver, a passenger, etc.) or an object, and this application does not limit it.
[0080] Optionally, the in-cabin sensors may also be used to detect objects outside the first vehicle (such as other vehicles, pedestrians, non-motor vehicles, road facilities, etc.), such as a driving recorder that can detect environmental data outside the vehicle.
[0081] Optionally, extravehicular sensors may also be used to detect components of the first vehicle or objects in the cabin of the first vehicle (such as a driver), such as lights and wheels deployed outside the cabin, which may detect control operations performed by the driver on the first vehicle.
[0082] 3) The processing device is used to determine whether the driver's behavior is abnormal (such as road rage) based on the data sent by the external sensor and / or the internal sensor.
[0083] As an example, the processing device may include an external vehicle perception module, a self-vehicle perception module, and a decision module.
[0084] A. The off-vehicle sensing module is used to identify objects outside the first vehicle based on data sent by off-cabin sensors, such as identifying one or more of the following: changing lanes / cutting in line, sudden braking of the vehicle in front, horn urging of the vehicle behind, license plates, pedestrians, etc., and then outputting behavior information of objects outside the first vehicle (such as the first object).
[0085] Understandably, Figure 2 Only some recognition functions of the vehicle exterior sensing module are shown, but the actual functions are not limited to these.
[0086] B. The vehicle perception module is used to identify the first vehicle and / or objects on the first vehicle based on the data sent by the in-cabin sensor, such as identifying one or more of the following: sudden braking, honking the horn, sudden acceleration, frequent hazard lights, sudden turns, the driver's key words, the driver's speaking tone, the driver's gestures, the driver's face direction, the driver's eye direction, etc., and then output the behavior information of the driver of the first vehicle and the attention information of the driver of the first vehicle. It can be understood that Figure 2 Only part of the recognition function of the vehicle perception module is shown, and it is not limited to this.
[0087] C. The decision module is used to determine the object that the driver of the first vehicle pays attention to (such as the first object) according to the behavior information of the object outside the first vehicle output by the external vehicle perception module and the attention information of the driver of the first vehicle output by the own vehicle perception module, and to determine whether the driver's behavior of the first vehicle is abnormal (such as whether he is in a road rage state) in combination with the behavior information of the driver of the first vehicle output by the own vehicle perception module. Optional, such as Figure 2 As shown, the decision module further specifically includes an object locking module and an abnormality judgment module, wherein the object locking module is used to determine the object that the driver of the first vehicle is paying attention to, and the abnormality judgment module is used to judge whether the driver behavior of the first vehicle is abnormal.
[0088] In specific implementation, the processing device can be implemented by one processor or by multiple processors. When there are multiple processors, the multiple processors can be all set outside the cockpit of the first vehicle, or all set inside the cockpit of the first vehicle, or partly set outside the cockpit of the first vehicle and partly set outside the cockpit of the first vehicle, and this application does not limit it. Optionally, the processor can also be deployed in the cloud. The external vehicle perception module, the self-vehicle perception module and the decision-making module can be integrated in the same processor or in different processors, and this application does not limit it.
[0089] Understandably, Figure 2 The architecture shown is only an example and is not limited to this.
[0090] See also Figure 3 , is a flowchart of a method for detecting abnormal driver behavior provided in an embodiment of the present application, wherein the method is applied to Figure 2 Taking the first vehicle shown as an example, the method includes:
[0091] S301, obtaining behavior information of a first object, behavior information of a driver of a first vehicle, and attention information of the driver.
[0092] Among them, the behavior information of the first object includes first information, and the first information is used to indicate the behavior performed by the first object. For the convenience of description, the behavior performed by the first object is referred to as the first behavior in this article. It can be understood that the first object can be a person or an object, and this application does not limit it. For example, if the first object is another car, the first behavior is the driving behavior performed by the other car, such as cutting in line, sudden braking, honking the horn, sudden acceleration, frequent hazard lights, etc. For example, if the first object is a pedestrian, the first behavior can be an action performed by the pedestrian, such as crossing the road, approaching the first vehicle, etc. For example, if the first object is the driver of another car, the first behavior can be the gestures and language of the driver of the other car.
[0093] Optionally, the behavior information of the first object may also include the position information of the first object. In specific implementation, the position information of the first object may be absolute position information, such as the longitude and latitude of the first object or the coordinates of the first object in the world coordinate system, etc. The position information of the first object may also be relative position information, such as the coordinates of the first object in the coordinate system of the first vehicle, the direction and / or distance of the first object relative to the first vehicle, etc., which is not limited in this application. For example, in Figure 1 In the scenario shown, the position information of the second vehicle may be the direction of the second vehicle relative to the first vehicle. For example, the rear of the first vehicle is facing south and the front of the vehicle is facing north. The position information of the second vehicle is 30° to 45° north-east, indicating that the second vehicle is located within 30° to 45° north-east of the first vehicle.
[0094] Optionally, the behavior information of the first object may further include the occurrence time of the first behavior, that is, the time when the first object performs the first behavior. For example, the first object performs the first behavior at 19:20:10 on May 9, 2023, and the behavior information of the first object may include 2023-05-09 19:20:10.
[0095] In some embodiments, the first vehicle may detect the behavior information of the first object through an off-board sensor, where the off-board sensor is a sensor deployed outside the cabin of the first vehicle. For example, the off-board sensor may include one or more of a radar, a camera, or a microphone disposed around the vehicle.
[0096] In a specific implementation, the outboard sensor collects the sensor data of the first object, for example, the radar collects the point cloud data corresponding to the first object, or the camera collects the image data of the first object, or the microphone collects the voice data of the first object, etc. After the outboard sensor obtains the sensor data, it can process the sensor data, for example, analyze the sensor data to obtain the behavior information of the first object; or, after obtaining the sensor data, the outboard sensor sends the sensor data to the processing device of the first vehicle, and the processing device processes the sensor data, and the processing device obtains the behavior information of the first object according to the analysis of the sensor data.
[0097] In the following, taking the first object as another vehicle (i.e., another vehicle other than the first vehicle) as an example, several possible ways of detecting the behavior of the first object are described:
[0098] 1) Cutting in or jumping in line: The first vehicle identifies the position and distance relationship between the other vehicle and the first vehicle (i.e., the first vehicle) based on a camera or radar, and determines whether the other vehicle has cut in or jumped in line based on the position and distance relationship.
[0099] Exemplarily, LaneIndex_0 represents the identification of the lane where the own vehicle is located, and LaneIndex_x represents the identification of the lane where other vehicles (a maximum number can be set, for example, up to 5 other vehicles can be tracked) located around the own vehicle are located; when the identification of the lane where a certain other vehicle is located becomes laneIndex_0, and the distance between the vehicle (rear edge) and the own vehicle is less than the set distance threshold (for example, 10m), and the lateral speed of the vehicle when changing lanes is greater than the set speed threshold (for example, 0.5m / s), it can be determined that the vehicle has the behavior of cutting in or cutting in line, and the identification of the vehicle and the behavior information of the vehicle (cutting in or cutting in line) are recorded.
[0100] During the specific implementation process, when the first vehicle is unable to identify the model or license plate of the other vehicle (such as being unable to identify the model or license plate of the other vehicle at a long distance), it can identify the other vehicle through lane markings or preset parameters (such as numbers 1, 2, 3, etc.); when the license plate (or model) of the other vehicle can be identified, it can record the license plate (or model) of the other vehicle and identify the other vehicle through the license plate (or model).
[0101] In some embodiments, the processing device of the first vehicle may input the position distance relationship identified by the camera or radar into a pre-trained event classification model, and determine whether the other vehicle has cut in line or engaged in queue-jumping based on the output information of the event classification model.
[0102] 2) Emergency braking: The first vehicle identifies the position, distance relationship and relative speed of the other vehicle (such as the second vehicle) and the position, distance relationship and relative speed of the first vehicle based on a camera or radar, and determines that the other vehicle should brake urgently based on the position, distance relationship and relative speed relationship.
[0103] Exemplarily, the processing device of the first vehicle can record the relative distance and relative speed between the vehicle and the front vehicle, calculate it 10 times per second, and subtract the relative speed difference between the two vehicles from the speed difference of the vehicle (same time interval, such as 100ms) (calculate the acceleration). If it exceeds the set speed threshold (such as a=-6m / s2), it can be determined that the front vehicle brakes suddenly, and the identification and behavior information of the vehicle will be recorded.
[0104] In some embodiments, the processing device of the first vehicle may input the position distance relationship and relative speed identified by the camera or radar into a pre-trained event classification model, and determine whether the other vehicle has sudden braking behavior based on the output information of the event classification model.
[0105] 3) Double flash of lights: The camera at the rear of the first vehicle identifies and tracks the position and brightness of the lights of the following vehicle, and compares the brightness of the lights between frames.
[0106] For example, the camera collects an image of the rear vehicle's headlights, and the processing device of the first vehicle detects the position of the rear vehicle's headlights based on the image, calculates the pixel value of the headlight position, and determines the on and off state of the headlights by comparing the pixel values of the headlight positions of different frame images. If the number of times the headlights are alternately on and off in a short period of time (for example, within 3 seconds) exceeds a threshold (for example, 3 times), it can be determined that the rear vehicle's headlights are flashing, and the vehicle's identification and behavior information are recorded. In addition, the position of the flashing vehicle relative to the vehicle itself can also be recorded.
[0107] In some embodiments, the processing device of the first vehicle may input the image of the other vehicle captured by the camera into a pre-trained event classification model, and determine whether the other vehicle has double flashing headlights based on the output information of the event classification model.
[0108] 4) Horn urging: The microphone collects voice data in the environment, and the processing device of the first vehicle inputs the collected voice data into a pre-trained recognition module, and identifies the horn sound of the rear vehicle according to the output of the module. When the frequency of the horn sound of the rear vehicle exceeds the preset frequency, it is determined that the rear vehicle has horn urging behavior, and the identification and behavior information of the vehicle are recorded.
[0109] It can be understood that the above first behaviors and detection methods are only examples and are not limited thereto.
[0110] The above are all about detecting the behavior information of the first object by using extravehicular sensors. In actual applications, the first vehicle can also detect the behavior information of the first object by using intravehicular sensors, such as a driving recorder. The first vehicle can collect the external environment image of the first vehicle based on the driving recorder, and then determine the behavior information of other vehicles or pedestrians based on the image collected by the driving recorder.
[0111] In addition, the first object may be a person or object outside the cabin of the first vehicle, or may be a person or object inside the cabin of the first vehicle, such as a passenger in the first vehicle.
[0112] It can be understood that the above takes the first object as an example. In actual implementation, the first vehicle can obtain behavior information of more objects. The method of obtaining behavior information of other objects can refer to the method of obtaining behavior information of the first object above, which will not be repeated here.
[0113] The driver's attention information is used to determine the object of the driver's attention, that is, the person or object that the driver's psychological activities are directed and focused on.
[0114] In a possible implementation, the driver's attention information may include the driver's attention direction. The driver's attention direction may be determined based on one or more of the driver's sight direction, face orientation, finger direction, or arm direction.
[0115] For example, the processing device of the first vehicle may determine the driver's sight direction, face orientation, finger direction, or arm direction as the driver's attention direction. Figure 4A In the example, the driver's face is facing the front of the first vehicle, and the attention direction is forward, while Figure 4B The driver's face is facing the right side of the first vehicle, then the focus direction is to the right. Figure 4A The driver's eyes are facing forward, that is, the sight is in front of the first vehicle, so the attention direction is forward, and Figure 4C The driver's eyes are facing right front, that is, his sight is to the right of the first vehicle, so the attention direction is to the right.
[0116] It is understandable that when there is a high-reflectivity object such as a mirror in the driver's attention direction or gaze area, it is also necessary to combine the position of the mirror to determine the driver's attention information. Figure 4D As shown, the driver is looking at the right rearview mirror. Although the driver's line of sight is towards the right front, and the apparent direction of attention is the right front, it can be seen from the principle of mirror reflection that the driver is actually paying attention to the right rear through the right rearview mirror, so the actual direction of attention is the right rear.
[0117] In a possible implementation, the driver's attention information may include the driver's attention area. The attention area may be determined based on the driver's line of sight, face orientation, finger direction, or arm direction. For example, Figure 4A In the example, the driver looks ahead of the first vehicle, the attention information may be a spatial area ahead of the first vehicle. Figure 4B or Figure 4C If the driver is looking at the right side of the first vehicle, the attention information may be a spatial area located to the right of the first vehicle.
[0118] In a possible implementation, the driver's attention information may also include the driver's attention direction and attention area.
[0119] In some embodiments, the first vehicle may detect the driver's attention information via an in-cabin sensor.
[0120] The in-cabin sensor is deployed in the cabin of the first vehicle, such as a camera set in the cabin, such as a CMS or a DMS. Exemplarily, the camera can capture an image of the driver, determine the driver's attention information based on the image, and send the driver's attention information to the processing device of the first vehicle; or the camera can capture an image of the driver, send the driver's image to the processing device of the first vehicle, and the processing device determines the driver's attention information based on the image.
[0121] See also Figure 5A, which is an example of the position deployment of the in-cabin camera. The camera is set at the upper left of the driving position in the cockpit. Figure 5B This is a schematic diagram of the camera's shooting angle, which can capture the driver's and co-driver's seats as well as the environment outside the window. Figure 5C It is the image of the driver captured by the camera when the driver's face is facing forward, and the image corresponds to the left side of the driver's face. Figure 5D It is the image of the driver captured by the camera when the driver's face is facing right, and the image corresponds to the back of the driver's head.
[0122] by Figure 5D As an example, the processing device of the first vehicle receives the image sent by the camera in the cabin (such as Figure 5C The image shown in FIG. 1 ) can be used to determine that the driver's attention direction is the right direction of the first vehicle.
[0123] Understandably, Figure 5A to Figure 5D These are just some examples and are not intended to be limiting.
[0124] Optionally, the attention information may also include attention time. For example, the time corresponding to the attention direction and / or the attention area, or the occurrence time of the attention direction and / or the attention area. Figure 5D In the example, if the attention direction is toward the right, then the attention time is the time when the driver turns his head toward the right.
[0125] It is understandable that, since it takes time for the camera to capture images and for the processing device to process images, there is a time difference between the actual occurrence time of the focus direction and / or focus area and the time when the camera captures the image or the time when the processing device determines the focus direction and / or focus area. In some embodiments, the time difference can be ignored, and the time when the camera captures the image or the time when the processing device determines the focus direction and / or focus area is directly used as the attention time; in some embodiments, the time when the camera captures the image or the time when the processing device determines the focus direction and / or focus area can be time-offset, and the offset time is used as the attention time.
[0126] For example, the image captured by the in-cabin camera at 19:20:05 on May 9, 2023 is Figure 5C The image shown, the processor of the first vehicle can determine Figure 5C The time corresponding to the direction of interest (forward) shown is 2023-05-09 19:20:05; the image captured by the in-cabin camera at 19:20:10 on May 9, 2023 is Figure 5D The image shown, the processor of the first vehicle can determine Figure 5D The time corresponding to the direction of attention shown (towards the right) is 2023-05-09 19:20:10.
[0127] It is understandable that the driver may have different focus directions and / or focus areas at different times. For example, the driver may have different focus directions and / or focus areas at 2023-05-09 19:20:05. Figure 5C The direction of attention shown (forward) is Figure 5D The attention direction shown is (towards the right). Accordingly, the processing device of the first vehicle can collect different attention information at different times.
[0128] The driver's behavior information includes second information, and the second information is used to indicate a second behavior performed by the driver. The second behavior can be an operation performed by the driver on the first vehicle (such as braking, honking, accelerating, turning on the hazard lights, turning the steering wheel, etc.), or a physical action performed by the driver while driving the vehicle, such as making facial expressions, speaking, gesturing, etc.
[0129] In some embodiments, the first vehicle can detect the driver's behavior information through an in-cabin sensor. The in-cabin sensor is deployed in the cockpit of the first vehicle; the in-cabin sensor includes, but is not limited to, one or more of a camera (such as a DMS, CMS), a microphone, a functional part for controlling the throttle (such as an accelerator pedal), a functional part for controlling the brake (such as a foot brake or a hand brake), a functional part for controlling the horn (such as a horn control in the middle of the steering wheel), a functional part for controlling the lights (such as a light control under the steering wheel, a double flash button, etc.), a steering wheel, etc.
[0130] Optionally, the driver's behavior information may also include the time when the second behavior occurred. In some embodiments, the time when the first vehicle detects the second behavior may be used as the time when the second behavior occurred. For example, if the processing device of the first vehicle detects that the driver suddenly stepped on the foot brake at 19:20:15 on May 9, 2023, it can be determined that the driver suddenly braked, and the time when the sudden brake occurred was 2023-05-09 19:20:15.
[0131] Here are some possible ways to detect the driver's behavior:
[0132] 1) Emergency braking: The first vehicle detection processing device measures the acceleration of the brake pedal (i.e., foot brake). When the acceleration exceeds a set threshold (e.g., a=-6m / s2), it is determined that an emergency braking behavior exists, and the behavior and the time when the behavior occurs are recorded.
[0133] 2) Pressing the horn suddenly: The processing device of the first vehicle detects the frequency of the driver pressing the horn (i.e., the frequency of the function component for controlling the horn being pressed) or the duration. When the frequency exceeds a set threshold (such as 2 times / s) or the duration exceeds a set time (such as 1s), it can be determined that the horn is pressed suddenly, and the behavior and the time when the behavior occurs are recorded.
[0134] 3) Rapid acceleration (i.e., stepping on the accelerator pedal hard): The processing device of the first vehicle detects the acceleration of the accelerator pedal. When the acceleration exceeds a set threshold (such as 50% / s), it is determined that there is rapid acceleration behavior, and the behavior and the time when the behavior occurs are recorded.
[0135] 4) Frequent double flashing: The processing device of the first vehicle counts the number of double flashes within a period of time. If the number of double flashes exceeds a threshold (such as more than 3 times within 10 seconds), it is determined that there is frequent double flashing, and the behavior and the time when the behavior occurs are recorded.
[0136] 5) Sharp turn (i.e., sudden turning of the steering wheel): The processing device of the first vehicle detects the angular acceleration of the steering wheel and determines whether there is a sharp turn behavior in combination with the vehicle speed. For example, when the vehicle speed reaches 40km / h, if the angular acceleration exceeds 500 degrees / s, it can be determined that there is a frequent flashing of the double flashes, and the behavior and the time when the behavior occurs are recorded.
[0137] 6) Keyword detection: The microphone of the first vehicle collects the driver's voice, and the processing device of the first vehicle inputs the voice into a pre-trained semantic recognition model library, and determines whether the driver has said a keyword based on the output of the model.
[0138] 7) Tone detection: The microphone of the first vehicle collects the driver's voice, and the processing device of the first vehicle inputs the voice into a pre-trained tone recognition model library, and determines whether the driver's tone is angry based on the output of the model.
[0139] It can be understood that the above second behaviors and detection methods thereof are merely examples and are not limited thereto.
[0140] It can be understood that the above is an example of detecting a driver's behavior (i.e., the second behavior). In actual implementation, multiple behaviors may occur during the driver's driving of the first vehicle, and the behaviors occurring at different times may be the same or different. For example, the driver braked suddenly at 19:20:15 on May 9, 2023, and the driver accelerated suddenly at 19:21:10 on May 9, 2023. The behavior information corresponding to each behavior performed by the driver and the method of obtaining it can all refer to the behavior information corresponding to the second behavior and the method of obtaining it above, which will not be repeated here.
[0141] S302: Determine whether the driver's behavior is abnormal based on the behavior information of the first object, the behavior information of the driver, and the attention information of the driver.
[0142] Specifically, when the behavior information of the first object, the behavior information of the driver, and the attention information of the driver meet preset conditions, the processing device of the first vehicle determines that the driver's behavior is abnormal.
[0143] In a possible design, the preset condition may include: the first behavior belongs to the first preset behavior, and the driver pays attention to the first object, and the second behavior belongs to the second preset behavior. Exemplarily, the processing device of the first vehicle determines that the driver's behavior is abnormal when it determines that the first behavior belongs to the first preset behavior according to the first information, and determines that the driver pays attention to the first object according to the position information of the first object and the driver's attention information, and determines that the second behavior belongs to the second preset behavior according to the second information.
[0144] In some embodiments, the first preset behavior may be one or more behaviors that may affect the first vehicle. For example, when the first object is another vehicle, the first preset behavior may include but is not limited to one or more of cutting in, cutting in line, sudden braking, honking the horn, sudden acceleration, frequent hazard lights, and sudden turns. For example, when the first object is a pedestrian, the first preset behavior may include but is not limited to one or more of crossing the road, approaching the first vehicle, or obstructing the first vehicle.
[0145] For example, Figure 1 Take the scenario shown as an example: when the first vehicle is moving forward, the second vehicle and the first vehicle are on the right side of the first vehicle and turning left, which may affect the normal forward movement of the first vehicle; or, when the first vehicle is moving forward, a pedestrian crosses the road in front of the first vehicle, which may affect the normal forward movement of the first vehicle.
[0146] In addition, because the first vehicle is controlled by the driver of the first vehicle, affecting the driver of the first vehicle will indirectly affect the first vehicle, so the first preset behavior can also be one or more behaviors that affect the driver of the first vehicle. For example, when the first object is the driver of another vehicle, the first preset behavior can include but is not limited to one or more of the driver of another vehicle performing a preset gesture, saying a keyword, and speaking in an angry tone to the driver of the first vehicle.
[0147] In some embodiments, when the driver's attention information and the position information of the first object satisfy a preset spatial position relationship, the processing device of the first vehicle determines that the driver is paying attention to the first object. Exemplarily, the driver's attention information includes the driver's attention direction and / or attention area, and when the first object is located in the attention direction and / or the first object is located in the attention area, the processing device of the first vehicle (for example) determines that the driver is paying attention to the first object. Exemplarily, when the angle between the direction of the first object relative to the first vehicle and the driver's attention direction is within a preset angle range (e.g., -15° to 15°), the processing device of the first vehicle can determine that the first object is located in the driver's attention direction.
[0148] For example, see Figure 6 The first object is a pedestrian, who approaches the first vehicle from the right side (first behavior), the driver's head is facing to the right, and the angle between the direction of the pedestrian relative to the first vehicle and the driver's attention direction is 5°, then the first vehicle can determine that the driver is paying attention to the pedestrian.
[0149] Optionally, when the processing device of the first vehicle determines that the driver is paying attention to the first object, the processing device of the first vehicle can lock the first object. For example, the processing device of the first vehicle locks the first object including: recording the attribute information of the object, the attribute information includes but is not limited to the type of the object (such as pedestrian / motor vehicle / non-motor vehicle, etc.), the identifier of the object (such as license plate number), the model of the object (such as vehicle model), the real-time angle range of the object relative to the first vehicle (0-360 degrees), the real-time distance of the object relative to the first vehicle, etc. Further optionally, after locking the first object for a preset time or after the first vehicle travels a preset distance, the lock on the first object is released, for example, the attribute information of the object is cleared.
[0150] In the embodiment of the present application, the second preset behavior includes but is not limited to one or more of sudden braking, honking the horn, sudden acceleration, frequent hazard flashing, sudden turning, saying keywords, angry tone, preset gestures, preset expressions, etc.
[0151] It can be understood that in the embodiment of the present application, the first vehicle can obtain different behavior information of the same object at different times, can obtain behavior information of different objects at different times, and can also obtain behavior information of multiple different objects at the same time; the driver can pay attention to different objects at different times, that is, the first vehicle can obtain different attention information of the driver at different times; the driver can perform different behaviors at different times, that is, the first vehicle can obtain multiple behaviors of the driver (such as the second behavior, the third behavior, etc.) within a period of time. The first vehicle can execute the above method for any behavior information of any object, any attention information of the driver, and any behavior information of the driver.
[0152] In the above scheme, when the first object performs a first behavior, the driver of the first vehicle pays attention to the first object (or the driver pays attention to the first object performing the first behavior), and the driver of the first vehicle performs a second behavior, it is determined that the driver's behavior of the first vehicle is abnormal. When judging that the driver's behavior of the first vehicle is abnormal, this scheme not only identifies the driver's behavior of the first vehicle, but also identifies the triggering cause that causes the driver of the first vehicle to perform the second behavior (that is, the first object performs the first behavior), and identifies the driver's attention to the triggering cause. The judgment logic is close to the scene of abnormal driver behavior in daily life (such as the driver's road rage is generally triggered by a reason and the driver generally quickly develops road rage after paying attention to the triggering cause). Therefore, the accuracy and robustness of the abnormal driver behavior detection results can be improved, and the false detection rate can be reduced.
[0153] In some embodiments, the first vehicle may determine whether the first behavior belongs to the first preset behavior, whether the driver pays attention to the first object, and whether the second behavior belongs to the second preset behavior in a set order. Figure 7 , taking the first object as the other car, the judgment of abnormal driver behavior is to judge whether the driver is road angry as an example: the first vehicle detects that the other car is abnormal (such as the other car performs one or more of the following behaviors: cutting in line, sudden braking, honking the horn, sudden acceleration, frequent double flashing, and sudden turning); the first vehicle obtains the location information of the other car, and judges whether the driver pays attention to the other car in combination with the driver's face direction; if so, the first vehicle locks the other car as the road rage object, and further identifies and tracks the license plate of the other car; the first vehicle judges whether the self-vehicle is abnormal (such as the self-vehicle has one or more behaviors such as sudden braking, honking the horn, sudden acceleration, frequent double flashing, and sudden turning, or the driver of the self-vehicle has one or more behaviors such as saying keywords, angry tone, preset gestures, etc.), if the self-vehicle is abnormal under the state of the other car being locked (that is, there is a triggering cause of road rage and the driver pays attention to the cause), it can be determined that the driver of the self-vehicle is road angry. In this way, computing resources can be saved and the detection efficiency of abnormal behavior can be improved.
[0154] In one possible design, when the first vehicle determines whether the driver of the first vehicle is paying attention to the first object, in addition to determining the spatial position relationship between the driver's attention information and the first object, it can also determine the temporal relationship between the driver's attention information and the first behavior performed by the first object. Exemplarily, the behavior information of the first object may also include the occurrence time of the first behavior, and the driver's attention information may also include the attention time (such as the time corresponding to the attention direction and / or the time corresponding to the attention area). When the first object is located in the attention direction and / or the attention area within a first time period after the first behavior occurs, the processing device of the first vehicle determines that the driver is paying attention to the first object. In this way, the accuracy of the first vehicle in determining the object that the driver is paying attention to can be improved, which helps to further improve the accuracy and robustness of the abnormal driver behavior detection results.
[0155] In a possible design, when the first vehicle determines that the driver's behavior is abnormal, it can also determine the relationship between the time when the second behavior occurs and the time when the driver pays attention to the first object. Exemplarily, the processing device of the first vehicle can also determine the time when the driver pays attention to the first object, and the preset condition also includes that the second behavior occurs within a second time period after the driver pays attention to the first object. For example, when the first behavior belongs to the first preset behavior, and the driver pays attention to the first object within the first time period after the first behavior occurs, and the driver performs the second behavior within the second time period after paying attention to the first object, and the second behavior belongs to the second preset behavior, it is determined that the driver's behavior is abnormal. In this way, the accuracy of the first vehicle in judging whether the driver performs the second behavior after paying attention to the first object and performing the first behavior can be improved, which helps to further improve the accuracy and robustness of the abnormal driver behavior detection results.
[0156] In a possible design, when the first vehicle determines that the driver's behavior is abnormal, it can also determine the time relationship between the driver's second behavior and the first behavior of the first object. Exemplarily, the driver's behavior information also includes the occurrence time of the second behavior, and the preset condition also includes that the second behavior occurs within a third time period after the first behavior occurs. For example, when the first behavior belongs to the first preset behavior, and when the driver pays attention to the first object within the first time period after the first behavior occurs, and when the second behavior is performed within the third time period after the first behavior occurs, and the second behavior belongs to the second preset behavior, the processing device of the first vehicle determines that the driver's behavior is abnormal. In this way, the accuracy of the first vehicle's judgment that the second behavior is triggered by the first behavior can be improved, which helps to further improve the accuracy and robustness of the abnormal driver behavior detection results.
[0157] In a possible design, the first vehicle can also simultaneously combine multiple behaviors performed by the driver to determine whether the driver's behavior is abnormal. Exemplarily, when the first behavior performed by the first object belongs to the first preset behavior, and the driver pays attention to the first object within the first time period after the first behavior occurs, and the driver performs multiple behaviors (such as the second behavior, the third behavior, etc.) within the third time period after paying attention to the first object, and the multiple behaviors all belong to the second preset behavior, the driver's behavior is determined to be abnormal. For example, in real life, when a driver is road rage, he may simultaneously engage in multiple behaviors such as verbal insults, deliberately driving the vehicle in an unsafe or safety-threatening manner, etc. Therefore, combining multiple behaviors performed by the driver to determine the driver's behavior is abnormal can further improve the accuracy and robustness of the driver's behavior abnormality detection results.
[0158] In one possible design, after determining that the driver's behavior is abnormal, the first vehicle can also determine the degree, grade or level of the abnormal driver's behavior, for example, and then give the driver relevant reminders or restrict the driver's control operations on the first vehicle according to the degree, grade or level of the abnormal driver's behavior to ensure the safety of the first vehicle.
[0159] In some embodiments, weight coefficients for various behaviors may be predefined, and a correspondence between behaviors and weight coefficients may be established. The first vehicle may calculate the level of abnormal driver behavior based on the preset correspondence between behaviors and weight coefficients.
[0160] Optionally, the behaviors in the correspondence between the preset behaviors and the weight coefficients include honking the horn, accelerating suddenly, and frequently turning on the hazard lights, wherein the weight coefficient corresponding to rapid acceleration ≥ the weight coefficient corresponding to honking the horn ≥ the weight coefficient corresponding to frequently turning on the hazard lights. Further optionally, the behaviors in the correspondence between the preset behaviors and the weight coefficients include sudden braking and sudden turning, wherein the weight coefficient corresponding to sudden braking ≥ the weight coefficient corresponding to honking the horn, and the weight coefficient corresponding to sudden turning ≤ the weight coefficient corresponding to frequently turning on the hazard lights.
[0161] For example, Table 1 is an example of the corresponding relationship between behaviors and weight coefficients:
[0162] Table 1
[0163]
[0164] It can be understood that Table 1 is only an example and is not limited thereto. In addition, this article takes the example that a larger weight coefficient indicates a higher level of abnormal behavior, but is not limited thereto.
[0165] In some embodiments, the first vehicle may calculate the level of abnormal driver behavior at the granularity of a single behavior performed by the driver. For example, after the second vehicle has abnormal behavior, the driver of the first vehicle pays attention to the second vehicle and performs an emergency braking behavior, then it can be determined that the driver of the first vehicle has abnormal behavior and the level of abnormal behavior is a=0.3.
[0166] In some embodiments, the first vehicle can also count the level of abnormal driver behavior at the granularity of the object. Specifically, for each object, the weight coefficient of the abnormal behavior of the driver on the object within a period of time is accumulated, and the accumulated weight coefficient is used as the level of abnormal behavior performed by the driver on the object. Exemplarily, each time the driver performs an abnormal behavior (i.e., any of the second preset behaviors), the driver's face direction is detected to ensure that the driver is paying attention to the object that performs the first preset behavior. When the face attention direction and the angle between the object and the first vehicle remain within the preset angle range, it is determined that the abnormal behavior is a valid abnormal behavior, and the weight coefficient of the abnormal behavior is recorded on the object; all weight coefficients recorded on the object within 2 minutes are accumulated, and the accumulated result is used as the level of the driver's abnormal behavior on the object. For example, if the driver slams the accelerator 2 times and slams the horn 3 times within 2 minutes, the level is 2b+2c=1.
[0167] In some embodiments, the first vehicle may also record the abnormal driver behavior each time the abnormal driver behavior is determined, such as recording it as an abnormal event, and may also record the time of the abnormal driver behavior (such as the time when the abnormal driver behavior is determined). Optionally, the object that triggers the abnormal driver behavior, the specific behavior of the driver, etc. may also be recorded.
[0168] Furthermore, when determining the level of abnormal driver behavior, the first vehicle may accumulate multiple abnormal events triggered by the same object within a preset time range to determine the level of abnormal driver behavior.
[0169] For example, Table 2 is an example of an abnormal event list, in which multiple abnormal events are recorded:
[0170] Table 2
[0171]
[0172]
[0173] Taking the preset time range of 5 minutes as an example, in the example given in Table 2, the time interval between abnormal event 1 and abnormal event 3 is within 5 minutes, and abnormal event 1 and abnormal event 3 are triggered by the same object A, so abnormal event 1 and abnormal event 3 can be accumulated to determine the level of abnormal driver behavior. For example, according to the weight coefficient shown in Table 1, the cumulative level of abnormal event 1 and abnormal event 3 is a+b+c=0.3+0.2+0.3=0.8.
[0174] In some embodiments, after obtaining the level of abnormal driver behavior, the first vehicle may perform a control operation corresponding to the level of abnormal driver behavior according to the correspondence between the level of abnormal driver behavior, the preset level and the control operation. For example, Table 2 is an example of the correspondence between the level and the control operation:
[0175] Table 3
[0176]
[0177] For example, if the driver slams on the accelerator twice and honks the horn three times within 2 minutes, the level is 2b+2c=1, and the driver can be reminded by voice to pay attention to driving safety.
[0178] For example, if the driver brakes 4 times, steps on the accelerator 2 times, and honks the horn 3 times within 2 minutes, the level is 4a+2b+2c=2.2, and the vehicle speed can be limited and voice notification can be given.
[0179] It can be understood that Table 3 is only an example and is not limited thereto.
[0180] In some embodiments, the first vehicle may also use an artificial intelligence (AI) model to identify abnormal driver behavior. Exemplarily, after obtaining the behavior information of the first object, the behavior information of the driver, and the driver's attention information to determine that the driver's behavior is abnormal, the processing device of the first vehicle may input the behavior information of the first object, the behavior information of the driver, and the state information of the first vehicle as original feature data into the trained model, and run the model to obtain output information. The output information includes a behavior judgment result, which is used to indicate whether the driver's behavior is normal or abnormal. The processing device of the first vehicle can determine whether the driver's behavior is abnormal based on the behavior judgment result. Optionally, when the behavior recognition result indicates that the driver's behavior is abnormal, the output information may also include a level recognition result to indicate the level of abnormal driver behavior.
[0181] In specific implementation, the model may include a feature extraction module and a classification module. The feature extraction module is used to extract features from the original feature data and output feature vectors. The classification module is used to predict the feature vectors and obtain output information. Among them, the feature extraction module can be implemented based on the Transformer model, and the classification module can be implemented based on the Liner model.
[0182] In some embodiments, in order to improve the accuracy of classification, a multi-layer feature extraction module can be set to obtain feature vectors with more dimensions. Figure 8 , which is a schematic diagram of a possible model, including a first-level feature extraction (Transformer) module, a concatenation (Concact) module, a second-level feature extraction (Transformer) module and a classification (Liner) module.
[0183] 1) The first vehicle inputs the behavior information of the first object (such as other vehicle data), the behavior information of the driver and the state information of the first vehicle (such as own vehicle data) as original feature data into the first-level feature extraction module. The first-level feature extraction module performs feature extraction and encodes each information into a one-dimensional feature vector to obtain multiple one-dimensional feature vectors.
[0184] For example, the feature data input by the first-level feature extraction module include speech tone, sensitive words, time of changing lanes / cutting in line, acceleration of the preceding vehicle when braking suddenly, etc. The feature vector output by the first-level feature extraction module is a one-dimensional feature vector after encoding each feature data.
[0185] The calculation logic of the first-level feature extraction module:
[0186] 1. Calculate Q, K, V: Q = W q X,K=W k X,V=W v X; Q is query, K is key, and V is value; W q , W k , W v is the weight coefficient.
[0187] 2. Calculate the value of Self-Attention:
[0188] Scaling factor, softmax is a multi-classifier;
[0189] 3. Calculate the value of Multi-Head Attention:
[0190] Attention(Q,K,V)=Concact(head1,…,head s)W0;
[0191] Among them head i =Attention(Q i ,K i ,V i ); W0 is the dot product weight, X is the value of the input feature data; i is the index of the attention head, and the value of i is a positive integer.
[0192] In specific implementation, multiple first-level feature extraction modules can be set to simultaneously extract features from different feature data to speed up feature extraction efficiency, for example Figure 8 Two first feature extraction modules are used to extract feature vectors of own vehicle data and other vehicle data respectively.
[0193] 2) The splicing module splices multiple one-dimensional feature vectors into a multi-dimensional feature vector to obtain a first feature vector, which is a multi-dimensional feature vector;
[0194] 3) The second-level feature extraction module performs dimension expansion on the first feature vector to obtain a second feature vector, wherein the dimension of the second feature vector is higher than that of the first feature vector;
[0195] The calculation logic of the second-level feature extraction module is similar to that of the first-level feature extraction module. The weight coefficient W q , W k , W v , W0, etc. can have different values.
[0196] 4) The classification module classifies the second feature to obtain output information.
[0197] The calculation logic of the classification module:
[0198] Calculation logic: For each input X', the calculation formula is: Out = X'W + b, where W and b are weight and bias respectively. X' is the high-dimensional feature vector output by the second-level feature extraction module, such as the second feature vector. Out includes behavior recognition results and / or level recognition results.
[0199] In some embodiments, in order to obtain the above-mentioned model, the sample data set may be input into the initial model for training in advance. The sample data set includes a number of sample data with positive labels and / or sample data with negative labels, and the sample data may include the behavior information of the sample object, the behavior information of the driver of the sample vehicle, and the attention information of the driver of the sample vehicle. The positive label is used to indicate that the driver behavior of the sample vehicle is abnormal and / or the level of abnormal driver behavior of the sample vehicle, and the negative label is used to indicate that the driver behavior of the sample vehicle is normal; or, the negative label is used to indicate that the driver behavior of the sample vehicle is abnormal and / or the level of abnormal driver behavior of the sample vehicle, and the positive label is used to indicate that the driver behavior of the sample vehicle is normal.
[0200] For example, Table 4 shows an example of sample data:
[0201] Table 4
[0202]
[0203] where d k for
[0204]
[0205] It can be understood that Table 4 is only some examples and is not limited thereto.
[0206] It can be understood that the various embodiments described above can be implemented separately or in combination with each other to achieve different technical effects, and this application does not limit this.
[0207] The method provided by the embodiment of the present application is introduced above in combination with the accompanying drawings, and the device provided by the embodiment of the present application is introduced below in combination with the accompanying drawings.
[0208] Based on the same technical concept, the embodiment of the present application provides a driver behavior abnormality detection device, which includes a module / unit / means for executing the method executed by the first vehicle in the above method embodiment. The module / unit / means can be implemented by software, or by hardware, or by hardware executing the corresponding software implementation.
[0209] For example, see Fig. 9 , the device may include:
[0210] An acquisition module 901 is used to acquire behavior information of a first object, behavior information of a driver of a first vehicle, and attention information of the driver; wherein the behavior information of the first object includes first information and position information of the first object, the first information is used to indicate a first behavior performed by the first object, the behavior information of the driver includes second information, the second information is used to indicate a second behavior performed by the driver, and the attention information of the driver is used to determine the object that the driver pays attention to;
[0211] The processing module 902 is used to determine whether the driver's behavior is abnormal according to the behavior information of the first object, the behavior information of the driver, and the attention information of the driver.
[0212] It should be understood that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.
[0213] In specific implementation, the above device can have a variety of product forms. Several possible product forms are introduced below.
[0214] Based on the same technical concept, the present application also provides a driver behavior abnormality detection device, see Fig.10 The device includes at least one processor 1001 and an interface circuit 1002; the interface circuit 1002 is used to receive code instructions and transmit to at least one processor 1001; at least one processor 1001 runs the code instructions to execute the method executed by the first vehicle in the above method embodiment.
[0215] See also Fig.11 The embodiment of the present application also provides a driver behavior abnormality detection device, including at least one processor 1101 and at least one memory 1102; the memory 1102 is used to store computer execution instructions; the processor 1101 is used to execute the computer execution instructions stored in the memory 1102, so that the device executes the method executed by the first vehicle in the above method embodiment.
[0216] It should be understood that the processor mentioned in the embodiments of the present application can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor implemented by reading software code stored in a memory.
[0217] Exemplarily, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0218] It should be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DirectRambus RAM, DR RAM).
[0219] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.
[0220] It should be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0221] Based on the same technical concept, the embodiment of the present application also provides a vehicle-mounted terminal, comprising Fig. 9 or Fig.10 or Fig.11 The device shown.
[0222] Based on the same technical concept, an embodiment of the present application also provides a computer-readable storage medium, which is used to store instructions. When the instructions are executed, the method executed by the first vehicle in the above method embodiment is implemented.
[0223] Based on the same technical concept, an embodiment of the present application also provides a computer program product comprising instructions. The computer program product stores instructions that, when executed on a computer, enable the computer to execute the method executed by the first vehicle in the above method embodiment.
[0224] Based on the same technical concept, the embodiment of the present application also provides a means of transport, which may include a processor, and the processor is used to execute the method executed by the first vehicle in the above method embodiment. Optionally, the means of transport is a vehicle. Optionally, the means of transport also includes a memory for storing computer programs or instructions.
[0225] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0226] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0227] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0228] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
Claims
1. A method for detecting abnormal driver behavior, characterized in that: The method comprises: Acquire behavior information of a first object, behavior information of a driver of a first vehicle, and attention information of the driver; wherein the behavior information of the first object includes first information and position information of the first object, the first information is used to indicate a first behavior performed by the first object, the behavior information of the driver includes second information, the second information is used to indicate a second behavior performed by the driver, and the driver attention information is used to determine the object that the driver is paying attention to; The abnormal behavior of the driver is determined according to the behavior information of the first object, the behavior information of the driver, and the attention information of the driver.
2. The method according to claim 1, characterized in that The acquiring behavior information of the first object, behavior information of the driver of the first vehicle, and attention information of the driver includes: detecting the behavior information of the first object by an out-of-cabin sensor, where the out-of-cabin sensor is deployed outside the cabin of the first vehicle, and the out-of-cabin sensor includes one or more of a radar, a camera around the vehicle, or a microphone of a rear vehicle; Collecting the image of the driver through a driver monitoring system DMS or a camera monitoring system CMS; performing feature analysis on the image to obtain the attention information of the driver; The behavior information of the driver of the first vehicle is detected by an in-cabin sensor, and the in-cabin sensor is deployed in the cabin of the first vehicle; the in-cabin sensor includes one or more of the DMS, the CMS, a microphone, a functional component for controlling the throttle, a functional component for controlling the brakes, a functional component for controlling the horn, a functional component for controlling the lights, and a steering wheel.
3. The method according to claim 1 or 2, characterized in that The determining that the driver's behavior is abnormal according to the behavior information of the first object, the behavior information of the driver, and the attention information of the driver includes: Determining, according to the first information, that the first behavior belongs to a first preset behavior; and Determining that the driver is paying attention to the first object according to the position information of the first object and the attention information of the driver; and Determine, according to the second information, that the second behavior belongs to a second preset behavior.
4. The method according to claim 3, characterized in that The driver attention information includes the driver's attention direction; The determining that the driver is paying attention to the first object according to the behavior information of the first object and the attention information of the driver includes: According to the position information of the first object and the attention direction of the driver, it is determined that the first object is located in the attention direction of the driver.
5. The method according to claim 4, characterized in that The attention direction includes one or more of the driver's sight direction, face orientation, finger direction or arm direction.
6. The method according to claim 4 or 5, characterized in that The behavior information of the first object further includes the occurrence time of the first behavior, and the driver's attention information further includes the time corresponding to the attention direction; The determining that the driver is paying attention to the first object according to the behavior information of the first object and the attention information of the driver further includes: According to the occurrence time of the first behavior and the time corresponding to the attention direction, it is determined that within a first time period after the first behavior occurs, the first object is located in the attention direction of the driver.
7. The method according to any one of claims 3 to 6, characterized in that: The driver's behavior information also includes the occurrence time of the second behavior; The determining that the driver's behavior is abnormal according to the behavior information of the first object, the behavior information of the driver, and the attention information of the driver further includes: determining a time during which the driver paid attention to the first object; According to the time when the driver pays attention to the first object and the time when the second behavior occurs, it is determined that the driver performs the second behavior within a second time period after paying attention to the first object.
8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Calculating the level of abnormal driver behavior according to the corresponding relationship between the preset behavior and the weight coefficient; According to the correspondence between the level of the abnormal driver behavior, the preset level and the control operation, a control operation corresponding to the level of the abnormal driver behavior is performed.
9. The method according to claim 8, characterized in that The behaviors in the corresponding relationship between the preset behaviors and the weight coefficients include honking the horn suddenly, accelerating suddenly, and frequently turning on the hazard lights, wherein the weight coefficient corresponding to the sudden acceleration ≥ the weight coefficient corresponding to honking the horn suddenly ≥ the weight coefficient corresponding to the frequent turning on the hazard lights.
10. The method according to claim 9, characterized in that The behaviors in the correspondence relationship between the preset behaviors and the weight coefficients also include sudden braking and sudden turning. The weight coefficient corresponding to sudden braking is ≥ the weight coefficient corresponding to honking the horn, and the weight coefficient corresponding to sudden turning is ≤ the weight coefficient corresponding to frequent hazard flashing.
11. The method according to claim 1 or 2, characterized in that: The determining that the driver's behavior is abnormal according to the behavior information of the first object, the behavior information of the driver, and the attention information of the driver includes: Inputting the behavior information of the first object, the behavior information of the driver, and the state information of the first vehicle into a trained model, running the model, and obtaining output information, wherein the output information includes a behavior judgment result, and the behavior judgment result is used to indicate that the driver's behavior is abnormal; The driver's behavior is determined to be abnormal according to the behavior judgment result.
12. The method according to claim 11, characterized in that The model includes a first-level feature extraction module, a splicing module, a second-level feature extraction module and a classification module; Run the model to obtain output information, including: Based on the first-level feature extraction module, feature extraction is performed on each of the behavior information of the first object, the behavior information of the driver, and the state information of the first vehicle to obtain a plurality of one-dimensional feature vectors; Based on the splicing module, multiple one-dimensional feature vectors are spliced into a multi-dimensional feature vector to obtain a first feature vector; Expanding the dimension of the first feature vector based on the second-level feature extraction module to obtain a second feature vector, wherein the dimension of the second feature vector is higher than that of the first feature vector; The second feature is predicted based on the classification module to obtain output information, where the output information includes a behavior recognition result, and the behavior recognition result is used to indicate that the driver's behavior is abnormal.
13. The method according to claim 11 or 12, characterized in that The output information also includes a level identification result, and the level identification result is used to indicate the level of abnormal driver behavior; the method also includes: According to the correspondence between the level of the abnormal driver behavior, the preset level and the control operation, a control operation corresponding to the level of the abnormal driver behavior is performed.
14. The method according to any one of claims 11 to 13, characterized in that: The method further comprises: The sample data set is input into the initial model for training to obtain the trained model; wherein the sample data set includes a number of labeled sample data, the sample data includes behavior information of the sample object, behavior information of the driver of the sample vehicle and attention information of the driver of the sample vehicle, and the label of the sample data is used to indicate the abnormal behavior of the driver of the sample vehicle and / or the level of abnormal behavior of the driver of the sample vehicle.
15. A driver behavior abnormality detection device, characterized in that: The device comprises at least one processor and at least one memory; the memory is used to store computer-executable instructions; the processor is used to execute the computer-executable instructions stored in the memory, so that the device executes the method according to any one of claims 1 to 14.
16. A vehicle, characterized in that: include: At least one sensor, used to obtain behavior information of a first object, behavior information of a driver of a first vehicle, and attention information of the driver; wherein the behavior information of the first object includes first information and position information of the first object, the first information is used to indicate a first behavior performed by the first object, the behavior information of the driver includes second information, the second information is used to indicate a second behavior performed by the driver, and the driver attention information is used to determine the object of attention of the driver; At least one processor is used to determine whether the driver's behavior is abnormal based on the behavior information of the first object, the behavior information of the driver, and the attention information of the driver.
17. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store instructions, and when the instructions are executed, the method according to any one of claims 1 to 14 is implemented.
