Method and device for determining behavior of driver, computer equipment and storage medium

Abstract

The invention relates to the technical field of vehicles, and discloses a method and device for determining behaviors of a driver, computer equipment and a storage medium, and the method comprises the steps: obtaining target data which comprises an image which is collected by a camera of a target vehicle and is related to the driver of the target vehicle, and pressure related data corresponding to the image; determining a first key point set of the driver according to an image related to the driver in the target data, and determining a second key point set of the driver according to pressure associated data in the target data; determining a target key point set according to the first key point set and the second key point set; according to the target key point set, the behavior of the driver is determined, and when the behavior of the driver is a non-driving behavior, processing operation for the behavior of the driver is carried out.

Description

Technical Field

[0001] The present disclosure relates to the technical field of vehicles, and in particular, to a method, an apparatus, a computer device, and a storage medium for determining a driver's behavior. Background Art

[0002] Non-driving behaviors of a vehicle driver during driving, such as making a phone call or picking up an item, are one of the main causes of traffic accidents. Determining the driver's behavior and performing corresponding processing on the non-driving behavior of the driver are means to improve driving safety.

[0003] In the related art, the behavior of the driver is determined based on key points in an image captured by a vehicle camera. However, due to problems such as the narrow space inside the vehicle and the camera angle, usually only a small part of the driver's body, such as the part above the shoulders, can be captured, and the number of key points involved in determining the behavior of the vehicle driver is small, resulting in a decrease in the accuracy of the determined driver behavior. Summary of the Invention

[0004] In view of this, embodiments of the present disclosure provide a method, an apparatus, a computer device, and a storage medium for determining a driver's behavior.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for determining a driver's behavior, the method including:

[0006] Obtain target data, where the target data includes: an image related to the driver of a target vehicle collected by a camera of the target vehicle, and pressure correlation data corresponding to the image, where the pressure correlation data includes: the position of a pressure sensor installed on the seat on which the driver sits, and the pressure value collected by the pressure sensor;

[0007] Determine a first key point set of the driver according to the image related to the driver in the target data, and determine a second key point set of the driver according to the pressure correlation data in the target data, where the first key point is a key point of the body part of the driver captured by the camera, and the second key point is a key point of the body part of the driver in contact with the seat on which the driver sits;

[0008] Determine a target key point set according to the first key point set and the second key point set;

[0009] Determine the behavior of the driver according to the target key point set, and when the behavior of the driver is a non-driving behavior, perform a processing operation on the behavior of the driver.

[0010] In a possible implementation manner, determining the behavior of the driver according to the target key point set includes:

[0011] Determine the angle between the body parts of the driver according to the target key point set;

[0012] Determine the behavior of the driver according to the target key point set and the angle between the body parts of the driver.

[0013] In a possible implementation, determining the angle between the body parts of the driver according to the target key point set includes:

[0014] Generate a first vector representing the first body part of the driver according to the first target key point of the first body part of the driver in the target key point set;

[0015] Generate a second vector representing the second body part of the driver according to the second target key point of the second body part of the driver in the target key point set;

[0016] Determine the angle between the first vector and the second vector as the angle between the first body part and the second body part.

[0017] In a possible implementation, determining the target key point set according to the first key point set and the second key point set includes:

[0018] When there are a first key point and a second key point belonging to the same key point category, determine the first key point among the first key point and the second key point belonging to the same key point category as the target key point, where the first key point among the first key point and the second key point belonging to the same key point category belongs to the image related to the driver and the second key point among the first key point and the second key point belonging to the same key point category belongs to the pressure correlation data corresponding to the image.

[0019] In a possible implementation, the number of images related to the driver in the target data is multiple, the number of pressure correlation data in the target data is multiple, and the multiple images related to the driver in the target data are in one-to-one correspondence with the multiple pressure correlation data in the target data.

[0020] In a second aspect, an embodiment of the present disclosure provides a device for determining the behavior of a driver. The device for determining the behavior of a driver includes:

[0021] A target data acquisition unit, configured to acquire target data, where the target data includes: images related to the driver of the target vehicle collected by a camera of the target vehicle, and pressure correlation data corresponding to the images, and the pressure correlation data includes: the position of a pressure sensor installed on the seat where the driver sits, and the pressure value collected by the pressure sensor;

[0022] A key point determination unit, configured to determine a first set of key points of the driver according to an image related to the driver in target data, and determine a second set of key points of the driver according to pressure-related data in the target data, where the first key points are key points of body parts of the driver captured by the camera, and the second key points are key points of body parts of the driver in contact with the seat on which the driver sits;

[0023] A target key point set determination unit, configured to determine a target key point set according to the first set of key points and the second set of key points;

[0024] A driver behavior determination unit, configured to determine the behavior of the driver according to the target key point set, and perform a processing operation on the behavior of the driver when the behavior of the driver is a non-driving behavior.

[0025] In a possible implementation, the driver behavior determination unit is further configured to determine an angle between body parts of the driver according to the target key point set; and determine the behavior of the driver according to the target key point set and the angle between body parts of the driver.

[0026] In a possible implementation, the driver behavior determination unit is further configured to generate a first vector representing a first body part of the driver according to a first target key point of the first body part of the driver in the target key point set; generate a second vector representing a second body part of the driver according to a second target key point of the second body part of the driver in the target key point set; and determine an angle between the first vector and the second vector as the angle between the first body part and the second body part.

[0027] In a possible implementation, the target key point set determination unit is further configured to, when there are a first key point and a second key point belonging to the same key point category, determine the first key point among the first key point and the second key point belonging to the same key point category as the target key point, where the first key point among the first key point and the second key point belonging to the same key point category belongs to an image related to the driver, and the second key point among the first key point and the second key point belonging to the same key point category belongs to pressure-related data corresponding to the image.

[0028] In a possible implementation, the number of images related to the driver in the target data is multiple, the number of pressure-related data in the target data is multiple, and the multiple images related to the driver in the target data are in one-to-one correspondence with the multiple pressure-related data in the target data.

[0029] In a third aspect, embodiments of the present disclosure provide a computer device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to execute the method according to the first aspect or any corresponding implementation manner thereof as described above.

[0030] In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to cause a computer to execute the method according to the first aspect or any corresponding implementation manner thereof as described above.

[0031] In a fifth aspect, the present invention provides a computer program product, including computer instructions, and the computer instructions are used to cause a computer to execute the method according to the first aspect or any corresponding implementation manner thereof as described above.

[0032] The method for determining the behavior of a driver provided by the embodiments of the present disclosure determines a first key point set of the driver of the target vehicle according to an image related to the driver in the target data, and determines a second key point set of the driver of the target vehicle according to pressure-related data in the target data; determines a target key point set according to the first key point set and the second key point set; determines the behavior of the driver according to the target key point set, and when the behavior of the driver of the target vehicle is a non-driving behavior, performs a processing operation on the behavior of the driver of the target vehicle. The first key point is a key point of a body part of the driver that can be captured by a camera of the target vehicle, and the second key point is a key point of a body part of the driver of the target vehicle that contacts the seat on which the driver of the target vehicle sits. Thus, in a common case where the camera of the target vehicle cannot capture the body part of the driver of the target vehicle that contacts the seat on which the driver of the target vehicle sits, it is still possible to determine the key point, that is, the second key point, of the body part of the driver of the target vehicle that contacts the seat on which the driver of the target vehicle sits and cannot be captured by the camera of the target vehicle. Thus, the behavior of the driver is determined by using relatively rich key points of the driver of the target vehicle, improving the accuracy of the determined behavior of the driver. When the driver of the target vehicle makes a non-driving behavior during the process of driving the target vehicle, it is possible to accurately determine that the behavior of the driver of the target vehicle is a non-driving behavior, perform a processing operation on the behavior of the driver of the target vehicle, and improve the driving safety of the target vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the specific implementation manners of the present disclosure or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific implementation manners or the prior art. Obviously, the drawings in the following description are some implementation manners of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0034] Figure 1 is a schematic diagram of the principle of the method for determining the behavior of a driver provided by an embodiment of the present disclosure;

[0035] Figure 2 is a flowchart of the method for determining the behavior of a driver provided by an embodiment of the present disclosure;

[0036] Figure 3 is a flowchart of another method for determining the behavior of a driver provided by an embodiment of the present disclosure;

[0037] Figure 4 is a schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure. Detailed implementation manners

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are some, but not all, of the embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present disclosure without creative efforts shall fall within the protection scope of the present disclosure.

[0039] Refer to Figure 1 , which shows a schematic diagram of the principle of the method for determining the behavior of a driver provided by an embodiment of the present disclosure.

[0040] The method for determining the behavior of a driver provided by an embodiment of the present disclosure can be executed by a computer device of an in-vehicle unit of a target vehicle. The first key point set of the driver of the target vehicle can be obtained according to the image related to the driver in the target data. The second key point set of the driver of the target vehicle can be obtained according to the pressure correlation data. The target key point set can be determined according to the first key point set and the second key point set. When determining the behavior of the driver according to the target key point set, the feature data of the target key point set can be input into a driver behavior recognition model, and the driver behavior recognition model outputs the scores of each driver behavior in the driver behavior set. The driver behavior with the highest score is determined as the behavior of the driver of the target vehicle.

[0041] Refer to Figure 2 , which shows a flowchart of the method for determining the behavior of a driver provided by an embodiment of the present disclosure. The method for determining the behavior of a driver provided by an embodiment of the present disclosure can be executed by a target vehicle, and the target vehicle can be any vehicle to which the method for determining the behavior of a driver provided by an embodiment of the present disclosure can be applied.

[0042] In step 201, obtain target data.

[0043] The target data includes: an image related to the driver of the target vehicle collected by a camera of the target vehicle, and pressure correlation data corresponding to the image related to the driver of the target vehicle.

[0044] The pressure correlation data corresponding to the image related to the driver of the target vehicle includes: the positions of pressure sensors installed on the seat where the driver of the target vehicle sits, and the pressure values collected by the pressure sensors.

[0045] The camera of the target vehicle periodically collects images related to the driver of the target vehicle at the acquisition frequency of the camera of the target vehicle.

[0046] Generally, an image related to the driver of the target vehicle includes: a part of the body of the driver of the target vehicle.

[0047] In the embodiments of the present disclosure, for an image i related to the driver of the target vehicle, the image i related to the driver of the target vehicle corresponds to an acquisition moment j, and the image i related to the driver of the target vehicle is collected at the acquisition moment j.

[0048] Wherein, the image i related to the driver of the target vehicle can be any image related to the driver of the target vehicle.

[0049] A plurality of pressure sensors are installed on the seat where the driver of the target vehicle sits.

[0050] As an example, 7*7 pressure sensors are installed on the backrest of the seat where the driver of the target vehicle sits, and 6*7 pressure sensors are installed on the seat cushion of the seat where the driver of the target vehicle sits.

[0051] The pressure sensors on the seat where the driver of the target vehicle sits can collect pressure values periodically at the acquisition frequency of the pressure sensors.

[0052] For the pressure correlation data k, the pressure correlation data k corresponds to an acquisition moment m, and the acquisition moment m corresponding to the pressure correlation data k is the acquisition moment of the pressure value in the pressure correlation data k. The pressure correlation data k includes: the positions of each of the plurality of pressure sensors installed on the seat where the driver of the target vehicle sits, and the pressure values respectively collected by each of the plurality of pressure sensors at the acquisition moment m.

[0053] Wherein, the pressure correlation data k can be any pressure correlation data.

[0054] In an embodiment of the present disclosure, for an image related to the driver of a target vehicle in the target data, the pressure correlation data corresponding to the image related to the driver of the target vehicle may be: the pressure correlation data having the closest acquisition time to the acquisition time of the image related to the driver of the target vehicle.

[0055] In step S202, according to the image related to the driver of the target vehicle in the target data, a first key point set of the driver of the target vehicle is determined, and according to the pressure correlation data in the target data, a second key point set of the driver of the target vehicle is determined.

[0056] Among them, the first key points in the first key point set of the driver of the target vehicle are the key points of the body parts of the driver of the target vehicle captured by the camera of the target vehicle, and the second key points in the second key point set of the driver of the target vehicle are the key points of the body parts of the driver of the target vehicle in contact with the seat on which the driver of the target vehicle sits.

[0057] In a possible implementation manner, in step S202, using a first key point detection model, according to the image related to the driver of the target vehicle in the target data, a first key point set of the driver of the target vehicle is determined. In step S202, using a second key point detection model, according to the pressure correlation data in the target data, a second key point set of the driver of the target vehicle is determined.

[0058] In an embodiment of the present disclosure, the first key point detection model may be any neural network that can perform a regression task to determine the key points of a human body. The second key point detection model may be any neural network that can perform a regression task to determine the key points of a human body.

[0059] As an example, both the first key point detection model and the second key point detection model are the yolo11n-pose model.

[0060] Each first key point in the first key point set belongs to the corresponding key point category in the key point category set respectively. Each second key point in the second key point set belongs to the corresponding key point category in the key point category set respectively.

[0061] As an example, the key point category set includes: 68 human face key point categories, 42 hand key point categories, left shoulder key point category, right shoulder key point category, left elbow key point category, right elbow key point category, left hip key point category, right hip key point category, left knee key point category, right knee key point category, neck key point category. The 42 hand key point categories include: 21 left hand key point categories and 21 right hand key point categories.

[0062] In a possible implementation, the number of images related to the driver of the target vehicle in the target data is one. In step S202, the image related to the driver of the target vehicle in the target data is input into the first key point detection model, and the first key point detection model outputs the first key point set of the driver of the target vehicle. In step S202, the pressure correlation data corresponding to the image related to the driver of the target vehicle in the target data is input into the second key point detection model, and the second key point detection model outputs the second key point set of the driver of the target vehicle.

[0063] It should be noted that determining the driver's behavior requires the coordinates of the first key points and the second key points in the same coordinate system. In the embodiments of the present disclosure, the coordinates of the first key points output by the first key point detection model are the coordinates in the image coordinate system of the image related to the driver of the target vehicle. The coordinates output by the second key point detection model are the coordinates in the vehicle body coordinate system of the target vehicle. In order to obtain the coordinates of the first key points and the second key points in the same coordinate system, the coordinates of the second key points in the vehicle body coordinate system can be converted into the coordinates of the second key points in the image coordinate system of the image related to the driver of the target vehicle.

[0064] In a possible implementation, in order to convert the coordinates of the second key points in the vehicle body coordinate system into the coordinates of the second key points in the image coordinate system of the image related to the driver of the target vehicle, the coordinates of the second key points in the vehicle body coordinate system can be converted into the coordinates of the second key points in the image coordinate system of the image related to the driver of the target vehicle according to the internal parameters and external parameters of the camera of the target vehicle.

[0065] In another possible implementation, in order to convert the coordinates of the second key points in the vehicle body coordinate system into the coordinates of the second key points in the image coordinate system of the image related to the driver of the target vehicle, the ratio of the width (seat size) of the seat on which the driver of the target vehicle sits in the image related to the driver of the target vehicle to the actual width of the seat on which the driver of the target vehicle sits can be calculated; the coordinate values in the coordinates of the second key points in the vehicle body coordinate system are reduced by a multiple equal to this ratio to obtain the coordinates of the second key points in the image coordinate system of the image related to the driver of the target vehicle.

[0066] In the embodiments of the present disclosure, the first key point detection model is trained.

[0067] Before executing step S201, the first key point detection model is trained.

[0068] In order to train the first key point detection model, a first training set of the first key point detection model is obtained.

[0069] The first training set of the first key point detection model includes: the first training subsets corresponding to each driver behavior in the driver behavior set.

[0070] As an example, the driver behavior set includes: two driver behaviors, namely non-driving behavior and driving behavior.

[0071] As another example, the driver behavior set includes: multiple non-driving behaviors and driving behavior. The multiple non-driving behaviors include: non-driving behaviors such as making a phone call, smoking, sending a message, drinking water, looking in the mirror, taking an item, eating, talking, listening to the radio, being lost in thought, reading a text message, etc.

[0072] For a driver behavior, the first training subset corresponding to the driver behavior includes: multiple first training data corresponding to the driver behavior. The first training data corresponding to the driver behavior includes: the image for training corresponding to the driver behavior and the annotation data of the image for training corresponding to the driver behavior. Among them, the image for training corresponding to the driver behavior is collected when the corresponding driver makes the driver behavior, and the annotation data of the image for training corresponding to the driver behavior includes: the coordinates of the key points of the corresponding driver annotated in the image for training corresponding to the driver behavior in the image coordinate system when the image for training corresponding to the driver behavior is collected.

[0073] Training the first key point detection model includes: using the first training set of the first key point detection model to repeatedly perform training operations on the first key point detection model until the training end condition of the first key point detection model is satisfied.

[0074] In one training operation of the first key point detection model, input a first training data into the first key point detection model.

[0075] Input the image for training in a first training data into the first key point detection model, obtain the coordinates of the first key points predicted by the first key point detection model in the image coordinate system, determine the loss between the coordinates of the predicted first key points in the image coordinate system and the coordinates in the annotation data of the first training data, and the loss between the coordinates of the predicted first key points in the image coordinate system and the coordinates in the annotation data of the first training data is used to update the parameters of the first key point detection model.

[0076] In the embodiments of the present disclosure, the second key point detection model is trained.

[0077] Before executing step S201, train the second key point detection model.

[0078] To train the second key point detection model, obtain the second training set of the second key point detection model.

[0079] The second training set includes: a second training subset corresponding to each driver behavior in the driver behavior set.

[0080] For a driver behavior, the second training subset corresponding to the driver behavior includes: a plurality of second training data corresponding to the driver behavior, and the second training data corresponding to the driver behavior includes: pressure correlation data for training corresponding to the driver behavior, and annotation data of the pressure correlation data for training corresponding to the driver behavior. Among them, the pressure correlation data for training corresponding to the driver behavior is collected when the corresponding driver makes the driver behavior, and the annotation data of the pressure correlation data for training corresponding to the driver behavior includes: the coordinates of the annotated key points of the body part in contact with the seat where the corresponding driver sits in the vehicle body coordinate system when the pressure correlation data for training corresponding to the driver behavior is collected.

[0081] Training the second key point detection model includes: using the second training set of the second key point detection model to repeatedly perform training operations on the second key point detection model until the training end condition of the second key point detection model is met.

[0082] In one training operation of the second key point detection model, input the pressure correlation data in a second training data into the second key point detection model to obtain the coordinates of the second key points predicted by the second key point detection model in the vehicle body coordinate system, determine the loss between the coordinates of the predicted second key points in the vehicle body coordinate system and the coordinates in the annotation data of the second training data, and the loss between the coordinates of the predicted second key points in the vehicle body coordinate system and the coordinates in the annotation data of the second training data is used to update the parameters of the second key point detection model.

[0083] In step S203, determine the target key point set according to the first key point set and the second key point set.

[0084] In a possible implementation, the number of images related to the driver of the target vehicle in the target data is one. In step S203, for a key point category, if the first key point set includes the first key point of this key point category and the second key point set does not include the second key point of this key point category, then the first key point of this key point category is determined as the target key point. In step S203, for a key point category, if the first key point set does not include the first key point of this key point category and the second key point set includes the second key point of this key point category, then the second key point of this key point category is determined as the target key point. In step S203, for a key point category, if the first key point set includes the first key point of this key point category and the second key point set includes the second key point of this key point category, determine the target key point corresponding to the first key point of this key point category and the second key point of this key point category. Among them, the coordinate values of the target key point corresponding to the first key point of this key point category and the second key point of this key point category on the same image coordinate system axis are: the average value of the coordinate values of the first key point of this key point category and the second key point of this key point category on the same image coordinate system axis.

[0085] In step S204, according to the target key point set, determine the behavior of the driver of the target vehicle, and when the behavior of the driver of the target vehicle is a non-driving behavior, perform a processing operation on the behavior of the driver of the target vehicle.

[0086] In a possible implementation, in step S204, use the driver behavior recognition model to determine the behavior of the driver of the target vehicle according to the target key point set.

[0087] As an example, the processing operation for the behavior of the driver of the target vehicle includes: reminding the driver of the target vehicle to stop the non-driving behavior and / or switching the driving mode of the target vehicle from the manual driving mode to the automatic driving mode.

[0088] In the embodiments of the present disclosure, the driver behavior recognition model can be any neural network that can be used to perform classification tasks.

[0089] As an example, the driver behavior recognition model is a convolutional neural network (CNN), a deep neural network (DNN), etc.

[0090] In step S204, the feature data of the target key point set can be input into the driver behavior recognition model, and the driver behavior recognition model outputs the scores of each driver behavior in the driver behavior set. The driver behavior with the highest score is determined as the behavior of the driver of the target vehicle.

[0091] Among them, the feature data of the target key point set includes: the key point category of each target key point in the target key point set, and the coordinates of each target key point in the target key point set in the image coordinate system.

[0092] In the embodiments of the present disclosure, the driver behavior recognition model is trained.

[0093] Before performing step S201, train the driver behavior recognition model. To train the driver behavior recognition model, obtain the third training set of the driver behavior recognition model.

[0094] The third training set of the driver behavior recognition model includes: the third training subset corresponding to each driver behavior in the driver behavior set.

[0095] For a driver behavior, the third training subset corresponding to the driver behavior includes: a plurality of third training data corresponding to the driver behavior, and the third training data corresponding to the driver behavior includes: the category of each key point in the key point set for training corresponding to the driver behavior, and the coordinates of each key point in the key point set for training corresponding to the driver behavior in the image coordinate system.

[0096] Training the driver behavior recognition model includes: using the third training set of the driver behavior recognition model to repeatedly perform training operations on the driver behavior recognition model until the training end condition of the driver behavior recognition model is met.

[0097] In one training operation of the driver behavior recognition model, input a third training data into the driver behavior recognition model to obtain the driver behavior predicted by the driver behavior recognition model. Among them, the driver behavior predicted by the driver behavior recognition model refers to the driver behavior with the highest score in the driver behavior set for this third training data. After obtaining the driver behavior predicted by the driver behavior recognition model, determine the loss between the predicted driver behavior and the driver behavior corresponding to the third training subset to which the third training data belongs. The loss between the predicted driver behavior and the driver behavior corresponding to the third training subset to which the third training data belongs is used to update the parameters of the driver behavior recognition model.

[0098] Reference Figure 3 shows a schematic flowchart of a method for determining the behavior of a driver provided by an embodiment of the present disclosure.

[0099] In step S301, obtain target data.

[0100] The target data includes: an image related to the driver of the target vehicle collected by a camera of the target vehicle, and pressure correlation data corresponding to the image related to the driver of the target vehicle.

[0101] If the target data includes multiple images related to the driver of the target vehicle and multiple pressure-related data, the multiple images related to the driver of the target vehicle correspond one-to-one to the multiple pressure-related data. Among them, the multiple images related to the driver of the target vehicle can specifically be multiple consecutive images related to the driver of the target vehicle.

[0102] If the target data includes multiple images related to the driver of the target vehicle and multiple pressure-related data, then the behavior of the driver of the target vehicle is determined by using the images related to the driver of the target vehicle and the multiple pressure-related data, the richness of the key points participating in determining the behavior of the driver of the target vehicle is improved, and the accuracy of the determined behavior of the driver of the target vehicle is improved.

[0103] In the embodiments of the present disclosure, for an image related to the driver of the target vehicle in the target data, the pressure-related data corresponding to the image related to the driver of the target vehicle can be: the pressure-related data with the acquisition time closest to the acquisition time of the image related to the driver of the target vehicle.

[0104] In step S302, according to the image related to the driver of the target vehicle in the target data, a first key point set of the driver of the target vehicle is determined, and according to the pressure-related data in the target data, a second key point set of the driver of the target vehicle is determined.

[0105] In a possible implementation manner, in step S302, a first key point detection model is used to determine a first key point set of the driver of the target vehicle according to the image related to the driver of the target vehicle in the target data. In step S302, a second key point detection model is used to determine a second key point set of the driver of the target vehicle according to the pressure-related data in the target data.

[0106] If the target data includes multiple images related to the driver of the target vehicle and multiple pressure-related data, the first key point set of the driver of the target vehicle includes: the first key point subsets of each image related to the driver of the target vehicle in the multiple images related to the driver of the target vehicle.

[0107] If the target data includes multiple images related to the driver of the target vehicle and multiple pressure-related data, the second key point set of the driver of the target vehicle includes: the second key point subsets of each pressure-related data in the multiple pressure-related data.

[0108] If the target data includes multiple images related to the driver of the target vehicle and multiple pressure-related data, each first key point subset corresponds to a different second key point subset.

[0109] Among them, for an image related to the driver of the target vehicle and pressure correlation data corresponding to the image related to the driver of the target vehicle, a first key point subset of the image related to the driver of the target vehicle corresponds to a second key point subset of the pressure correlation data corresponding to the image related to the driver of the target vehicle.

[0110] If the target data includes multiple images related to the driver of the target vehicle and multiple pressure correlation data, step S301 may include: step S3011 - step S3012.

[0111] In step S3011, for each image related to the driver of the target vehicle among the multiple images related to the driver of the target vehicle in the target data, the image related to the driver of the target vehicle is input into the first key point detection model, and the coordinates of each first key point in the first key point subset of the image related to the driver of the target vehicle output by the first key point detection model in the image coordinate system of the image related to the driver of the target vehicle are obtained.

[0112] For example, for an image i related to the driver of the target vehicle, the image i related to the driver of the target vehicle is input into the first key point detection model, and the coordinates of the first key point subset of the image i related to the driver of the target vehicle output by the first key point detection model in the image coordinate system are obtained.

[0113] In step S3012, for each pressure correlation data among the multiple pressure correlation data in the target data, the pressure correlation data is input into the second key point detection model, and the coordinates of each second key point in the second key point subset of the pressure correlation data output by the second key point detection model in the vehicle body coordinate system of the target vehicle are obtained.

[0114] In step S3012, for each pressure correlation data among the multiple pressure correlation data in the target data, the coordinates of each second key point in the second key point subset of the pressure correlation data in the vehicle body coordinate system of the target vehicle are converted into the coordinates in the image coordinate system of the image related to the driver of the target vehicle corresponding to the pressure correlation data.

[0115] For example, for a pressure correlation data k, the pressure correlation data k corresponds to an image n related to the driver of the target vehicle. The pressure correlation data k is input into the second key point detection model, and the coordinates of each second key point in the second key point subset of the pressure correlation data k output by the second key point detection model in the vehicle body coordinate system of the target vehicle are obtained. The coordinates of each second key point in the second key point subset of the pressure correlation data k in the vehicle body coordinate system of the target vehicle are converted into the coordinates in the image coordinate system of the image n related to the driver of the target vehicle.

[0116] In step S303, a target key point set is determined according to the first key point set and the second key point set.

[0117] It should be noted that if the target data includes multiple images related to the driver of the target vehicle and multiple pressure-related data, the target key point set may include: a target key point subset corresponding to each image related to the driver of the target vehicle among the multiple images related to the driver of the target vehicle.

[0118] If the target data consists of an image related to the driver of the target vehicle and a pressure-related data corresponding to the image related to the driver of the target vehicle, step S303 may include: step S3031.

[0119] In step S3031, for a key point category, if the first key point set includes the first key point of the key point category and the second key point set does not include the second key point of the key point category, the first key point of the key point category is determined as the target key point. In step S3031, for a key point category, if the first key point set does not include the first key point of the key point category and the second key point set includes the second key point of the key point category, the second key point of the key point category is determined as the target key point. In step S3031, for a key point category, if the first key point set includes the first key point of the key point category and the second key point set includes the second key point of the key point category, the first key point and the second key point of the key point category are the first key point and the second key point belonging to the same key point category, and the first key point of the key point category is determined as the target key point.

[0120] If the target data includes multiple images related to the driver of the target vehicle and multiple pressure-related data, step S303 may include S3032.

[0121] In step S3032, for an image related to the driver of the target vehicle, a target key point subset corresponding to the image related to the driver of the target vehicle is determined according to the first key point subset of the image related to the driver of the target vehicle and the second key point subset of the pressure-related data corresponding to the image related to the driver of the target vehicle.

[0122] In step S3032, for an image related to the driver of the target vehicle, in a possible implementation of determining a target key-point subset corresponding to the image related to the driver of the target vehicle, for a key-point category, if the first key-point subset of the image related to the driver of the target vehicle includes the first key-point of the key-point category and the second key-point subset corresponding to the pressure-related data of the image related to the driver of the target vehicle does not include the second key-point of the key-point category, then the first key-point of the key-point category is determined as the target key-point in the target key-point subset corresponding to the image related to the driver of the target vehicle. In step S3032, for a key-point category, if the first key-point subset of the image related to the driver of the target vehicle does not include the first key-point of the key-point category and the second key-point subset corresponding to the pressure-related data of the image related to the driver of the target vehicle includes the second key-point of the key-point category, then the second key-point of the key-point category is determined as the target key-point in the target key-point subset corresponding to the image related to the driver of the target vehicle. For a key-point category, if the first key-point subset of the image related to the driver of the target vehicle includes the first key-point of the key-point category and the second key-point subset corresponding to the pressure-related data of the image related to the driver of the target vehicle includes the second key-point of the key-point category, then the first key-point of the key-point category is determined as the target key-point in the target key-point subset corresponding to the image related to the driver of the target vehicle.

[0123] In step S3032, for an image related to the driver of the target vehicle, in another possible implementation of determining the target key point subset corresponding to the image related to the driver of the target vehicle, for a key point category, if the first key point subset of the image related to the driver of the target vehicle includes the first key point of the key point category and the second key point subset corresponding to the pressure correlation data of the image related to the driver of the target vehicle does not include the second key point of the key point category, then determine the first key point of the key point category as the target key point in the target key point subset corresponding to the image related to the driver of the target vehicle. For a key point category, if the first key point subset of the image related to the driver of the target vehicle does not include the first key point of the key point category and the second key point subset corresponding to the pressure correlation data of the image related to the driver of the target vehicle includes the second key point of the key point category, then determine the second key point of the key point category as the target key point in the target key point subset corresponding to the image related to the driver of the target vehicle. For a key point category, if the first key point subset of the image related to the driver of the target vehicle includes the first key point of the key point category and the second key point subset corresponding to the pressure correlation data of the image related to the driver of the target vehicle includes the second key point of the key point category, determine the target key points in the target key point subset corresponding to the image related to the driver of the target vehicle that correspond to the first key point of the key point category and the second key point of the key point category, where the coordinate values of the target key points corresponding to the first key point of the key point category and the second key point of the key point category on the same image coordinate system axis are: the average of the coordinate values of the first key point of the key point category and the second key point of the key point category on the same image coordinate system axis.

[0124] In step S304, according to the target key point set, determine the angle between the body parts of the driver of the target vehicle; according to the target key point set and the angle between the body parts of the driver of the target vehicle, determine the behavior of the driver of the target vehicle, and when the behavior of the driver of the target vehicle is a non-driving behavior, perform a processing operation on the behavior of the driver of the target vehicle.

[0125] In a possible implementation, in step S304, use a driver behavior recognition model to determine the behavior of the driver of the target vehicle according to the target key point set and the angle between the body parts of the driver of the target vehicle.

[0126] In step S304, it is considered that the non-driving postures of the driver have a high degree of correlation with non-driving behaviors. At the same time, there is a high correlation between the non-driving postures of the driver and the angles between the driver's body parts. Considering the angles between the body parts of the driver of the target vehicle is equivalent to considering the posture of the driver of the target vehicle. Thus, when determining the behavior of the driver of the target vehicle, both the key points of the driver of the target vehicle and the posture of the driver of the target vehicle are considered, improving the accuracy of the determined driving behavior of the driver of the target vehicle.

[0127] The driver behavior recognition model can be any neural network that can be used to perform classification tasks.

[0128] As an example, the driver behavior recognition model is a convolutional neural network, a deep neural network, etc. If the target data includes multiple images related to the driver of the target vehicle and multiple pressure correlation data, the driver behavior recognition model can be a Gated Recurrent Unit (GRU).

[0129] If the target data consists of an image related to the driver of the target vehicle and a pressure correlation data corresponding to the image related to the driver of the target vehicle, step S304 may include step S3041.

[0130] In step S3041, in a possible implementation of determining the angle between the body parts of the driver of the target vehicle, for two corresponding body parts of the driver of the target vehicle, according to the positions of the target key points in the target key point set used to determine the two corresponding body parts, the angle between the two corresponding body parts is determined. The categories of the two corresponding body parts are preset, and the key point categories used to determine the target key points of the two corresponding body parts are preset.

[0131] In step S3041, in another possible implementation of determining the angle between the body parts of the driver of the target vehicle, a first vector representing the first body part of the driver of the target vehicle is generated according to the first target key point of the first body part of the driver of the target vehicle in the target key point set; a second vector representing the second body part of the driver of the target vehicle is generated according to the second target key point of the second body part of the driver of the target vehicle in the target key point set; the angle between the first vector representing the first body part of the driver of the target vehicle and the second vector representing the second body part of the driver of the target vehicle is determined as the angle between the first body part of the driver of the target vehicle and the second body part of the driver of the target vehicle. Among them, both the first target key point and the second target key point come from the target key point set.

[0132] As an example, through step S3041, vectors such as the vector representing the left thigh of the driver of the target vehicle, the vector representing the right thigh of the driver of the target vehicle, the vector representing the shoulder of the driver of the target vehicle, and the vector representing the spine of the driver of the target vehicle are obtained. The vector representing the left thigh of the driver of the target vehicle is the vector from the target key point of the left hip key point category to the target key point of the left knee key point category. The vector representing the right thigh of the driver of the target vehicle is the vector from the target key point of the right hip key point category to the target key point of the right knee key point category. The vector representing the left arm of the driver of the target vehicle is the vector from the target key point of the left shoulder key point category to the target key point of the left elbow key point category. The vector representing the right arm of the driver of the target vehicle is the vector from the target key point of the right shoulder key point category to the target key point of the right elbow key point category. The vector representing the shoulder of the driver of the target vehicle is the vector from the target key point of the left shoulder key point category to the target key point of the right shoulder key point category. The included angles between the vector representing the left thigh of the driver of the target vehicle and each of the vector representing the right thigh of the driver of the target vehicle, the vector representing the left arm of the driver of the target vehicle, the vector representing the left forearm of the driver of the target vehicle, the vector representing the right forearm of the driver of the target vehicle, the vector representing the shoulder of the driver of the target vehicle, and the vector representing the spine of the driver of the target vehicle can be determined. The included angles between the vector representing the right thigh of the driver of the target vehicle and each of the vector representing the left arm of the driver of the target vehicle, the vector representing the left forearm of the driver of the target vehicle, the vector representing the right forearm of the driver of the target vehicle, the vector representing the shoulder of the driver of the target vehicle, and the vector representing the spine of the driver of the target vehicle can be represented.

[0133] If the target data includes multiple images related to the driver of the target vehicle and multiple pressure correlation data, step S304 may include step S3042. Each image related to the driver of the target vehicle has an included angle set respectively. For an image related to the driver of the target vehicle, the included angle set of this image related to the driver of the target vehicle includes: the included angles between the body parts of the driver of the target vehicle in this image related to the driver of the target vehicle.

[0134] In step S3042, for an image related to the driver of the target vehicle, in a possible implementation of determining the included angle between the body parts of the driver of the target vehicle in this image related to the driver of the target vehicle, for the corresponding two body parts of the driver of the target vehicle in this image related to the driver of the target vehicle, according to the positions of the target key points in the target key point subset corresponding to this image related to the driver of the target vehicle, which are used to determine the corresponding two body parts, the included angle between the corresponding two body parts is determined.

[0135] In step S3042, for an image related to the driver of the target vehicle, in another possible implementation of determining the angle between the body parts of the driver of the target vehicle in the image related to the driver of the target vehicle, according to the first target key point of the first body part of the driver of the target vehicle in the target key point subset corresponding to the image related to the driver of the target vehicle, a first vector representing the first body part of the driver of the target vehicle in the image related to the driver of the target vehicle is generated; according to the second target key point of the second body part of the driver of the target vehicle in the target key point subset corresponding to the image related to the driver of the target vehicle, a second vector representing the second body part of the driver of the target vehicle in the image related to the driver of the target vehicle is generated; the angle between the first vector and the second vector is determined as the angle between the first body part and the second body part.

[0136] In step S304, the feature data of the target key point set can be input into the driver behavior recognition model, and the driver behavior recognition model outputs the scores of each driver behavior in the driver behavior set. The driver behavior set can include: driving behaviors, multiple non-driving behaviors. The driver behavior with the highest score can be determined as the behavior of the driver of the target vehicle.

[0137] If the target data consists of an image related to the driver of the target vehicle and a pressure correlation data corresponding to the image related to the driver of the target vehicle, the feature data of the target key point set includes: the key point category of each target key point in the target key point set, and the coordinates of each target key point in the image coordinate system.

[0138] If the target data includes multiple images related to the driver of the target vehicle and multiple pressure correlation data, the feature data of the target key point set includes: the key point category of each target key point in the target key point set, the coordinates of each target key point in the image coordinate system, and each angle determined according to the target key point set.

[0139] If the target data includes multiple images related to the driver of the target vehicle, and the feature data of multiple pressure - associated data target key - point sets includes: sub - feature data of a target key - point subset corresponding to each image related to the driver of the target vehicle. The driver behavior recognition model can distinguish the sub - feature data of the target key - point subset corresponding to each image related to the driver of the target vehicle. For an image related to the driver of the target vehicle, the sub - feature data of the target key - point subset corresponding to this image related to the driver of the target vehicle includes: the key - point category of each target key - point in the target key - point subset corresponding to this image related to the driver of the target vehicle, the coordinates of each target key - point in the target key - point subset corresponding to this image related to the driver of the target vehicle in the image coordinate system of this image related to the driver of the target vehicle, and the angle set of this image related to the driver of the target vehicle.

[0140] Embodiments of the present disclosure provide a device for determining the behavior of a driver. The device for determining the behavior of a driver is used to implement the above - mentioned embodiments and preferred implementation manners, and those that have been described will not be repeated. As used hereinafter, the term "unit" can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0141] The device for determining the behavior of a driver includes:

[0142] A target data acquisition unit, configured to acquire target data, where the target data includes: images related to the driver of the target vehicle collected by a camera of the target vehicle, and pressure - associated data corresponding to the images, and the pressure - associated data includes: the position of a pressure sensor installed on the seat where the driver sits, and the pressure value collected by the pressure sensor;

[0143] A key - point determination unit, configured to determine a first key - point set of the driver according to the images related to the driver in the target data, and determine a second key - point set of the driver according to the pressure - associated data in the target data;

[0144] A target key - point set determination unit, configured to determine a target key - point set according to the first key - point set and the second key - point set;

[0145] A driver behavior determination unit, configured to determine the behavior of the driver according to the target key - point set, and when the behavior of the driver is a non - driving behavior, perform a processing operation for the behavior of the driver.

[0146] In a possible implementation, the driver behavior determination unit is further configured to determine the angle between the body parts of the driver according to the target key point set; and determine the driver's behavior according to the target key point set and the angle between the body parts of the driver.

[0147] In a possible implementation, the driver behavior determination unit is further configured to generate a first vector representing the first body part of the driver according to the first target key point of the first body part of the driver in the target key point set; generate a second vector representing the second body part of the driver according to the second target key point of the second body part of the driver in the target key point set; and determine the angle between the first vector and the second vector as the angle between the first body part and the second body part.

[0148] In a possible implementation, the target key point set determination unit is further configured to, when there are a first key point and a second key point belonging to the same key point category, determine the first key point among the first key point and the second key point belonging to the same key point category as the target key point, where the first key point among the first key point and the second key point belonging to the same key point category belongs to the image related to the driver and the second key point among the first key point and the second key point belonging to the same key point category belongs to the pressure correlation data corresponding to the image.

[0149] In a possible implementation, the number of images related to the driver in the target data is multiple, the number of pressure correlation data in the target data is multiple, and the multiple images related to the driver in the target data correspond one-to-one to the multiple pressure correlation data in the target data.

[0150] In this embodiment, the device is presented in the form of functional units. Here, the unit refers to an ASIC circuit, a processor and a memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0151] The further function descriptions of the above respective units are the same as those in the corresponding above embodiments and will not be elaborated herein.

[0152] Reference Figure 4, which shows a schematic structural diagram of a computer device provided by an embodiment of the present disclosure. The computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including a high-speed interface and a low-speed interface. Each component communicates with each other through different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in the memory or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some alternative embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (such as a server array, a set of blade servers, or a multi-processor system).

[0153] The processor 10 can be a central processing unit, a network processor, or a combination thereof. Among them, the processor 10 can further include a hardware chip. The above hardware chip can be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The above programmable logic device can be a complex programmable logic device, a field-programmable gate array, a general array logic, or any combination thereof.

[0154] Among them, the memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0155] The memory 20 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the computer device, etc. In addition, the memory 20 can include a high-speed random access memory and can also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some alternative embodiments, the memory 20 can optionally include a memory remotely set relative to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0156] The memory 20 can include a volatile memory, such as a random access memory; the memory can also include a non-volatile memory, such as a flash memory, a hard disk, or a solid-state drive; the memory 20 can also include a combination of the above types of memories.

[0157] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30, and the output device 40 can be connected through a bus or other means.

[0158] The input device 30 can receive input digital or character information, and generate key signal inputs related to the user settings and function controls of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED), and a haptic feedback device (e.g., a vibration motor), etc. The above display device includes but is not limited to a liquid crystal display, a light emitting diode, a display, and a plasma display. In some alternative embodiments, the display device may be a touch screen.

[0159] The embodiments of the present disclosure also provide a computer-readable storage medium. The methods according to the embodiments of the present disclosure can be implemented in hardware, firmware, or be implemented as computer code that can be recorded on a storage medium, or be implemented by downloading over a network the original computer code stored in a remote storage medium or a non-transitory machine-readable storage medium and to be stored in a local storage medium, so that the methods described herein can be stored as such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid-state drive, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the methods shown in the above embodiments are implemented.

[0160] A part of the embodiments of the present disclosure can be applied as a computer program product, such as computer program instructions, which when executed by a computer, can call or provide the methods and / or technical solutions according to the present invention through the operation of the computer. Those skilled in the art should be able to understand that the forms in which computer program instructions exist in a computer-readable medium include but are not limited to source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include but are not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Herein, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible by the computer.

[0161] Although the embodiments of the present disclosure are described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for determining a driver's behavior, characterized in that, The method includes: Obtain target data, where the target data includes: an image related to the driver of the target vehicle collected by a camera of the target vehicle, and pressure correlation data corresponding to the image, and the pressure correlation data includes: the position of a pressure sensor installed on the seat where the driver sits, and the pressure value collected by the pressure sensor; Determine a first key point set of the driver according to the image related to the driver in the target data, and determine a second key point set of the driver according to the pressure correlation data in the target data, where the first key point is the key point of the body part of the driver captured by the camera, and the second key point is the key point of the body part of the driver in contact with the seat where the driver sits; Determine a target key point set according to the first key point set and the second key point set; Determine the behavior of the driver according to the target key point set, and when the behavior of the driver is a non-driving behavior, perform a processing operation on the behavior of the driver.

2. The method according to claim 1, characterized in that, Determining the behavior of the driver according to the target key point set includes: Determine the included angle between the body parts of the driver according to the target key point set; Determine the behavior of the driver according to the target key point set and the included angle between the body parts of the driver.

3. The method according to claim 2, wherein Determining the included angle between the body parts of the driver according to the target key point set includes: Generate a first vector representing the first body part of the driver according to the first target key point of the first body part of the driver in the target key point set; Generate a second vector representing the second body part of the driver according to the second target key point of the second body part of the driver in the target key point set; Determine the included angle between the first vector and the second vector as the included angle between the first body part and the second body part.

4. The method according to claim 1, wherein Determining the target key point set according to the first key point set and the second key point set includes: When there are a first key point and a second key point belonging to the same key point category, determine the first key point among the first key point and the second key point belonging to the same key point category as the target key point, where the first key point among the first key point and the second key point belonging to the same key point category belongs to the image related to the driver, and the second key point among the first key point and the second key point belonging to the same key point category belongs to the pressure correlation data corresponding to the image.

5. The method according to claim 1, characterized in that, The number of images related to the driver in the target data is multiple, the number of pressure correlation data in the target data is multiple, and the multiple images related to the driver in the target data and the multiple pressure correlation data in the target data are in one-to-one correspondence.

6. A device for determining a driver's behavior, characterized in that, Installed on a vehicle, the device includes: A target data acquisition unit for acquiring target data, where the target data includes: an image related to the driver of the target vehicle collected by a camera of the target vehicle, and pressure correlation data corresponding to the image, and the pressure correlation data includes: the position of a pressure sensor installed on the seat where the driver sits, and the pressure value collected by the pressure sensor; A key point determination unit, configured to determine a first set of key points of the driver according to an image related to the driver in target data, and determine a second set of key points of the driver according to pressure-related data in the target data, where the first key points are key points of body parts of the driver captured by the camera, and the second key points are key points of body parts of the driver in contact with the seat on which the driver sits; A target key point set determination unit, configured to determine a target key point set according to the first set of key points and the second set of key points; A driver behavior determination unit, configured to determine the behavior of the driver according to the target key point set, and perform a processing operation on the behavior of the driver when the behavior of the driver is a non-driving behavior.

7. The device according to claim 6, characterized in that, The driver behavior determination unit is further configured to determine an angle between body parts of the driver according to the target key point set; and determine the behavior of the driver according to the target key point set and the angle between body parts of the driver.

8. A computer device, characterized in that, Installed on a vehicle, including: A memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to execute the method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, Computer instructions are stored on the computer-readable storage medium, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 5.

10. A computer program product, characterized in that, Including computer instructions, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 5.

Images