Driver data processing system and method for collecting driver data

Abstract

A driver data processing system includes: a plurality of guide marks arranged in front of the driving position of a vehicle body; a prompting device for prompting the driver in the driving position to look at any target guide mark among the plurality of guide marks; a plurality of image acquisition devices for installation at set positions of the vehicle's cockpit to at least acquire image data of the driver in the driving position; a monitoring data processing device for generating a training data set based on the image data of the driver acquired by the plurality of image acquisition devices, the three-dimensional position information of the plurality of image acquisition devices, and the three-dimensional position information of the plurality of guide marks, for training a driver monitoring model. By acquiring data in a real vehicle, driver data that is closer to the actual situation can be obtained, so that it can be used to train a driver monitoring model that accurately monitors the driver's status.

Description

Technical Field

[0001] The present invention relates to the field of assisted driving technology, and in particular to a driver data processing system, a method for collecting driver data, a driver monitoring model training method, an assisted driving system and a vehicle. Background Art

[0002] In the prior art, driver data is collected by using an eye tracker in hardware to obtain the true value of gaze, and the position of the center point of the head is obtained through an RGBD camera to calculate the head pose. In software, the appearance of the eye tracker in the image is removed through a generative adversarial network to obtain normal image information and the true value information of gaze and head pose.

[0003] These solutions are often implemented by building simulation scenes indoors. The indoor simulation scenes are quite different from the actual use scenes, resulting in the imaging effect of the collected two-dimensional images and the gaze / head pose true value distribution being quite different from the actual use scenes. If this method is used to build a dataset and train CNN, the actual performance is often poor when running in the cockpit because the generalization ability of CNN is limited. The reason is that the in-car scenes and simulation scenes are not similar. Summary of the invention

[0004] Embodiments of the present invention provide a driver data processing system, a method for collecting driver data, a driver monitoring model training method, an assisted driving system and a vehicle, which are used to solve at least one of the above-mentioned technical problems.

[0005] In a first aspect, an embodiment of the present invention provides a driver data processing system, comprising:

[0006] A plurality of guide marks are arranged in front of the driving position of the vehicle body;

[0007] A prompting device, used for prompting a driver in a driving position to look at any target guide mark among the plurality of guide marks;

[0008] A plurality of image acquisition devices, for installation at set positions of the vehicle's cockpit, to at least acquire image data of a driver in a driving position;

[0009] A monitoring data processing device is used to generate a training data set for training a driver monitoring model based on the image data of the driver collected by the multiple image acquisition devices, the three-dimensional position information of the multiple image acquisition devices and the three-dimensional position information of the multiple guide marks.

[0010] In a second aspect, an embodiment of the present invention provides a method for collecting driver data, which includes:

[0011] A plurality of guide marks are arranged in front of the driving position of the vehicle body;

[0012] Installing a plurality of image acquisition devices at set positions in the cockpit to at least acquire image data of the driver in the driving position;

[0013] Prompting the driver to look at any target guide mark among the plurality of guide marks by a prompting device;

[0014] Collecting image data of the driver through the multiple image acquisition devices;

[0015] The monitoring data processing device generates a training data set for training a driver monitoring model based on the image data of the driver collected by the multiple image acquisition devices, the three-dimensional position information of the multiple image acquisition devices and the three-dimensional position information of the multiple guide marks.

[0016] In a third aspect, an embodiment of the present invention provides a driver monitoring model training method, which comprises: collecting driver data using the method for collecting driver data of any embodiment of the present invention to construct a training data set; and training a driver monitoring model using the training data set.

[0017] In a fourth aspect, an embodiment of the present invention provides an assisted driving system, which is equipped with a driver monitoring model trained by a driver monitoring model training method according to any embodiment of the present invention.

[0018] In a fifth aspect, an embodiment of the present invention provides a vehicle equipped with an assisted driving system according to any embodiment of the present invention.

[0019] The beneficial effect of the embodiments of the present invention is that by collecting data in a real vehicle, driver data that is closer to the actual situation can be obtained, which can be used to train a driver monitoring model that accurately monitors the driver's status. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0021] Figure 1 A schematic diagram of an embodiment of an application scenario of a driver data processing system of the present invention;

[0022] Figure 2 A schematic diagram of a flow chart of an embodiment of a method for collecting driver data of the present invention;

[0023] Figure 3A schematic diagram of a flow chart of an embodiment of a method for collecting driver data of the present invention;

[0024] Figure 4 A schematic diagram of a flow chart of an embodiment of a method for collecting driver data of the present invention;

[0025] Figure 5 The figure is a flow chart of an embodiment of a method for collecting driver data of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0028] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0029] In the present invention, "module", "device", "system" and the like refer to related entities applied to computers, such as hardware, a combination of hardware and software, software or software in execution, etc. In detail, for example, an element can be, but is not limited to, a process, a processor, an object, an executable element, an execution thread, a program and / or a computer running on a processor. In addition, an application program or a script program running on a server, a server can all be an element. One or more elements can be in an execution process and / or thread, and an element can be localized on a computer and / or distributed between two or more computers, and can be operated by various computer-readable media. An element can also communicate through local and / or remote processes according to a signal with one or more data packets, for example, a signal from a data that interacts with another element in a local system, a distributed system, and / or a network on the Internet through a signal and interacts with other systems.

[0030] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include" and "comprise" include not only those elements, but also other elements that are not explicitly listed, or also include elements inherent to such processes, methods, articles or equipment. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or equipment that includes the elements.

[0031] In order to solve the technical problems existing in the prior art, an embodiment of the present invention provides a driver data processing system, which includes: multiple guide marks for guiding the driver to look at each guide mark; multiple image acquisition devices for acquiring image data of the driver; a prompting device for prompting the driver to look at the target guide mark; and a monitoring data processing device for processing the image data acquired by multiple image acquisition devices.

[0032] like Figure 1 FIG. 1 is a schematic diagram of an embodiment of an application scenario of the driver data processing system of the present invention. In this embodiment, the driver data processing system includes:

[0033] A plurality of guide marks (1-17) are arranged in front of the driving position of the vehicle body. Exemplarily, the arrangement positions of the plurality of guide marks (1-17) are selected from a plurality of positions within the driver's normal line of sight during driving. The plurality of guide marks (1-17) may be located entirely or partially inside or outside the vehicle, and the present invention is not limited thereto. The driver may look at different guide marks by adjusting the head posture and / or the line of sight of the human eye.

[0034] Multiple image acquisition devices (cam_0-cam_n) are used to be installed at set positions in the cockpit of the vehicle to at least collect image data of the driver in the driving position. Exemplarily, multiple image acquisition devices are installed in the cockpit or outside the cockpit. For example, the image acquisition device can be installed on the front windshield (inside or outside the vehicle) by a suction cup or double-sided tape, or multiple image acquisition devices can be installed on the outside of the front windshield by a bracket installed on the roof and outside the vehicle. The present invention does not limit the fixing and installation method of multiple image acquisition devices.

[0035] Exemplarily, the multiple image acquisition devices may be any device with video and / or photography functions, for example, a camera, or a hardware device with a camera function (for example, a smart phone, etc.), which is not limited in the present invention. The multiple image acquisition devices may acquire image data of the driver at multiple different angles. The multiple image acquisition devices may take pictures at preset program intervals or in response to the control of an operator, which is not limited in the present invention.

[0036] A prompting device is used to prompt the driver in the driving seat to look at any target guide mark among the multiple guide marks. Exemplarily, the prompting device can be a voice broadcasting device or an indicating device using visible light. For example, the voice broadcasting device can prompt the driver participating in data collection to look at the target guide point through voice; the indicating device using visible light can guide the driver participating in data collection to look at the target guide point by projecting visible light to the target guide point. The prompting device can be an independent device or a module integrated in a terminal device, and the present invention is not limited to this.

[0037] The monitoring data processing device is used to generate a training data set based on the image data of the driver collected by the multiple image acquisition devices, the three-dimensional position information of the multiple image acquisition devices, and the three-dimensional position information of the multiple guide marks, so as to train the driver monitoring model. The monitoring data processing device can be an independent device or a module integrated in a terminal device.

[0038] Exemplarily, the monitoring data processing device is a mobile control terminal, which is communicatively connected to the multiple image acquisition devices, and the mobile control terminal is also used to control the multiple image acquisition devices to take pictures in response to the driver's operation.

[0039] Exemplarily, the prompting device and the monitoring data processing device may be independent devices or different modules integrated with the same terminal device, which is not limited in the present invention.

[0040] The driver data processing system of the embodiment of the present invention can be directly used in the vehicle to collect and process driver data, so that by collecting data in the actual vehicle, driver data that is closer to the actual situation can be obtained, which can be used to train a driver monitoring model that accurately monitors the driver's status.

[0041] In some embodiments, generating a training data set based on the image data of the driver captured by the multiple image capture devices, the three-dimensional position information of the multiple image capture devices, and the three-dimensional position information of the multiple guide markers includes: determining the driver's eye line of sight data and head posture data based on the image data, the three-dimensional position information of the multiple image capture devices, and the three-dimensional position information of the multiple guide markers to generate a training data set.

[0042] This embodiment can obtain the true value information of the human eye gaze and head pose and the corresponding two-dimensional image of the human head. The application method is mainly to obtain the true value of gaze and head pose to produce a data set, and to build a neural network for training on the corresponding data set, so that the neural network has the ability to predict gaze and head pose information. It is mainly used in the fields of intelligent driving DMS (driver monitoring system), human eye attention detection, etc., and the present invention does not limit this.

[0043] In some embodiments, determining the driver's eye sight data and head posture data based on the image data, the three-dimensional position information of the multiple image acquisition devices, and the three-dimensional position information of the multiple guide marks includes: determining the driver's eye sight data and head posture data when looking at the target guide mark based on the image data, the three-dimensional position information of the multiple image acquisition devices, and the three-dimensional position information of the multiple guide marks.

[0044] The present invention can obtain the true value information of gaze and head pose when the head is looking at a preset point at a specified position in the cockpit. Other similar solutions are often implemented by building simulated scenes indoors, and the scene differences are large. If other solutions are used to build a data set to train CNN, the actual performance is often not good when running in the car, because the generalization ability of CNN is limited. In the final analysis, the scene in the car is not similar to the simulated scene. The present invention can obtain the true value information of gaze and head pose in the car, thereby solving this problem.

[0045] In some embodiments, image data, three-dimensional position information of multiple image acquisition devices, and three-dimensional position information of multiple guide markers are inputs of the driver monitoring model, and human eye sight data and head posture data are outputs of the driver monitoring model. When training the driver monitoring model, image data, three-dimensional position information of multiple image acquisition devices, and three-dimensional position information of multiple guide markers are obtained from the constructed database as inputs, and corresponding human eye sight data and head posture data are obtained as output targets for training.

[0046] In some embodiments, determining the human eye sight data and head posture data of the driver when looking at the target guide mark according to the image data, the three-dimensional position information of the multiple image acquisition devices, and the three-dimensional position information of the multiple guide marks includes:

[0047] Determine the three-dimensional position information of the key points of the face of the driver when he looks at the target guide mark according to the image data, wherein the three-dimensional position information of the key points of the face includes the three-dimensional position information of the center point of the human eye and the three-dimensional position information of the center point of the head;

[0048] The eye sight data and head posture data of the driver when looking at the target guide mark are determined according to the three-dimensional position information of the key points of the face, the three-dimensional position information of the multiple image acquisition devices and the three-dimensional position information of the target guide mark.

[0049] In this embodiment, the 3D position information of key points of the face is determined by image data collected by multiple cameras, ensuring the accuracy of the obtained 3D position information of key points of the face; and then the eye sight data and head posture data are determined according to the 3D position information of multiple cameras calibrated in advance and the 3D position information of the current first preset guide point, so that both real and more accurate data can be obtained.

[0050] In some embodiments, the number of the plurality of image acquisition devices is n, and the corresponding image data includes n images; determining the three-dimensional position information of the key points of the face of the driver when looking at the target guide mark according to the image data includes:

[0051] Restore the scene depth map based on n images;

[0052] Determine the two-dimensional position information of the key points of the face in the i-th image, the i-th image corresponding to the i-th image acquisition device;

[0053] The three-dimensional position information of the key points of the face of the driver in the coordinate system of the i-th image acquisition device when the driver looks at the target guidance mark is determined according to the scene depth map and the two-dimensional position information of the key points of the face in the i-th image.

[0054] In this embodiment, each time the driver looks at a preset guide point, the driver simultaneously obtains the three-dimensional position information of multiple sets of facial key points in each camera coordinate system, thereby facilitating the acquisition of richer human eye sight data and head posture data.

[0055] In some embodiments, the plurality of image acquisition devices include a first image acquisition device and a second image acquisition device, and positions of the first image acquisition device and the second image acquisition device are determined according to the spatial distribution of the plurality of guide marks.

[0056] In this embodiment, two optimal positions are determined according to the distribution of preset guide points to set the first camera and the second camera, thereby ensuring the data collection requirements while using a minimum number of cameras.

[0057] In some embodiments, the plurality of guide marks include a plurality of guide marks in front of the cockpit and a plurality of guide marks on the left and right sides of the cockpit. Exemplarily, the plurality of guide marks in front of the cockpit include one or more guide marks set on the windshield and one or more guide marks set on the instrument panel.

[0058] For example, multiple guide marks in front of the cockpit are set on the front windshield and the instrument panel using discrete patches. By setting the guide marks in the form of patches, it is convenient to quickly complete the arrangement of preset points in different models.

[0059] In some embodiments, the multiple guide marks on the left and right sides of the cockpit include one or more guide marks set on the left and right front door glasses, and one or more guide marks set on the left and right rearview mirrors. Exemplarily, the multiple guide marks on the left and right sides of the cockpit are set on the left and right front door glasses and on the left and right rearview mirrors using discrete patches.

[0060] In some embodiments, the front windshield of the cockpit is a transparent display screen, and the multiple guide marks in front of the cockpit are presented on the transparent display screen according to set rules; the multiple guide marks on the left and right sides of the cockpit are set on the left and right front door glasses and the left and right rearview mirrors using discrete patches. Alternatively, the multiple guide marks in front of the cockpit are presented according to set rules through a display screen set on the front outer side of the cockpit; the multiple guide marks on the left and right sides of the cockpit are set on the left and right front door glasses and the left and right rearview mirrors using discrete patches.

[0061] In this embodiment, the preset guide points are presented through the “screen”, which can provide sufficiently dense preset guide points and avoid the data sparseness problem caused by discrete guide marks of physical marks.

[0062] In some embodiments, the three-dimensional position information of the plurality of image acquisition devices and the three-dimensional position information of the plurality of guide marks are predetermined.

[0063] Exemplarily, the present invention uses two or more image acquisition devices (e.g., cameras), and does not require excessive modification of the actual vehicle cockpit, so it is relatively simple and convenient to implement. It mainly includes two components: a system calibration solution and a self-labeling solution to determine the three-dimensional position information of multiple image acquisition devices and the three-dimensional position information of the multiple guide markers.

[0064] 1. The system calibration solution mainly consists of three components, namely, the intrinsic calibration of multiple cameras, the extrinsic calibration between multiple cameras, and the extrinsic calibration between the camera and the preset point (for example, the preset guide mark). Assume that we use n cameras for data acquisition, where the camera intrinsic parameters of cam_1~cam_n and the camera extrinsic parameters between cam_1~cam_n can be calculated by the existing camera extrinsic calibration method, and the existing mature technical solutions can be used. For example, the open source solution kalibr can achieve better results. The following will focus on how to perform the extrinsic calibration between the camera and the preset point.

[0065] The following combination Figure 1 This section describes how to calibrate the external parameters between the preset points and the camera. The cockpit scene can define several preset points according to the needs, such as Figure 1 17 preset points are defined in the schematic diagram. There are n cameras (n is greater than or equal to 2) for data collection. Take cam_0 as the reference camera. We need to calculate the coordinates of all preset points in the cam_0 camera coordinate system. Because spatial points have no perception ability, we consider placing an additional camera cam_s on each preset point (such as Figure 1 8 in the figure), and then perform pairwise extrinsic calibration with cam_0, we can get the rotation and translation of the preset point position camera cam_s to the reference camera cam_0. We only need to translate to get the spatial coordinates of the preset point in the cam_0 camera coordinate system, thus completing the extrinsic calibration between the preset point and the camera.

[0066] 2. The self-labeling algorithm of this solution is similar to the existing solution in general, but a unique process design is added to make the whole algorithm fast and efficient. The present invention has only one guide point. The spatial coordinates of the center point of the human eye and the center point of the head are obtained through 3D reconstruction and face key point detection. The calibration results of the camera and the preset point in the previous step can convert the center point and the guide point to the same camera coordinate system, so that the spatial vector can be obtained, and then converted into the pitch angle and yaw angle.

[0067] The key to the whole problem is how to obtain the spatial position of the key points of the face in the camera coordinate system. The present invention adopts a simple and fast method to calculate. When the present invention collects data, there are multiple sets of cameras that can be triggered at the same time. The external parameters between the cameras can be obtained by calibration, so the depth map of the head can be obtained by three-dimensional reconstruction.

[0068] Reconstruction using image data from multiple cameras is often very time-consuming and increases the complexity of the algorithm. In theory, the scene depth of the face can be restored by two cameras. However, it is necessary to determine the positions of the two cameras with the best observation effect. On the one hand, the larger the overlapping area of ​​the images between the cameras, the better the reconstruction effect; on the other hand, we are concerned about the reproduction effect of the face area, and the other parts of the head are not needed, so it is necessary to determine the two cameras facing the face. In actual use, since the spatial position of the guide point in the cockpit is known, and the external parameters between each camera are also known, the two cameras with the best observation positions can be found by comparing the distances from the spatial points to each camera, and then the scene depth is restored by stereo matching. At the same time, the face key point detection is performed on the image of one of the cameras, and the two-dimensional position of the key point is found and mapped to the three-dimensional spatial position. In this way, the spatial position of the face key point is obtained, and then the gaze and head pose are self-annotated.

[0069] In some embodiments, the present invention further provides a method for collecting driver data based on the system described in any of the aforementioned embodiments.

[0070] like Figure 2 FIG. 1 is a flow chart of an embodiment of a method for collecting driver data of the present invention. The embodiment includes the following steps:

[0071] System initialization: including the camera's intrinsic calibration, the extrinsic calibration between multiple cameras, and the extrinsic calibration between the camera and the cabin's preset points.

[0072] 1. Preset point selection

[0073] 1.1. Select a point G as the guide point from all the preset points in the cockpit according to a certain strategy (for example, randomly select a point);

[0074] 1.2. Under the guidance of the system (voice prompt or text prompt), the collector looks at the guidance point G (the head faces the guidance point and the eyes look at the guidance point).

[0075] 2. Acquire face box position

[0076] 2.1. Multiple cameras (for example, two cameras cam_0 and cam_1) restore the depth map of the scene through a multi-view reconstruction method (such as sgbm);

[0077] 2.2. Execute the face det and face landmark inference processes to obtain the 2D positions of the eye center and the head center in the camera imaging image;

[0078] 2.3. Take the 2D position area of ​​the head center point and the eye center point in the depth map, and perform threshold and average operations in sequence to roughly obtain the depth value of the relevant center point after filtering, and then obtain the spatial position of the corresponding point in the camera coordinate system through coordinate system conversion.

[0079] 3. Calculate the human eye sight and head posture

[0080] 3.1. Map the spatial positions of the head center and eye center obtained in the previous step to the coordinate systems of other cameras (external parameters between cameras are required);

[0081] If we take two cameras cam_0 and cam_1 as an example, and the coordinate system originally used is the coordinate system of cam_0, then the "other camera" refers to camera cam_1.

[0082] 3.2. Map the guide point G in the cockpit to the coordinate system of each camera (need external parameters between cameras and external parameters between the camera and the preset guide point G);

[0083] 3.3. In each camera coordinate system, calculate the gaze vector and head pose vector respectively and convert them into the pitch and yaw of the gaze and the pitch and yaw of the head pose;

[0084] Take two cameras cam_0 and cam_1 as an example:

[0085] In the coordinate system of camera cam_0:

[0086] When the preset point (1) is the guide point G, the gaze vector_01 and head pose vector_01 are calculated and converted into the gaze pitch_01, yaw_01 and the head pose pitch_01, yaw_01;

[0087] When the preset point (2) is the guide point G, the gaze vector_02 and head pose vector_02 are calculated and converted into the gaze pitch_02, yaw_02 and the head pose pitch_02, yaw_02;

[0088] …

[0089] When the preset point (18) is the guide point G, the gaze vector_018 and head pose vector_018 are calculated and converted into the pitch_018, yaw_018 of the gaze and the pitch_018, yaw_018 of the head pose.

[0090] In the coordinate system of camera cam_1:

[0091] When the preset point (1) is the guide point G, the gaze vector_11 and the head pose vector_11 are calculated and converted into the gaze pitch_11, yaw_11 and the head pose pitch_11, yaw_11;

[0092] When the preset point (2) is the guide point G, the gaze vector_12 and the head pose vector_12 are calculated and converted into the gaze pitch_12, yaw_12 and the head pose pitch_12, yaw_12;

[0093] …

[0094] When the preset point (18) is the guide point G, the gaze vector_118 and the head pose vector_118 are calculated and converted into the pitch_118, yaw_118 of the gaze and the pitch_118, yaw_118 of the head pose.

[0095] 3.4. Finally, multiple self-labeling results (pitch, yaw of head pose and pitch, yaw of gaze) are output.

[0096] Finally, it is determined whether the data collection time for each person has reached the scheduled time. If it exceeds the scheduled time, the collection ends, otherwise the cycle continues.

[0097] like Figure 3 FIG. 2 is a method for collecting driver data according to another embodiment of the present invention. In this embodiment, the method includes:

[0098] S100, setting a plurality of guide marks in front of the driving position of the vehicle body;

[0099] S200, installing a plurality of image acquisition devices at set positions in the cockpit to at least acquire image data of the driver in the driving seat;

[0100] S300, prompting the driver to look at any target guide mark among the multiple guide marks through a prompting device;

[0101] S400, collecting image data of the driver through the multiple image collection devices;

[0102] S500. The monitoring data processing device generates a training data set for training a driver monitoring model based on the image data of the driver collected by the multiple image acquisition devices, the three-dimensional position information of the multiple image acquisition devices, and the three-dimensional position information of the multiple guide marks.

[0103] The method for collecting driver data in an embodiment of the present invention can be directly applied to collect and process driver data in a vehicle, so that by collecting data in a real vehicle, driver data that is closer to the actual situation can be obtained, which can be used to train a driver monitoring model that accurately monitors the driver's status.

[0104] In some embodiments, generating a training data set based on the image data of the driver collected by the multiple image acquisition devices, the three-dimensional position information of the multiple image acquisition devices, and the three-dimensional position information of the multiple guide marks includes:

[0105] The driver's eye sight data and head posture data are determined based on the image data, the three-dimensional position information of the multiple image acquisition devices, and the three-dimensional position information of the multiple guide marks to generate a training data set.

[0106] In some embodiments, determining the driver's eye sight data and head posture data according to the image data, the three-dimensional position information of the multiple image acquisition devices, and the three-dimensional position information of the multiple guide marks includes:

[0107] The human eye sight data and head posture data of the driver when looking at the target guide mark are determined according to the image data, the three-dimensional position information of the multiple image acquisition devices and the three-dimensional position information of the multiple guide marks.

[0108] In some embodiments, the input of the driver monitoring model is the image data, the three-dimensional position information of the multiple image acquisition devices and the three-dimensional position information of the multiple guide marks, and the human eye line of sight data and head posture data are the output of the driver monitoring model.

[0109] like Figure 4 As shown, in some embodiments, determining the human eye sight data and head posture data of the driver when looking at the target guide mark according to the image data, the three-dimensional position information of the multiple image acquisition devices and the three-dimensional position information of the multiple guide marks includes:

[0110] S510, determining three-dimensional position information of key points of the face of the driver when he looks at the target guide mark according to the image data, wherein the three-dimensional position information of key points of the face includes three-dimensional position information of the center point of the eye and three-dimensional position information of the center point of the head;

[0111] S520. Determine the eye sight data and head posture data of the driver when he looks at the target guide mark based on the three-dimensional position information of the facial key points, the three-dimensional position information of the multiple image acquisition devices, and the three-dimensional position information of the target guide mark.

[0112] In some embodiments, the number of the plurality of image acquisition devices is n, and the corresponding image data includes n images; Figure 5 As shown, determining the three-dimensional position information of the key points of the driver's face when he looks at the target guide mark according to the image data includes:

[0113] S511, restoring a scene depth map according to the n images;

[0114] S512, determining the two-dimensional position information of the key points of the face in the i-th image, wherein the i-th image corresponds to the i-th image acquisition device;

[0115] S513. Determine the three-dimensional position information of the facial key points in the coordinate system of the i-th image acquisition device when the driver looks at the target guidance mark according to the scene depth map and the two-dimensional position information of the facial key points in the i-th image.

[0116] In some embodiments, the plurality of image acquisition devices include a first image acquisition device and a second image acquisition device, and positions of the first image acquisition device and the second image acquisition device are determined according to the spatial distribution of the plurality of guide marks.

[0117] In some embodiments, the plurality of guide marks include a plurality of guide marks in front of the cockpit and a plurality of guide marks on the left and right sides of the cockpit.

[0118] In some embodiments, the plurality of guide marks in front of the cockpit include one or more guide marks set on the windshield, and one or more guide marks set on the instrument panel.

[0119] In some embodiments, multiple guide markers in front of the cockpit are provided on the front windshield and the instrument panel using discrete patches.

[0120] In some embodiments, the multiple guide marks on the left and right sides of the cockpit include one or more guide marks set on the left and right front door glasses, and one or more guide marks set on the left and right rearview mirrors.

[0121] In some embodiments, multiple guide marks on the left and right sides of the cockpit are arranged on the left and right front door glasses and on the left and right rearview mirrors using discrete patches.

[0122] In some embodiments, the front windshield of the cockpit is a transparent display screen, and the multiple guide marks in front of the cockpit are presented on the transparent display screen according to set rules; the multiple guide marks on the left and right sides of the cockpit are set on the left and right front door glasses and the left and right rearview mirrors using discrete patches.

[0123] In some embodiments, multiple guide marks in front of the cockpit are presented according to set rules through a display screen arranged on the outside of the front of the cockpit; the multiple guide marks on the left and right sides of the cockpit are arranged on the left and right front door glasses and on the left and right rearview mirrors using discrete patches.

[0124] In some embodiments, multiple image acquisition devices are installed in the cockpit or outside the cockpit.

[0125] In some embodiments, the monitoring data processing device is a mobile control terminal, which is communicatively connected to the multiple image acquisition devices. The mobile control terminal is also used to control the multiple image acquisition devices to take pictures in response to the driver's operation.

[0126] In some embodiments, the method for collecting driver data further includes:

[0127] Before collecting the image data of the driver looking at the target guide mark through the multiple image acquisition devices, calibrating the multiple image acquisition devices and the multiple guide marks in the same coordinate system to obtain a calibration result;

[0128] After the image data of the driver looking at the target guide mark is collected by the multiple image acquisition devices, the monitoring data processing device determines the human eye sight data and head posture data of the driver looking at the target guide mark according to the calibration result and the image data.

[0129] In some embodiments, calibrating the plurality of image acquisition devices and the plurality of guide marks in the same coordinate system to obtain a calibration result includes:

[0130] Performing internal parameter calibration on the multiple image acquisition devices, and performing external parameter calibration between the multiple image acquisition devices;

[0131] Selecting one of the plurality of image acquisition devices as a reference image acquisition device;

[0132] The plurality of guide marks are calibrated according to the coordinate system where the reference image acquisition device is located.

[0133] In some embodiments, calibrating the plurality of guide marks according to the coordinate system where the reference image acquisition device is located includes:

[0134] For each boot marker of the plurality of boot markers, perform:

[0135] Arrange preset points at the guide marks to calibrate the image acquisition device;

[0136] The reference image acquisition device and the preset point calibration image acquisition device are calibrated with external parameters to complete the calibration of the guide mark.

[0137] In some embodiments, calibrating the plurality of guide marks according to the coordinate system where the reference image acquisition device is located includes:

[0138] Selecting a guide mark from the plurality of guide marks as a reference preset guide point;

[0139] Arranging a preset point calibration image acquisition device at the reference preset guide point;

[0140] Performing external parameter calibration on the reference image acquisition device and the preset point calibration image acquisition device to complete the calibration of the reference preset guide point;

[0141] Other guide marks among the plurality of guide marks are calibrated according to the calibration result of the reference preset guide point and the relative position relationship between the plurality of guide marks.

[0142] In some embodiments, calibrating the plurality of image acquisition devices and the plurality of guide marks in the same coordinate system to obtain a calibration result includes:

[0143] Performing internal parameter calibration on the multiple image acquisition devices, and performing external parameter calibration between the multiple image acquisition devices;

[0144] Selecting one of the plurality of image acquisition devices as a reference image acquisition device;

[0145] The calibration of the plurality of guide marks in the reference image acquisition device coordinate system is completed by pre-measurement.

[0146] In some embodiments, calibrating the plurality of image acquisition devices and the plurality of guide marks in the same coordinate system to obtain a calibration result includes:

[0147] Performing internal parameter calibration on the multiple image acquisition devices, and performing external parameter calibration between the multiple image acquisition devices;

[0148] Selecting one of the plurality of image acquisition devices as a reference image acquisition device;

[0149] The calibration of the multiple guide markers in the reference image acquisition device coordinate system is completed by constructing a cockpit simulation model.

[0150] In some embodiments, the present invention further provides a method for training a driver monitoring model, the method comprising: collecting driver data using the method described in any of the above embodiments to construct a training data set; and training a driver monitoring model using the training data set. Exemplarily, the driver monitoring model uses a convolutional neural network model.

[0151] In some embodiments, the present invention further provides an assisted driving system, which is configured with a driver monitoring model trained by the method described in any of the aforementioned embodiments.

[0152] In some embodiments, the present invention further provides a vehicle, which is equipped with the assisted driving system described in any of the aforementioned embodiments.

[0153] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of combined actions, but those skilled in the art should be aware that the present invention is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0154] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A driver data processing system, characterized in that: include: A plurality of guide marks are arranged in front of the driving position of the vehicle body; a prompting device, used for prompting a driver in a driving position to look at any target guide mark among the plurality of guide marks; A plurality of image acquisition devices are installed at set positions of the cockpit of the vehicle to at least acquire image data of the driver in the driving position, wherein two image acquisition devices with the best observation positions are found by comparing the distances from the arbitrary target guide mark to the plurality of image acquisition devices, and then the scene depth is restored by stereo matching based on the two image acquisition devices with the best observation positions; A monitoring data processing device, used to generate a training data set for training a driver monitoring model based on the image data of the driver collected by the multiple image acquisition devices, the three-dimensional position information of the multiple image acquisition devices, and the three-dimensional position information of the multiple guide marks, wherein when training the driver monitoring model, the image data, the three-dimensional position information of the multiple image acquisition devices, and the three-dimensional position information of the multiple guide marks are obtained as input.

2. The system according to claim 1, characterized in that Generating a training data set based on the image data of the driver collected by the multiple image acquisition devices, the three-dimensional position information of the multiple image acquisition devices and the three-dimensional position information of the multiple guide marks includes: The driver's eye sight data and head posture data are determined based on the image data, the three-dimensional position information of the multiple image acquisition devices, and the three-dimensional position information of the multiple guide marks to generate a training data set.

3. The system according to claim 2, characterized in that Determining the driver's eye sight data and head posture data according to the image data, the three-dimensional position information of the multiple image acquisition devices, and the three-dimensional position information of the multiple guide marks includes: The human eye sight data and head posture data of the driver when looking at the target guide mark are determined according to the image data, the three-dimensional position information of the multiple image acquisition devices and the three-dimensional position information of the multiple guide marks.

4. The system according to claim 3, characterized in that The eye sight data and head posture data are outputs of the driver monitoring model.

5. The system according to claim 3, characterized in that Determining the human eye sight data and head posture data of the driver when looking at the target guide mark according to the image data, the three-dimensional position information of the multiple image acquisition devices and the three-dimensional position information of the multiple guide marks includes: Determine the three-dimensional position information of the key points of the face of the driver when he looks at the target guide mark according to the image data, wherein the three-dimensional position information of the key points of the face includes the three-dimensional position information of the center point of the human eye and the three-dimensional position information of the center point of the head; The eye sight data and head posture data of the driver when looking at the target guide mark are determined according to the three-dimensional position information of the key points of the face, the three-dimensional position information of the multiple image acquisition devices and the three-dimensional position information of the target guide mark.

6. The system according to claim 5, characterized in that The number of the plurality of image acquisition devices is n, and the corresponding image data includes n images; Determining the three-dimensional position information of the key points of the driver's face when he looks at the target guide mark according to the image data includes: Restoring a scene depth map according to the n images; Determine two-dimensional position information of key points of a face in an i-th image, wherein the i-th image corresponds to an i-th image acquisition device; The three-dimensional position information of the key points of the face of the driver in the coordinate system of the i-th image acquisition device when the driver looks at the target guidance mark is determined according to the scene depth map and the two-dimensional position information of the key points of the face in the i-th image.

7. The system according to claim 1, characterized in that The multiple image acquisition devices include a first image acquisition device and a second image acquisition device, and positions of the first image acquisition device and the second image acquisition device are determined according to the spatial distribution of the multiple guide marks.

8. The system according to claim 1, characterized in that The multiple guide marks include multiple guide marks in front of the cockpit and multiple guide marks on the left and right sides of the cockpit.

9. The system according to claim 8, characterized in that The multiple guide marks in front of the cockpit include one or more guide marks set on the windshield and one or more guide marks set on the instrument panel.

10. The system according to claim 9, characterized in that The plurality of guide marks in front of the cockpit are arranged on the front windshield and the instrument panel using discrete patches.

11. The system according to claim 8, characterized in that The multiple guide marks on the left and right sides of the cockpit include one or more guide marks arranged on the left and right front door glasses, and one or more guide marks arranged on the left and right rearview mirrors.

12. The system according to claim 11, characterized in that The multiple guide marks on the left and right sides of the cockpit are arranged on the left and right front door glasses and the left and right rearview mirrors using discrete patches.

13. The system according to claim 8, characterized in that The front windshield of the cockpit is a transparent display screen, and multiple guide marks in front of the cockpit are presented on the transparent display screen according to set rules; multiple guide marks on the left and right sides of the cockpit are set on the left and right front door glasses and the left and right rearview mirrors using discrete patches.

14. The system according to claim 8, characterized in that The multiple guide marks in front of the cockpit are presented according to set rules through a display screen arranged on the outer side of the front of the cockpit; the multiple guide marks on the left and right sides of the cockpit are arranged on the left and right front door glasses and the left and right rearview mirrors using discrete patches.

15. The system according to any one of claims 1 to 14, characterized in that: The multiple image acquisition devices are installed in the cockpit or outside the cockpit.

16. The system according to any one of claims 1 to 14, characterized in that: The monitoring data processing device is a mobile control terminal, which is in communication connection with the plurality of image acquisition devices. The mobile control terminal is also used to control the plurality of image acquisition devices to take pictures in response to the operation of the driver.

17. The system according to any one of claims 1 to 14, characterized in that: The three-dimensional position information of the multiple image acquisition devices and the three-dimensional position information of the multiple guide marks are predetermined.

18. A method for collecting driver data, characterized in that: include: A plurality of guide marks are arranged in front of the driving position of the vehicle body; Installing a plurality of image acquisition devices at set positions in the cockpit to at least acquire image data of the driver in the driving position; Prompting the driver to look at any target guide mark among the plurality of guide marks by a prompting device; The image data of the driver is collected by the multiple image acquisition devices, wherein two image acquisition devices with the best observation positions are found by comparing the distances from the arbitrary target guide mark to the multiple image acquisition devices, and then the scene depth is restored by a stereo matching method based on the two image acquisition devices with the best observation positions; The monitoring data processing device generates a training data set for training a driver monitoring model based on the image data of the driver collected by the multiple image acquisition devices, the three-dimensional position information of the multiple image acquisition devices, and the three-dimensional position information of the multiple guide marks, wherein when training the driver monitoring model, the image data, the three-dimensional position information of the multiple image acquisition devices, and the three-dimensional position information of the multiple guide marks are obtained as input.

19. The method according to claim 18, characterized in that Generating a training data set based on the image data of the driver collected by the multiple image acquisition devices, the three-dimensional position information of the multiple image acquisition devices and the three-dimensional position information of the multiple guide marks includes: The driver's eye sight data and head posture data are determined based on the image data, the three-dimensional position information of the multiple image acquisition devices, and the three-dimensional position information of the multiple guide marks to generate a training data set.

20. The method according to claim 19, characterized in that Determining the driver's eye sight data and head posture data according to the image data, the three-dimensional position information of the multiple image acquisition devices, and the three-dimensional position information of the multiple guide marks includes: The human eye sight data and head posture data of the driver when looking at the target guide mark are determined according to the image data, the three-dimensional position information of the multiple image acquisition devices and the three-dimensional position information of the multiple guide marks.

21. The method according to claim 20, characterized in that The eye sight data and head posture data are outputs of the driver monitoring model.

22. The method according to claim 20, characterized in that Determining the human eye sight data and head posture data of the driver when looking at the target guide mark according to the image data, the three-dimensional position information of the multiple image acquisition devices and the three-dimensional position information of the multiple guide marks includes: Determine the three-dimensional position information of the key points of the face of the driver when he looks at the target guide mark according to the image data, wherein the three-dimensional position information of the key points of the face includes the three-dimensional position information of the center point of the human eye and the three-dimensional position information of the center point of the head; The eye sight data and head posture data of the driver when looking at the target guide mark are determined according to the three-dimensional position information of the key points of the face, the three-dimensional position information of the multiple image acquisition devices and the three-dimensional position information of the target guide mark.

23. The method according to claim 22, characterized in that The number of the plurality of image acquisition devices is n, and the corresponding image data includes n images; Determining the three-dimensional position information of the key points of the driver's face when he looks at the target guide mark according to the image data includes: Restoring a scene depth map according to the n images; Determine two-dimensional position information of key points of a face in an i-th image, wherein the i-th image corresponds to an i-th image acquisition device; The three-dimensional position information of the key points of the face of the driver in the coordinate system of the i-th image acquisition device when the driver looks at the target guidance mark is determined according to the scene depth map and the two-dimensional position information of the key points of the face in the i-th image.

24. The method according to claim 18, characterized in that The multiple image acquisition devices include a first image acquisition device and a second image acquisition device, and positions of the first image acquisition device and the second image acquisition device are determined according to the spatial distribution of the multiple guide marks.

25. The method according to claim 18, characterized in that The multiple guide marks include multiple guide marks in front of the cockpit and multiple guide marks on the left and right sides of the cockpit.

26. The method according to claim 25, characterized in that The multiple guide marks in front of the cockpit include one or more guide marks set on the windshield and one or more guide marks set on the instrument panel.

27. The method according to claim 26, characterized in that The plurality of guide marks in front of the cockpit are arranged on the front windshield and the instrument panel using discrete patches.

28. The method according to claim 25, characterized in that The multiple guide marks on the left and right sides of the cockpit include one or more guide marks arranged on the left and right front door glasses, and one or more guide marks arranged on the left and right rearview mirrors.

29. The method according to claim 28, characterized in that The multiple guide marks on the left and right sides of the cockpit are arranged on the left and right front door glasses and the left and right rearview mirrors using discrete patches.

30. The method according to claim 25, characterized in that The front windshield of the cockpit is a transparent display screen, and multiple guide marks in front of the cockpit are presented on the transparent display screen according to set rules; multiple guide marks on the left and right sides of the cockpit are set on the left and right front door glasses and the left and right rearview mirrors using discrete patches.

31. The method according to claim 25, characterized in that The multiple guide marks in front of the cockpit are presented according to set rules through a display screen arranged on the outer side of the front of the cockpit; the multiple guide marks on the left and right sides of the cockpit are arranged on the left and right front door glasses and the left and right rearview mirrors using discrete patches.

32. The method according to any one of claims 18 to 31, characterized in that: The multiple image acquisition devices are installed in the cockpit or outside the cockpit.

33. The method according to any one of claims 18 to 31, characterized in that: The monitoring data processing device is a mobile control terminal, which is in communication connection with the plurality of image acquisition devices. The mobile control terminal is also used to control the plurality of image acquisition devices to take pictures in response to the operation of the driver.

34. The method according to claim 18, characterized in that Also includes: Before collecting the image data of the driver looking at the target guide mark through the multiple image acquisition devices, calibrating the multiple image acquisition devices and the multiple guide marks in the same coordinate system to obtain a calibration result; After the image data of the driver looking at the target guide mark is collected by the multiple image acquisition devices, the monitoring data processing device determines the human eye sight data and head posture data of the driver looking at the target guide mark according to the calibration result and the image data.

35. The method according to claim 34, characterized in that The step of calibrating the plurality of image acquisition devices and the plurality of guide marks in the same coordinate system to obtain calibration results comprises: Performing internal parameter calibration on the multiple image acquisition devices, and performing external parameter calibration between the multiple image acquisition devices; Selecting one of the plurality of image acquisition devices as a reference image acquisition device; The plurality of guide marks are calibrated according to the coordinate system where the reference image acquisition device is located.

36. The method according to claim 35, characterized in that The calibrating the plurality of guide marks according to the coordinate system where the reference image acquisition device is located comprises: For each boot marker of the plurality of boot markers, perform: Arrange preset points at the guide marks to calibrate the image acquisition device; The reference image acquisition device and the preset point calibration image acquisition device are calibrated with external parameters to complete the calibration of the guide mark.

37. The method according to claim 35, characterized in that The calibrating the plurality of guide marks according to the coordinate system where the reference image acquisition device is located comprises: Selecting a guide mark from the plurality of guide marks as a reference preset guide point; Arranging a preset point calibration image acquisition device at the reference preset guide point; Performing external parameter calibration on the reference image acquisition device and the preset point calibration image acquisition device to complete the calibration of the reference preset guide point; Other guide marks among the plurality of guide marks are calibrated according to the calibration result of the reference preset guide point and the relative position relationship between the plurality of guide marks.

38. The method according to claim 34, characterized in that The step of calibrating the plurality of image acquisition devices and the plurality of guide marks in the same coordinate system to obtain calibration results comprises: Performing internal parameter calibration on the multiple image acquisition devices, and performing external parameter calibration between the multiple image acquisition devices; Selecting one of the plurality of image acquisition devices as a reference image acquisition device; The calibration of the plurality of guide marks in the reference image acquisition device coordinate system is completed by pre-measurement.

39. The method according to claim 34, characterized in that The step of calibrating the plurality of image acquisition devices and the plurality of guide marks in the same coordinate system to obtain calibration results comprises: Performing internal parameter calibration on the multiple image acquisition devices, and performing external parameter calibration between the multiple image acquisition devices; Selecting one of the plurality of image acquisition devices as a reference image acquisition device; The calibration of the multiple guide markers in the reference image acquisition device coordinate system is completed by constructing a cockpit simulation model.

40. A driver monitoring model training method, characterized in that: include: Using the method described in any one of claims 18 to 39 to collect driver data to construct a training data set; The training data set is used to train a driver monitoring model.

41. The method according to claim 40, characterized in that The driver monitoring model adopts a convolutional neural network model.

42. A driving assistance system, characterized in that: A driver monitoring model trained according to the method of claim 40 or 41 is configured.

43. A vehicle, characterized in that: A driving assistance system according to claim 42 is provided.

Images