A driver monitoring method and system based on structured light

Abstract

The present invention mainly describes a driver monitoring method, which mainly includes obtaining point cloud data of a monitored area, filtering the point cloud data, obtaining point cloud data of a person's face, performing curve fitting on the face point cloud data, obtaining curve fitting data, extracting and estimating smooth point cloud data, and obtaining the normal direction of the smooth point cloud data. According to the above information obtained, the depth of the driver's face can be accurately perceived, the driver's attention can be more accurately judged, and the vehicle can be controlled in time through the auxiliary system and corresponding warning information can be issued.

Description

Technical Field

[0001] The present invention belongs to the technical field of driver fatigue detection, and in particular relates to a driver monitoring method and system based on structured light. Background Art

[0002] With the rapid development of the economy and the steady increase in national income, cars have gradually entered ordinary families, the number of vehicles on the road has increased rapidly, and road safety accidents have increased. More methods are needed to reduce traffic accidents. On January 11 this year, the Ministry of Public Security announced that 31.72 million new motor vehicles were registered nationwide in 2018, and the number of cars in the country exceeded 200 million for the first time that year. In 2017, there were 203,049 traffic accidents nationwide, with a death toll of 63,772, accounting for 70% of the total number of traffic deaths in the country. Most of the traffic accidents were caused by drivers' inattention while driving.

[0003] The closest prior art is CN108921135A, which uses driver detection technology. The fatigue driving detection system and its use method include a 3D structured light module, a central processing unit module and an alarm system. The 3D structured light module is installed in front of the vehicle driver's seat. The 3D structured light module is used to capture the facial features of the driver while driving, and the data captured by the 3D structured light module is sent to the central processing unit module. The central processing unit module uses a bagging algorithm to detect and analyze whether the driver is in a fatigue driving state. If the driver is detected to be driving fatigued, a signal is sent to the alarm system, and the alarm system sends an alarm to notify the driver not to drive fatigued. The invention method uses a bagging algorithm to detect and analyze whether the driver is in a fatigue driving state. This method has low accuracy and large errors. Summary of the invention

[0004] The purpose of the present invention is to provide a driver monitoring system based on structured light, which monitors the driver's attention in real time and issues an alarm when the driver's attention is deviated for a period of time. The alarm includes but is not limited to sound, light, vibration and other forms until the driver's attention returns to the front of the vehicle and the alarm is exited.

[0005] The present invention provides a driver monitoring method based on structured light to obtain point cloud data of a monitored area;

[0006] Filter the point cloud data to obtain the driver's face point cloud data;

[0007] Perform curve fitting on the driver's facial point cloud data to obtain curve fitting data;

[0008] Extract and estimate smooth point cloud data, and obtain the normal direction of the smooth point cloud data;

[0009] The obtaining of the normal direction of the smoothed point cloud data comprises:

[0010] Determine the normal line of the tangent plane of each individual point after the curve fitting, wherein the tangent plane is obtained by least squares fitting;

[0011] Get the eigenvectors and eigenvalues ​​of the covariance matrix of the slice, which is created from the neighboring point data of the point;

[0012] Finally, the normal vectors of all points are corrected to face the sensor viewpoint, and the normal directions of all points are combined;

[0013] Determine the driver's attention direction based on the driver's facial signal characteristics and normal direction;

[0014] The method of judging the direction of attention of a person according to the facial signal features and normal direction of the driver comprises: extracting the signal features of the corresponding parts according to the features of the facial features of the driver, obtaining a spatial feature histogram by calculating each point and its adjacent points and their normals, representing the spatial features of the point with the spatial feature histogram, distinguishing the corresponding facial features and the corresponding facial positions between the facial features through the corresponding spatial features, and judging the direction of attention of the driver;

[0015] The method extracts the signal features of the corresponding parts based on the features of the driver's facial features, and uses a spatial feature histogram to represent the spatial features of the point: the spatial feature histogram is obtained by calculating each point and its adjacent points and their normals, including: setting a data point, setting a spherical space with a radius of 0.1 meters around the set data point as a corresponding neighborhood, calculating a corresponding parameter value for each point in the neighborhood, converting the deviation between the normal corresponding to the point and the normal corresponding to any point in the neighborhood into an angle difference, judging the characteristics of the space through the angle difference, obtaining a spatial feature histogram, representing the spatial features of the point with the spatial feature histogram, and extracting the spatial features.

[0016] Preferably, the acquisition of point cloud data of the monitored area includes emitting structured light in the monitored area by a coding structure transmitter, receiving an image signal of the monitored area by a two-dimensional plane sensor, and decoding the image information to obtain point cloud data.

[0017] Preferably, filtering the point cloud data includes compressing the point cloud data and performing through filtering on the point cloud data.

[0018] Preferably, the through filtering is to set a range of point cloud data to be retained and delete point cloud data outside the range to be retained.

[0019] Preferably, the filtering of the point cloud data includes outlier filtering, wherein the outlier filtering is to remove noise points in the point cloud data.

[0020] Preferably, performing curve fitting on the facial point cloud data includes performing fitting processing on the facial point cloud data through a high-order polynomial, and continuously expanding the smoothing range to perform global smoothing to obtain curve fitting data.

[0021] Preferably, the driver's facial expression and attention focus direction are judged based on the normal direction data and spatial feature data obtained by information processing, and the information is transmitted to the information decision module and the alarm module to make driving decisions and alarm information.

[0022] The present invention also provides a driver monitoring device based on structured light, comprising:

[0023] 3D face modeling module, used to create 3D face models;

[0024] 3D face point cloud coordinate information acquisition module, used to calculate face point cloud coordinate information;

[0025] The orientation information calculation module is used to calculate the normal information of the point cloud information; extract and estimate the smooth point cloud data, and obtain the normal direction of the smooth point cloud data;

[0026] The obtaining of the normal direction of the smoothed point cloud data comprises:

[0027] Determine the normal line of the tangent plane of each individual point after the curve fitting, wherein the tangent plane is obtained by least squares fitting;

[0028] Obtaining eigenvectors and eigenvalues ​​of a covariance matrix of the slice, the covariance matrix being created from neighboring point data of the point;

[0029] Finally, the normal vectors of all points are corrected to face the sensor viewpoint, and the normal directions of all points are combined;

[0030] The face direction and sight line judgment module is used to judge the face direction according to the normal information; judging the direction of attention according to the driver's facial signal characteristics and normal direction includes: extracting the signal characteristics of the corresponding parts according to the characteristics of the driver's facial features, obtaining the spatial feature histogram by calculating each point and its adjacent points and their normals, using the spatial feature histogram to represent the spatial feature of the point, distinguishing the corresponding facial features and the corresponding facial positions between the facial features through the corresponding spatial features, and judging the direction of the driver's attention;

[0031] The method extracts the signal features of the corresponding parts based on the features of the driver's facial features, and uses a spatial feature histogram to represent the spatial features of the point: the spatial feature histogram is obtained by calculating each point and its adjacent points and their normals, including: setting a data point, setting a spherical space with a radius of 0.1 meters around the set data point as a corresponding neighborhood, calculating a corresponding parameter value for each point in the neighborhood, converting the deviation between the normal corresponding to the point and the normal corresponding to any point in the neighborhood into an angle difference, judging the characteristics of the space through the angle difference, obtaining a spatial feature histogram, representing the spatial features of the point with the spatial feature histogram, and extracting the spatial features.

[0032] The present invention also provides a driver monitoring device based on structured light, comprising:

[0033] The 3D face modeling module includes: a coded structured light emitter, which is used to emit structured light in the monitored area; a 3D face point cloud coordinate information acquisition module includes: a sensor and a structured light decoder, which receives the driver's face reflection and is received by the photoreceptor and then decodes to obtain point cloud data; the 3D face point cloud coordinate information acquisition module includes: a point cloud processing module, which is compressed and then processed by through-filtering, outlier filtering, curve fitting, normal estimation, and feature extraction modules to obtain target point cloud data information and point cloud normal direction and spatial feature data; a logic processing module, which obtains decision information through the logic module according to the target point cloud data information, point cloud normal direction and spatial feature data; an execution module, which obtains decision information and issues alarm information, and the vehicle drives according to the decision information.

[0034] The present invention also provides a computer-readable storage medium having computer program instructions stored thereon, wherein when the computer program instructions are executed by a processor, the processor is enabled to execute the method as described in any of the preceding items.

[0035] The present invention provides an intelligent driving car, comprising: a processor; a memory; and computer program instructions stored in the memory, wherein the computer program instructions, when executed by the processor, enable the processor to execute any of the methods described above.

[0036] The present invention can simplify the detection algorithm through the 3D structured light algorithm, without the need to train complex visual algorithms, and can accurately perceive the driver's facial depth information and more accurately judge the driver's attention. At the same time, the normal direction is used to judge the driver's attention direction, which is more accurate and faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A diagram showing a method for judging driver attention according to an embodiment of the present application is shown;

[0038] Figure 2A block diagram of a driver attention judgment system according to an embodiment of the present application is shown;

[0039] Figure 3 A diagram showing a method for processing point cloud data according to an embodiment of the present application is shown;

[0040] Figure 4 A diagram showing a system processing method according to an embodiment of the present application is shown. . DETAILED DESCRIPTION

[0041] The following will describe the implementation methods of the present invention in detail with reference to the accompanying drawings and embodiments, so that the implementation process of how the present invention applies technical means to solve technical problems and achieve corresponding technical effects can be fully understood and implemented accordingly. The embodiments of the present application and the various features in the embodiments can be combined with each other without conflict, and the technical solutions formed are all within the protection scope of the present invention.

[0042] The present invention mainly obtains that the structured light emitter emits coded structured light through the processing of point cloud data, and the light is reflected by the driver's face and received by the photoreceptor and then decoded to obtain point cloud data. The point cloud data enters the point cloud module, and after compression processing, it is processed by the straight-through filtering, outlier filtering, curve fitting, normal estimation, feature extraction and other modules and then output to the logic processing module. The first one is the driver monitoring module, which is responsible for monitoring whether the driver is concentrating, for alarming and notifying the automatic driving to stop in a safe area, and finally transmitting it to the body HMI module through the body CAN; the second module is the driver information decision module of the automatic driving system, which is responsible for judging whether the driver is paying attention to road threatening objects, notifying the automatic driving system whether to use a more conservative and stable driving strategy, and finally transmitting it to the automatic driving module through the bus.

[0043] In each cycle, the sensor transmits a large amount of point cloud data to the processor. Point cloud data is a kind of structured data. It is difficult for embedded processors to control power consumption and temperature when processing a large amount of point cloud data. Therefore, we compress the massive point cloud data to improve the efficiency of the feature extraction algorithm. The structured light sensor performs 3D environmental perception in the direction of the driver. In order to further improve the processing efficiency of the point cloud, we only need to retain the data within the range of the driver's facial activity for the received point cloud data, and cut and delete the unexpected data in this range. As a perception sensor, it is inevitable that there are noise signals. In the point cloud data, the noise signal is manifested as an outlier signal point around the feature signal. For this, we calculate the average distance between adjacent points for each point. We believe that the average result conforms to the Gaussian distribution. The specific shape is determined by the mean and standard deviation. We set the standard deviation range and filter out outliers outside the standard range. In the point cloud data, the signal value of the curvature often cannot reflect the smoothness of the surface. In order to retain the curve characteristics, we fit the point cloud in a small range through a high-order polynomial, use the polynomial difference to generate a smooth signal, and continuously move the smooth range to smooth the global. After basic processing of the point cloud, the signal is extracted and estimated.

[0044] In order to obtain the driver's attention direction, the signals of the surrounding points are integrated for each individual point cloud to obtain the normal direction of the point. In this way, the normal directions of all points are obtained. By integrating the normal directions of all points, we can estimate the direction of the driver's attention. In order to monitor the driver's mental state and other detailed performances, we extract the signal features of the corresponding parts based on the main facial features of the human face. We calculate the spatial feature histogram for each point and the adjacent points and their normals, and use the feature map to represent the spatial features of the point. The corresponding features and the corresponding facial positions between the facial features are distinguished by the corresponding features. In this way, we extract the facial feature signals, so we can obtain the driver's mental state and emotional performance.

[0045] In addition to the above methods, point cloud feature extraction can also be obtained through training of a large amount of data using a deep neural network algorithm of point cloud signals. This method also falls within the scope of protection of this scheme.

[0046] The above method allows us to obtain the driver's facial orientation information and the driver's mental state and emotional expression. We can determine whether the driver is currently driving fatigued, distracted or emotionally agitated. The program uses images and voice to warn the driver or instruct the autonomous driving system to slowly stop the car in a safe area. Around the autonomous vehicle, we can use cameras, radars and lidars to determine the relative position of surrounding objects and vehicles. We can determine the driver's current attention range and gaze object by combining the driver's attention direction obtained through driver analysis. We can also understand whether the driver is paying attention to the urgent road conditions around the vehicle and determine whether to prompt the driver through the program.

[0047] Embodiment 1

[0048] Figure 1 The figure shows the logical processing block diagram of the driver attention information detection system.

[0049] like Figure 1 The driver attention information detection system according to this embodiment includes a structured light emitter and a detection device or sensor, such as a camera.

[0050] Step S101: The driver detection system is turned on.

[0051] Step S102: The coded structured light emitter sends structured light to the area where the driver is located. The structured light emitter emits light with specific phase coding, and the phase coding includes but is not limited to time coding, direct coding and space coding.

[0052] In step S103, the sensor or camera receives image information of the area where the driver is located, and obtains the phase difference of the same pixel position by comparing it with the signal phase when light is projected without an object. The coordinate information of the corresponding pixel point is then calculated by the phase difference. The data of multiple pixel points of a picture thus obtained constitutes a set of 3D data, namely point cloud data.

[0053] In step S104, after the processor obtains the point cloud data, it performs corresponding processing on the point cloud data. The processing method includes data compression, through filtering, outlier filtering, curve fitting, normal calculation, and feature extraction steps to process the point cloud data. The above point cloud processing steps obtain the required point cloud information.

[0054] Step S105, facial feature matching and attention monitoring, in order to monitor the driver's mental state and other details, we extract the signal features of the corresponding parts based on the features of the main facial features of the human face. We obtain the spatial feature histogram by calculating each point and its adjacent points and their normals, and use the feature map to represent the spatial features of the point. By using the corresponding features to distinguish the corresponding facial features and the corresponding facial positions between the facial features, we judge and extract the facial feature signals, so that we can obtain the driver's mental state and emotional expression.

[0055] Step S106, judging whether the driver is paying attention to the road according to the facial features extracted in step S105 and the normal vector calculated, if the driver is paying attention to the road condition normally, it is judged that the current state is safe and no measures will be taken.

[0056] Step S107: According to step S105, if it is determined that the driver is not paying enough attention, the intelligent driving system will select a safer driving strategy.

[0057] Step S108, judging whether the driver is concentrating according to the facial features judged in step S105, another way to judge whether the driver is concentrating, if not, an alarm should be issued to remind the driver, if the driver is not concentrating for many times, the automatic driving system should be notified to stop the car in a safe zone. Another way to judge whether the driver is concentrating, if not, an alarm should be issued to remind the driver, if the driver is not concentrating for many times, the automatic driving system should be notified to stop the car in a safe zone.

[0058] Step S109, judging whether the driver is paying attention based on the driver's facial features and the direction of his attention, if not, the driver should be reminded by an alarm, and judging whether the inattention occurs multiple times, if so, the driver should be reminded to pay attention in a certain way.

[0059] Step S111, determining whether there are multiple instances of inattention, and if there are no multiple instances of inattention, reminding the driver to concentrate in a certain manner.

[0060] Step S110, determining whether there are multiple instances of inattention, and if so, starting the setting to require the automatic driving system to stop in a safe zone.

[0061] Step S111: issue a graphic, sound or light prompt to remind the driver to concentrate.

[0062] Step S112, complete a program cycle.

[0063] Figure 2 A block diagram of a driver attention judgment system according to an embodiment of the present application is shown. Figure 2 The structure shown includes

[0064] The above method obtains the driver's facial orientation information and the driver's mental state and emotional expression. We can determine whether the driver is currently driving fatigued, distracted or emotionally excited. The program warns the driver through images and voice or commands the autonomous driving system to slowly stop the car in a safe area. Around the autonomous driving vehicle, we can determine the relative position of surrounding objects and vehicles through cameras, radars and lidars. We can determine the driver's current attention range and gaze object by combining the driver's attention direction obtained through driver analysis. We can understand whether the driver's attention is focused on the urgent road conditions around the vehicle, and determine whether to prompt the driver through the program.

[0065] In the embodiment of the present invention, the point cloud data is compressed before data calculation and processing is performed on a large amount of point cloud data. Data compression can improve the efficiency of the feature extraction algorithm.

[0066] Through filtering means that after identifying the range of the driver's face, only the driver's face point cloud data is retained. The driver's face range is selected for through filtering, and the data outside the range is cropped and deleted.

[0067] After the point cloud data is straight-through filtered, the point cloud data is filtered for outliers to further improve the computational efficiency of the point cloud data. Noise signals are impurities that are inevitable in the perception sensor. In the point cloud data, noise signals appear as outlier signal points around the characteristic signal. For this, we calculate the average distance between adjacent points for each point cloud data, and set the average result to conform to the Gaussian distribution. The specific shape is determined by the mean and standard deviation. We set the standard deviation range and filter out outliers outside the standard deviation range.

[0068] Then, curve fitting is performed on the point cloud data after multiple filtering, and the point cloud in a small range is fitted through high-order polynomials. The polynomial difference is used to generate a smooth signal, and the smooth range is continuously moved to smooth the whole. The curve fitting process can better retain the curve characteristics and better reflect the smoothness of the surface.

[0069] After curve fitting, each individual point cloud is integrated with the data information of the surrounding points to calculate the normal direction of the point. In this way, the normal directions of all points are obtained. By integrating the normal directions of all points and the normal directions of the point cloud data, we can estimate the direction of the driver's attention.

[0070] The resulting driver attention direction will be calculated.

[0071] Figure 2 A block diagram of a driver attention judgment system according to an embodiment of the present application is shown.

[0072] according to Figure 2 As shown in , the driver attention judgment system includes a coded structured light emitter 210, a structured light decoding module 211, a processor 220, an in-vehicle alarm display module 230, an automatic driving module 240, a point cloud processing module, a logic processing module, and an automatic driving module.

[0073] The processor 220 includes a point cloud processing module, including a structured light processing module 211, which processes point cloud information, including coordinate information of the point cloud.

[0074] The data compression module 212 compresses the data. The point cloud data compression is to compress a large amount of point cloud data and then perform data calculation processing.

[0075] Data compression includes: within the scope of three-dimensional space, virtual spatial segmentation of point cloud data, segmentation of the plane every 0.002m, and after segmentation of three-dimensional XYZ, a large number of 0.002m*0.002m*0.002m cubic spaces can be obtained, each space is encoded, and then each space is processed. Inside each cubic space, assuming that there are n point cloud data points, corresponding to numbers 1 to n, the four points will eventually be merged into one point. Each cubic space will eventually replace the original multiple data points with a single data point, while retaining the shape structure, which improves the calculation speed.

[0076] The compressed point cloud data is obtained through the straight-through filtering 213, and the point cloud data is intelligently filtered to retain only the driver's face point cloud data.

[0077] Through filtering, the point cloud data is processed as follows. Because the point cloud data is a data point represented by three coordinate axes, we set the coordinate range, such as setting the X coordinate from -0.05 to 0.7m, the Y coordinate from -0.1 to 0.5m, and the Z coordinate from -0.3 to 0.4m as the data range we need, so each point cloud data is filtered.

[0078] The outlier filtering module 214 only retains the driver's facial point cloud data, and the outlier filtering module 214 filters out outlier points outside the standard deviation range.

[0079] The point cloud information filtered by the outlier filter module 214 is then subjected to curve fitting by the curve fitting module 215 for the point cloud data after multiple filtering. The point cloud within a small range is fitted by a high-order polynomial, a smooth signal is generated by using the polynomial difference, and the smooth range is continuously moved to smooth the whole. The curve fitting process can better preserve the curve characteristics and better reflect the smoothness of the surface.

[0080] Since optical sensors inevitably have background noise and generate random noise, unstable random noise will affect the subsequent calculation of plane curvature and point cloud normal vectors, and will cause a series of chain negative effects.

[0081] In order to filter out such problem points, the following method is adopted: for each laser point, several data points closest to the point are selected, and for these data points, the average distance from the point to all points is calculated.

[0082] The standard deviation of several points is calculated based on the average value. Through the normal distribution function, its normal distribution graph can be obtained. The normal distribution function is compared with the judgment threshold. If the normal distribution exceeds the set threshold, the point is considered to be unusable and needs to be filtered out. If it does not exceed the threshold, the point is retained.

[0083] The feature extraction module 217 is used to integrate the data information of the surrounding points of each individual point cloud after curve fitting to calculate the normal direction of the point. In this way, the normal directions of all points are obtained. By integrating the normal directions of all points and the normal directions of the point cloud data, we can estimate the direction of the driver's attention.

[0084] The feature point extraction method includes setting a data point Pi, setting a corresponding neighborhood in a spherical space with a radius of 0.1m around it, and calculating a corresponding parameter value for each point in the neighborhood.

[0085] P i With P j Two points and the corresponding normal n i and n j The relative relationship between them is obtained by calculating the corresponding coordinate system. Using the above coordinate system, the normal n i With n j The deviation between them is converted into an angle difference, through which the characteristics of the space can be judged by this parameter, and the features of the space can be extracted and matched.

[0086] The feature information 217 of the point cloud information and the point cloud information obtained by the normal calculation module 216 are transmitted to the driver monitoring module 218 and the automatic driving system driver information decision module 219. The driver monitoring module 218 parses and processes the obtained feature information and normal information to obtain an alarm message, which is transmitted to the in-vehicle alarm display module 230 through the bus. The alarm display module 230 warns the driver through images and voices through the program.

[0087] Normal calculation module 216 includes determining the normal of a certain point, which is essentially determining the normal of the tangent surface of the point. In order to obtain the tangent surface, we use the least squares method to fit the tangent surface. Therefore, estimating a surface tangent surface becomes analyzing the eigenvectors and eigenvalues ​​of a covariance matrix, which is created from the neighboring data of the query point. Specifically, for each point P, the corresponding covariance matrix is ​​finally uniformly corrected to all normal vectors toward the sensor viewpoint.

[0088] The driver information decision module 219 of the automatic driving system transmits the information to the automatic driving module 240 via the bus to instruct the automatic driving system to slowly stop in a safe area.

[0089] Through the above-mentioned embodiments of the present invention, the driver's attention can be monitored. After the driver's attention reminder function is initialized, the structured light decoding module 211 emits infrared 3D structured light, and after reaching the face, the diffuse reflection echo signal is received by the sensor. The sensor obtains the phase difference of the image pixels by calculation to obtain the face depth image to establish a face 3D model. Through real-time face posture coordinate conversion, the current attention direction of the person is calculated in real time. If the driver does not maintain driving attention for more than a certain period of time, the driver detection system will issue an alarm such as sound, vibration, optical signal, etc. If it is installed in an autonomous driving car, it will send a driver attention distraction signal to the autonomous driving controller to remind the autonomous driving system to use a safety solution or exit autonomous driving control.

[0090] Figure 3 A diagram of a point cloud data processing method according to an embodiment of the present application is shown.

[0091] like Figure 3 The figure shows the coordinate information calculation diagram. I is the intensity value at point (m, n). There are four modulation images per cycle. The intensity obtained by the camera is described as follows. I0 is the background value (understood as background noise), γ is the intensity modulation function, θ is the phase to be calculated, and α I is the phase shift of each cycle, assuming that the four cycles differ by π / 2:

[0092] Then the four cycles will receive different intensity values, and the following formula is used to calculate the (m,n) point using the four intensity values Phase value (phase range [-π / 2π / 2]):

[0093] The positional relationship between the projection device and the reference surface satisfies: the reference surface OXY is parallel to the projection surface, its Y axis is parallel to the grating stripes, and the origin O is the projection of the projection center Op on this surface. According to the geometric relationship, △BPP'~△BOpO, so

[0094]

[0095] Since the reference plane OXY is parallel to the projection plane and the Y axis is parallel to the grating direction, the phase on the reference plane changes along the X axis. That is, for any point (X, Y) on the OXY plane, let its phase be θ, we have:

[0096]

[0097] θ0 is the phase of the origin O, λ0 is the grid line pitch. There is △APP'~△AOcO'c, then

[0098]

[0099] Combining the above formulas, we get:

[0100]

[0101] Get the Z coordinate of the target point, and then combine it with the X and Y coordinates of the pixel to get the three-dimensional coordinate information of a point. Each time from transmission to reception, a whole point cloud signal is obtained. The point cloud signal needs to be processed to obtain environmental information. Through the structured light sensor, we obtain the X, Y and Z coordinate positions of the corresponding position for each pixel. The data of multiple pixels in a picture constitute a set of 3D data information, which is the point cloud data.

[0102] Figure 4 A diagram of a system processing method according to an embodiment of the present application is shown.

[0103] like Figure 4 As shown, the method includes the steps of emitting structured light in the area where the driver is located 401, a sensor acquiring image information of the area where the driver is located 402, processing the image information to obtain point cloud data information, and processing the point cloud data 403, and processing the point cloud data information to obtain point cloud normal direction information 404.

[0104] The above-mentioned program processing method includes that the 3D face modeling module includes: a coded structured light emitter, which is used to emit structured light in the monitored area; the 3D face point cloud coordinate information acquisition module includes: a sensor and a structured light decoder, which receives the driver's face reflection and is received by the photoreceptor and then decodes to obtain point cloud data; the 3D face point cloud coordinate information acquisition module includes: a point cloud processing module, which is compressed and processed by through-filtering, outlier filtering, curve fitting, normal estimation, feature extraction and other modules to obtain target point cloud data information and point cloud normal direction and spatial feature data; a logic processing module, which obtains decision information through the logic module according to the target point cloud data information, point cloud normal direction and spatial feature data; an execution module, the execution module obtains decision information and issues alarm information, and the vehicle drives according to the decision information.

[0105] A computer-readable storage medium stores computer program instructions, which, when executed by a processor, cause the processor to execute any of the methods described above.

[0106] The present invention can be applied to an intelligent driving car, which is provided with a processor, a memory, and computer program instructions stored in the memory. When the computer program instructions are executed by the processor, the processor executes the driver attention detection method of the present invention.

[0107] The advantages of this embodiment are: the detection algorithm can be simplified by using the 3D structured light algorithm, and the present invention can accurately perceive the driver's facial depth information and more accurately judge the driver's attention. At the same time, the normal direction is used to judge the driver's attention direction, which is more accurate and fast.

[0108] Embodiment 2

[0109] The specific method uses a structured light emitter to emit structured light, and a 3D structured light module is used to capture the facial features of the driver while driving. The data captured by the 3D structured light module is sent to the central processing unit module, and the 3D coordinate information of the face is monitored in real time based on 3D face recognition. The current driver's line of sight is detected by coordinate conversion to see if it meets the requirements for safe driving. Innovation point 1: 3D recognition of the face is performed through structured light, and the coordinate information of the face is determined through face 3D to obtain the face's direction information. The driver's line of sight is monitored through a three-dimensional coordinate conversion algorithm.

[0110] The 3D face modeling module is used to create a 3D face model; the 3D face point cloud coordinate information acquisition module is used to calculate the face point cloud coordinate information; the orientation information calculation module is used to calculate the point cloud information normal information; the face orientation and line of sight judgment module is used to judge the face orientation based on the normal information.

[0111] In this embodiment, a projector is used to emit light with specific phase coding, and the phase coding methods include time coding, direct coding and space coding.

[0112] Use a structured light projector to project light onto the area to be detected. When the two-dimensional plane sensor or camera receives the image signal, it compares the signal phase with that when the light is projected without an object to obtain the phase difference at the same pixel position, and then calculates the Z coordinate of the corresponding pixel through the phase difference.

[0113] like Figure 3 As shown in , I is the intensity value at point (m,n), and there are four modulation images per cycle. The intensity obtained by the camera is described as follows, I0 is the background value (understood as background noise), γ is the intensity modulation function, θ is the phase to be determined, and α Iis the phase shift of each cycle, assuming that the four cycles differ by π / 2:

[0114] I i (m, n) = I0 (m, n) + γ (m, n) cos [θ (m, n) + α I ]i=1,2,3,4

[0115] Then the four cycles will receive different intensity values, and the phase value of point (m,n) is calculated using the following formula using the four intensity values ​​(phase range [-π / 2π / 2]):

[0116]

[0117] The positional relationship between the projection device and the reference surface satisfies: the reference surface OXY is parallel to the projection surface, its Y axis is parallel to the grating stripes, and the origin O is the projection of the projection center Op on this surface. According to the geometric relationship, △BPP'~△BOpO, so

[0118]

[0119] Since the reference plane OXY is parallel to the projection plane and the Y axis is parallel to the grating direction, the phase on the reference plane changes along the X axis. That is, for any point (X, Y) on the OXY plane, let its phase be θ, we have:

[0120] 0 is the phase of the origin O, λ0 is the grid line pitch. There is △APP'~△AOcO'c, then

[0121]

[0122] Combining the above formulas, we get:

[0123]

[0124] Get the Z coordinate of the target point, and then combine it with the X and Y coordinates of the pixel to get the three-dimensional coordinate information of a point. Each time from transmission to reception, a whole point cloud signal is obtained. The point cloud signal needs to be processed to obtain environmental information. Through the structured light sensor, we obtain the X, Y and Z coordinate positions of the corresponding position for each pixel. The data of multiple pixels in a picture constitute a set of 3D data, which is called point cloud data.

[0125] The point cloud data is further processed and the point cloud and data are transmitted to the point cloud processing module for point cloud data processing.

[0126] Since point cloud data is a kind of structured data, it is difficult for embedded processors to control power consumption and temperature when processing a large amount of point cloud data. Therefore, the point cloud data is compressed to improve the efficiency of the feature extraction algorithm and reduce the amount of information processing.

[0127] like Figure 2 As shown, after the structured light decoding module decodes and obtains the point cloud data, it transmits the point cloud data to the point cloud processing module.

[0128] The point cloud processing module processes the point cloud data, uses the structured light sensor or camera to perceive the 3D environment in the driver's direction, and builds a 3D model. The structured light decoding module parses the 3D module to obtain the point cloud data, and then Figure 4 The method parses and obtains the coordinate information of the point cloud data.

[0129] The point cloud data information is transmitted to the point cloud processing module, and in the data compression module, the massive point cloud data is compressed to improve the efficiency of the feature extraction algorithm.

[0130] The point cloud data is denoised, and the noise information is manifested as outlier signal points around the characteristic signal. For this, we calculate the average distance between adjacent points for each point. We believe that the average result conforms to the Gaussian distribution. The specific shape is determined by the mean and standard deviation. We set the standard deviation range and filter out outliers outside the standard range. In the point cloud data, the signal value of the curvature often cannot reflect the smoothness of the surface. In order to retain the curve characteristics, we fit the point cloud in a small range through a high-order polynomial, use the polynomial difference to generate a smooth signal, and continuously move the smooth range to smooth the global. After completing the basic processing of the point cloud, we extract and estimate the signal of the point cloud. In order to obtain the direction of the driver's attention, the signals of the surrounding points are integrated for each individual point cloud to obtain the normal direction of the point. In this way, the normal directions of all points are obtained. By integrating the normal directions of all points, we can estimate the direction of the driver's attention. In order to monitor the driver's mental state and other details, we extract the signal features of the corresponding parts of the main facial features of the human face. We obtain the spatial feature histogram by calculating each point and its adjacent points and their normals. We use the feature map to represent the spatial features of the point, and use the corresponding features to distinguish the corresponding facial features and the corresponding facial positions between the facial features. In this way, we extract the characteristic signals of the face and can obtain the driver's mental state and emotional expression.

[0131] In addition to the above methods, point cloud feature extraction can also be obtained through training of a large amount of data using a deep neural network algorithm of point cloud signals. This method also falls within the scope of protection of this scheme.

[0132] The above method allows us to obtain the driver's facial orientation information and the driver's mental state and emotional expression. We can determine whether the driver is currently driving fatigued, distracted or emotionally agitated. The program uses images and voice to warn the driver or instruct the autonomous driving system to slowly stop the car in a safe area. Around the autonomous vehicle, we can use cameras, radars and lidars to determine the relative position of surrounding objects and vehicles. We can determine the driver's current attention range and gaze object by combining the driver's attention direction obtained through driver analysis. We can also understand whether the driver is paying attention to the urgent road conditions around the vehicle and determine whether to prompt the driver through the program.

Claims

1. A driver monitoring method based on structured light, characterized in that: Obtain point cloud data of the monitored area; Filter the point cloud data to obtain the driver's face point cloud data; Perform curve fitting on the driver's facial point cloud data to obtain curve fitting data; Extract and estimate smooth point cloud data, and obtain the normal direction of the smooth point cloud data; The obtaining of the normal direction of the smoothed point cloud data comprises: Determine the normal line of the tangent plane of each individual point after the curve fitting, wherein the tangent plane is obtained by least squares fitting; Get the eigenvectors and eigenvalues ​​of the covariance matrix of the slice, which is created from the neighboring point data of the point; Finally, the normal vectors of all points are corrected to face the sensor viewpoint, and the normal directions of all points are combined; Determine the driver's attention direction based on the driver's facial signal characteristics and normal direction; The method of judging the direction of attention of a person according to the facial signal features and normal direction of the driver comprises: extracting the signal features of the corresponding parts according to the features of the facial features of the driver, obtaining a spatial feature histogram by calculating each point and its adjacent points and their normals, representing the spatial features of the point with the spatial feature histogram, distinguishing the corresponding facial features and the corresponding facial positions between the facial features through the corresponding spatial features, and judging the direction of attention of the driver; The method extracts the signal features of the corresponding parts based on the features of the driver's facial features, and uses a spatial feature histogram to represent the spatial features of the point: the spatial feature histogram is obtained by calculating each point and its adjacent points and their normals, including: setting a data point, setting a spherical space with a radius of 0.1 meters around the set data point as a corresponding neighborhood, calculating a corresponding parameter value for each point in the neighborhood, converting the deviation between the normal corresponding to the point and the normal corresponding to any point in the neighborhood into an angle difference, judging the characteristics of the space through the angle difference, obtaining a spatial feature histogram, representing the spatial features of the point with the spatial feature histogram, and extracting the spatial features.

2. The driver monitoring method based on structured light according to claim 1, characterized in that: The acquisition of point cloud data of the monitored area includes: a coding structure transmitter emits structured light in the monitored area, a two-dimensional plane sensor receives an image signal of the monitored area, and decodes the image information to obtain point cloud data.

3. The driver monitoring method based on structured light according to claim 1, characterized in that: The filtering of the point cloud data includes compressing the point cloud data and performing through filtering on the point cloud data.

4. The driver monitoring method based on structured light according to claim 3, characterized in that: The through filtering is to set the range of point cloud data to be retained and delete the point cloud data outside the range to be retained.

5. The driver monitoring method based on structured light according to claim 4, characterized in that: The filtering of the point cloud data includes outlier filtering, wherein the outlier filtering is to remove noise points in the point cloud data.

6. The driver monitoring method based on structured light according to claim 1, characterized in that: The curve fitting of the facial point cloud data includes fitting the facial point cloud data through a high-order polynomial, and continuously expanding the smoothing range to perform global smoothing to obtain curve fitting data.

7. The driver monitoring method based on structured light according to any one of claims 1 to 6, characterized in that: Based on the normal direction data and spatial feature data obtained through information processing, the driver's facial expression and attention direction are judged, and the information is transmitted to the information decision module and the alarm module to make driving decisions and alarm information.

8. A driver monitoring device based on structured light, characterized in that: include: 3D face modeling module, used to create 3D face models; 3D face point cloud coordinate information acquisition module, used to calculate face point cloud coordinate information; The orientation information calculation module is used to calculate the normal information of the point cloud information; Extract and estimate smooth point cloud data, and obtain the normal direction of the smooth point cloud data; The obtaining of the normal direction of the smoothed point cloud data comprises: Determine the normal line of the tangent plane of each individual point after the curve fitting, wherein the tangent plane is obtained by least squares fitting; Obtaining eigenvectors and eigenvalues ​​of a covariance matrix of the slice, the covariance matrix being created from neighboring point data of the point; Finally, the normal vectors of all points are corrected to face the sensor viewpoint, and the normal directions of all points are combined; The face direction and sight line judgment module is used to judge the face direction based on the normal information; judge the direction of attention based on the driver's facial signal characteristics and normal direction, including: extracting the signal characteristics of the corresponding parts according to the characteristics of the driver's facial features, obtaining the spatial feature histogram by calculating each point and its adjacent points and their normals, using the spatial feature histogram to represent the spatial feature of the point, and distinguishing the corresponding facial features and the corresponding facial positions between the facial features through the corresponding spatial features. Determine the direction of the driver's attention; The method extracts the signal features of the corresponding parts based on the features of the driver's facial features, and uses a spatial feature histogram to represent the spatial features of the point: the spatial feature histogram is obtained by calculating each point and its adjacent points and their normals, including: setting a data point, setting a spherical space with a radius of 0.1 meters around the set data point as a corresponding neighborhood, calculating a corresponding parameter value for each point in the neighborhood, converting the deviation between the normal corresponding to the point and the normal corresponding to any point in the neighborhood into an angle difference, judging the characteristics of the space through the angle difference, obtaining a spatial feature histogram, representing the spatial features of the point with the spatial feature histogram, and extracting the spatial features.

9. The driver monitoring device based on structured light according to claim 8, characterized in that: include: The 3D face modeling module includes: a coded structured light emitter, which is used to emit structured light in the monitored area; a 3D face point cloud coordinate information acquisition module includes: a sensor and a structured light decoder, which receives the driver's face reflection and is received by the photoreceptor and then decodes to obtain point cloud data; the 3D face point cloud coordinate information acquisition module includes: a point cloud processing module, which is compressed and then processed by through-filtering, outlier filtering, curve fitting, normal estimation, and feature extraction modules to obtain target point cloud data information and point cloud normal direction and spatial feature data; a logic processing module, which obtains decision information through the logic module according to the target point cloud data information, point cloud normal direction and spatial feature data; an execution module, which obtains decision information and issues alarm information, and the vehicle drives according to the decision information.

10. A computer-readable storage medium having computer program instructions stored thereon, wherein when the computer program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

11. An intelligent driving car, characterized in that: include: processor; Memory; and computer program instructions stored in the memory, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images