Driver monitoring system and vehicle
Through the combination of light field perception technology and infrared radiation technology, deep learning models are used to monitor driver status, and reminders are provided through light field naked-eye 3D display technology, which solves the problem of low monitoring accuracy in the existing technology and improves driver status monitoring Accuracy and driving safety.
Patent Information
- Application Number
- CN202411079782.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-07
AI Technical Summary
The existing driver monitoring system is based on two-dimensional image analysis, resulting in low monitoring accuracy and reducing the effect of driver status monitoring.
Light field perception technology is used to obtain three-dimensional surveillance video, combined with infrared radiation technology, and real-time analysis and monitoring of driver status through deep learning models, and use light field naked-eye 3D display technology to provide reminders to drivers.
It improves the accuracy and accuracy of driver status monitoring, reduces the occurrence of traffic accidents, enhances driving safety, and reduces driver fatigue and distraction through intuitive information presentation.
Smart Images

Figure CN119068464A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of driver monitoring, and in particular to a driver monitoring system and a vehicle. Background Art
[0002] The Driver Monitoring System (DMS) mainly monitors the driver's status through a camera, identifies the driver, and determines whether the driver has dangerous behaviors such as fatigue driving, so as to ensure driving safety.
[0003] In the related art, the facial and eye information is analyzed and processed based on the two-dimensional image of the driver taken by the camera, so as to determine the driver's state and monitor the driver's state. However, the monitoring accuracy of this technical solution is low, which reduces the state monitoring effect. Summary of the invention
[0004] The present invention aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the first object of the present invention is to provide a driver monitoring system, which uses a first monitoring video obtained based on light field sensing technology as input to analyze and monitor the driver's state, thereby improving the richness of input information used for state analysis, and thus improving the driver's state monitoring accuracy.
[0005] A second object of the present invention is to provide a vehicle.
[0006] To achieve the above-mentioned objectives, an embodiment of the first aspect of the present invention proposes a driver monitoring system, which is applied to a vehicle. The system includes: a first acquisition unit, which is used to obtain a first monitoring video of the driver based on light field perception technology; a data processing unit, which is used to determine the current state of the driver based on the first monitoring video, and generate target information according to the current state of the driver, so as to remind the driver based on the target information.
[0007] According to the driver monitoring system of the embodiment of the present invention, the first acquisition unit obtains the first monitoring video of the driver based on the light field sensing technology, and the data processing unit determines the current state of the driver based on the first monitoring video, and generates target information according to the current state of the driver, so as to remind the driver based on the target information. Therefore, the monitoring system uses the first monitoring video obtained based on the light field sensing technology as input to analyze and monitor the driver's state, thereby improving the richness of the input information used for state analysis, and thus improving the accuracy of driver state monitoring.
[0008] In addition, the driver monitoring system according to the above embodiment of the present invention may also have the following additional technical features:
[0009] According to one embodiment of the present invention, the driver monitoring system further includes: a display unit, configured to display the target information in front of the driver based on the light field naked-eye 3D display technology.
[0010] According to one embodiment of the present invention, the driver monitoring system also includes: a second acquisition unit, used to obtain a second monitoring video of the driver based on infrared radiation technology; the data processing unit is also used to determine the current state of the driver based on the first monitoring video and the second monitoring video.
[0011] According to an embodiment of the present invention, the first acquisition unit includes a light field camera, which is used to acquire the first monitoring video; the second acquisition unit includes an infrared camera, which is used to acquire the second monitoring video.
[0012] According to one embodiment of the present invention, the data processing unit includes: a decoding module, which is used to decode the first monitoring video and the second monitoring video to generate target image data; a preprocessing module, which is used to preprocess the target image data; a feature extraction and fusion module, which is used to extract and fuse features of the preprocessed target image data based on a deep learning model to generate target feature information; a feature classification and detection module, which is used to classify and detect the target feature information and generate a state detection result, wherein the state detection result is used to characterize the current state of the driver; and a control module, which is used to generate target information according to the state detection result.
[0013] According to one embodiment of the present invention, the control module is also connected to the first acquisition unit, the second acquisition unit and the decoding module respectively, for receiving the first monitoring video and the second monitoring video, and sending the first monitoring video and the second monitoring video to the decoding module.
[0014] According to one embodiment of the present invention, the control module is also connected to the preprocessing module and the feature extraction and fusion module respectively, and is used to respond to the notification instruction and generate the target work instruction; wherein the preprocessing module is also used to generate the notification instruction when the preprocessing of the target image data is completed; the feature extraction and fusion module is also used to perform feature extraction and fusion on the preprocessed target image data based on the target work instruction.
[0015] According to an embodiment of the present invention, the data processing unit further includes: a data buffer module, which is connected to the decoding module and the pre-processing module respectively and is used to store the target image data.
[0016] According to one embodiment of the present invention, the display unit includes: a HUD controller, which is used to establish a HUD display view corresponding to the target information based on the light field naked-eye 3D display technology, and render the HUD display view to generate a target display view; a HUD processor, which is used to process the target display view based on a preset light field texture, so as to display the processed target display view based on the vehicle's HUD display.
[0017] To achieve the above-mentioned objective, a second embodiment of the present invention provides a vehicle, comprising the above-mentioned driver monitoring system.
[0018] According to the vehicle of the embodiment of the present invention, based on the above-mentioned driver monitoring system, the driver's behavior is captured based on light field sensing technology to analyze the driver's state, realize real-time monitoring of the driver's state, improve driving safety and reduce the occurrence of traffic accidents.
[0019] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 A connection diagram of a driver monitoring system according to an embodiment of the present invention;
[0021] Figure 2 A connection diagram of a driver monitoring system according to an embodiment of the present invention;
[0022] Figure 3 A connection diagram of a driver monitoring system according to another embodiment of the present invention;
[0023] Figure 4 A connection diagram of a driver monitoring system according to a specific embodiment of the present invention;
[0024] Figure 5 FIG. 4 is a block diagram of a vehicle according to an embodiment of the present invention. DETAILED DESCRIPTION
[0025] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0026] The driver monitoring system and vehicle provided in the embodiments of the present invention are described below with reference to the accompanying drawings.
[0027] In related technologies, DMS driver status collection solutions are mainly divided into two categories:
[0028] The first type is to capture a two-dimensional plane image of the human face by an RGB camera, combined with an infrared LED (Light-Emitting Diode) and corresponding computer vision algorithms to accurately identify the driver's face under various lighting conditions.
[0029] The second category is to use binocular recognition technology, 3D structured light, and TOF (Time Of Flight) sensor technology to generate a "depth map" of the driver's head, and then accurately analyze the driver's facial features.
[0030] Although the second type of technical solution can provide depth information, its cost is relatively high, and the sensor technology is not very mature, making it difficult to ensure the quality of status monitoring. Therefore, most of the related technologies are still based on the two-dimensional images taken by the camera to analyze and process the facial and eye information. The system collects less content when processing images, which reduces the accuracy of status monitoring.
[0031] To this end, this application proposes a driver monitoring system, which shoots the driver based on light field sensing technology to obtain a first monitoring video of the driver, and uses the first monitoring video as input to perform real-time analysis and monitoring of the driver's status. Light field sensing technology can achieve three-dimensional imaging by obtaining multi-dimensional information of light through one imaging. At the same time, light field sensing technology can record the direction and angle information of light, and obtain images focused on different planes through post-focusing processing. Light field sensing technology can also use the multi-dimensional information of light to achieve de-occlusion and depth estimation, which greatly improves the richness of input information used for state analysis and improves the accuracy of driver state monitoring.
[0032] Figure 1 FIG. 4 is a connection diagram of a driver monitoring system according to an embodiment of the present invention.
[0033] like Figure 1 As shown, the driver monitoring system according to the embodiment of the present invention is applied to a vehicle and may include: a first acquisition unit 10 and a data processing unit 20 .
[0034] The first acquisition unit 10 is used to obtain a first monitoring video of the driver based on the light field sensing technology. The data processing unit 20 is used to determine the current state of the driver based on the first monitoring video, and generate target information according to the current state of the driver, so as to remind the driver based on the target information.
[0035] Specifically, the first acquisition unit 10 can obtain multi-dimensional information of light through one imaging based on the light field perception technology to achieve three-dimensional imaging and obtain the corresponding first monitoring video, wherein the first monitoring video can be a video of the driver's face. In addition, the light field perception technology can record the direction and angle information of the light, and obtain images focused on different planes through post-focusing processing. In addition, the light field perception technology can also use the multi-dimensional information of the light to achieve de-occlusion and depth estimation.
[0036] The data processing unit 20 processes the first monitoring video to obtain high-quality monitoring data, thereby determining the current state of the driver and improving the accuracy of state monitoring. The driver's state can be divided based on actual conditions, such as smoking, fatigue, making a phone call, normal, etc. After determining the driver's current state, the data processing unit 20 generates target information according to the driver's current state, so as to remind the driver based on the target information and improve driving safety.
[0037] Among them, the target information is the reminder information corresponding to the driving state. For example, when the driver is in a fatigue state, the target information is fatigue alert information; when the driver is in a phone call state, the target information is a reminder message of "Do not call", etc. In addition, the target information can be displayed in the form of sound, light, image, etc., without specific limitation. For example, when the driver is in a fatigue state, the target information is fatigue alert information in the form of an image, which is displayed through a display screen or other components; or the target information is fatigue reminder information in the form of sound, which can be corresponding reminder music or volume adjustment information, and the reminder is realized by controlling the speaker of the vehicle; or the target information is fatigue reminder information in the form of a warning light, which can be specifically reminded to the driver by lighting up the warning light on the dashboard or other locations.
[0038] The driver monitoring system can be applied in the field of intelligent assisted driving. It can capture the driver's behavior through light field sensing technology, and analyze the driver's status to achieve real-time monitoring of the driver's status, thereby improving driving safety and reducing the occurrence of traffic accidents.
[0039] Combination Figure 2 As shown, in one embodiment of the present invention, the driver monitoring system further includes: a display unit 30, which is used to display the target information in front of the driver based on the light field naked eye 3D display technology to remind the driver.
[0040] Specifically, when the target information is a warning information in the form of an image. The display unit 30 can render and process the received target information based on the light field naked eye 3D display technology to display the target information in front of the driver, thereby reminding the driver. Among them, the light field naked eye 3D display technology is closer to the natural visual perception of humans, so the driver does not need to perform additional visual conversion or adjustment when viewing the information, which can reduce the driver's eye fatigue and distraction when viewing the information. At the same time, the light field naked eye 3D display technology is applied to the vehicle's head-up display system (Head Up Display, HUD) in combination, which can bring many benefits to the driver, including more intuitive information presentation, enhanced spatial perception, improved clarity of navigation and route instructions, enhanced driving experience, and reduced driver fatigue and distraction, etc., further helping to improve the driver's driving safety, comfort and fun.
[0041] It is understandable that, in addition to warning information in the form of images, target information may also include warning information in the form of sound and warning lights. That is, in addition to displaying the target information through the display unit 30, it may also be output simultaneously in the form of sound, warning lights, etc. based on the target information to achieve the target reminder effect. For example, when the driver is in a fatigue state, the fatigue reminder information output by the data processing unit 20 is output to the display unit 30 on the one hand, and the display unit 30 outputs the corresponding display image in front of the driver based on the received fatigue reminder information, thereby providing a visual reminder to the driver; on the other hand, it is output to the reminder speaker, so that the reminder speaker outputs the corresponding refreshing music or increases the sound of the music being played based on the received fatigue reminder information, etc., so as to provide a sound reminder to the driver.
[0042] Combination Figure 3 As shown, in one embodiment of the present invention, the driver monitoring system also includes: a second acquisition unit 40, used to obtain a second monitoring video of the driver based on infrared radiation technology; the data processing unit 20 is also used to determine the current state of the driver based on the first monitoring video and the second monitoring video.
[0043] Specifically, the second acquisition unit 40 can perform non-interference remote detection of a person's pupil position, eyelid movement and related driver alertness indicators outside the range of human visual perception based on infrared radiation technology, thereby obtaining a second monitoring video. In combination with the first monitoring video collected by light field sensing technology, more comprehensive driver facial information can be obtained, further improving the quality of driver status monitoring.
[0044] Combination Figure 4As shown, in one embodiment of the present invention, the first acquisition unit 10 includes a light field camera 11, which is used to acquire a first monitoring video; the second acquisition unit 40 includes an infrared camera 41, which is used to acquire a second monitoring video.
[0045] Specifically, the light field camera 11 can obtain the depth information of the scene with only one imaging. The light field camera 11 adopts a large aperture and a large depth of field, and can collect light from multiple angles at the same position, so as to obtain better image effects and a larger depth range under low light conditions. In addition, the light field camera 11 has the ability to shoot first, focus later, and remove obstructions, providing more flexibility for the driver monitoring system.
[0046] The infrared camera 41 can perform non-interference remote detection of a person's pupil position, eyelid movement and related driver alertness indicators outside the range of human visual perception. In conjunction with the light field camera 11, more comprehensive driver facial information can be obtained.
[0047] In one embodiment of the present invention, the data processing unit 20 includes: a decoding module 21, which is used to decode the first monitoring video and the second monitoring video to generate target image data; a preprocessing module 22, which is used to preprocess the target image data; a feature extraction and fusion module 23, which is used to extract and fuse the preprocessed target image data based on a deep learning model to generate target feature information; a feature classification and detection module 24, which is used to classify and detect the target feature information and generate a state detection result, wherein the state detection result is used to characterize the current state of the driver; a control module 25, which is used to generate target information according to the state detection result.
[0048] Specifically, the driver's facial monitoring video continuously captured by the light field camera 11 and the infrared camera 41 is processed by the data processing unit 20 .
[0049] Since the video data acquired by the light field camera 11 and the infrared camera 41 are usually stored in a specific file format, they need to be decoded into a format that can be processed later. First, the first monitoring video and the second monitoring video are decoded by the decoding module 21. Specifically, the decoding module 21 decodes the first monitoring video to output a light field image, and decodes the second monitoring video to generate an infrared image. For example, the light field camera 11 records the light field information of the scene by capturing the propagation direction of light in space. The acquired light field data is usually stored in the form of a light field file, such as lfp format. Then, during the video decoding process, the light field data will be decoded into subviews for storage, and the subviews record data at different angles and directions. This acquisition technology can provide rich light information and depth information, thereby obtaining a more realistic and three-dimensional image of the driver's face. This helps the system to more accurately identify the driver's facial expressions, eyes, head posture, etc., and provide more reliable data for subsequent driver status analysis and judgment.
[0050] The preprocessing module 22 performs denoising, contrast enhancement, image super-resolution, normalization and other steps on the light field image and infrared image output by the decoding module 21. The light field image may also include denoising, white balance adjustment, exposure compensation and other processing processes.
[0051] The feature extraction and fusion module 23 is based on a loaded and trained deep learning model, such as a convolutional neural network (CNN) or a recurrent neural network (RNN), which can process image and video data and extract useful features therefrom, thereby automatically extracting features from the target image data after preprocessing. At this stage, the feature extraction and fusion module 23 can obtain meaningful features from the target image data to determine whether the driver has closed his eyes, lowered his head, made a phone call, smoked, looked around, and other actions, and fuse the features extracted from the video frames obtained by the infrared camera 41 and the light field camera 11 to obtain richer information and generate target feature information.
[0052] The feature classification and detection module 24 can classify and detect the target feature information to determine the current state of the driver and generate a state detection result. Specifically, the target feature information can be classified by a classifier, so that the driver's state can be classified into different categories according to the extracted and fused features, such as smoking, fatigue, making a phone call, normal, etc.
[0053] The control module 25 receives the status detection result and generates corresponding target information to the display module 30, so as to display the target information in front of the driver based on the light field naked-eye 3D display technology to remind the driver and improve driving safety.
[0054] In one embodiment of the present invention, the control module 25 is also connected to the first acquisition unit 10 , the second acquisition unit 40 and the decoding module 21 respectively, for receiving the first monitoring video and the second monitoring video, and sending the first monitoring video and the second monitoring video to the decoding module 21 .
[0055] That is to say, in addition to the decoding module 21 being directly connected to the first acquisition unit 10 and the second acquisition unit 40 to obtain the first monitoring video and the second monitoring video, the first monitoring video and the second monitoring video can also be forwarded through the control module 25, that is, the control module 25 obtains the video stream from the first acquisition unit 10 and the second acquisition unit 40, and sends it to the decoding module 21, that is, the video decoder for decoding.
[0056] In one embodiment of the present invention, the control module 25 is also connected to the preprocessing module 22 and the feature extraction and fusion module 23 respectively, and is used to respond to the notification instruction and generate the target work instruction; wherein the preprocessing module 22 is also used to generate the notification instruction when the preprocessing of the target image data is completed; the feature extraction and fusion module 23 is also used to perform feature extraction and fusion on the preprocessed target image data based on the target work instruction.
[0057] That is, when the target image data preprocessing is completed, the preprocessing module 23 generates a notification instruction to notify the control module 25, and the control module 25 gives a feature extraction and fusion instruction, i.e., a target working instruction. The feature extraction and fusion module 23 performs feature extraction and fusion processing through a deep learning model based on the target working instruction.
[0058] In one embodiment of the present invention, the data processing unit 20 further includes: a data buffer module 26, which is connected to the decoding module 21 and the pre-processing module 22 respectively, and is used to store the target image data.
[0059] That is, the target image data decoded by the decoding module 21 is stored in the data buffer module 26, and the pre-processing module 22 obtains the target image data from the data buffer module 26 and performs image processing. Thus, decoding and pre-processing can work in parallel, the decoding module 21 is responsible for decoding new video frames, and the pre-processing module 22 processes the target image data that has been decoded in the data buffer module 26, thereby improving efficiency.
[0060] In one embodiment of the present invention, the display unit 30 includes: a HUD controller 31, which is used to establish a HUD display view corresponding to the target information based on the light field naked-eye 3D display technology, and render the HUD display view to generate a target display view; a HUD processor 32, which is used to process the target display view based on a preset light field texture, so as to display the processed target display view based on the vehicle's HUD display.
[0061] That is to say, the HUD controller 31 receives the target information from the data processing unit 20, processes and renders the content to be displayed on the HUD display, generates a target display view, and then the HUD processor 32 is responsible for sending the generated target display view to the HUD display for display.
[0062] Specifically, after receiving the target information from the data processing unit 20, the HUD controller 31 displays the target information on the HUD with the help of the light field naked eye 3D display technology. Then, the light field data (including multi-view information and depth information) of the pre-calculated target information is rendered. The HUD controller 31 creates a HUD display view and sets the vertex shader, fragment shader and rendering parameters in the OpenGL rendering pipeline. Among them, the fragment shader is used to process the rendering of the light field naked eye 3D data. The fragment shader reads the image data and depth information of the multi-viewpoints and synthesizes the final image based on this information. Texture sampling is required to access the image data of the multi-viewpoints, and the depth information is used to determine how to synthesize these images. Once the light field data is rendered in the HUD controller 31, the rendering result can be stored in the frame buffer object (FBO), and then the FBO is passed to the HUD processor 32 as a texture. Of course, the HUD controller can receive data from different sources, which may include sensor information, vehicle status, user input, etc.
[0063] The HUD processor 32 applies the received light field texture to the display area of the HUD. The rendered light field image will be an image with a sense of depth. Compared with the traditional 2D display, the light field naked eye 3D display technology can reduce the driver's eye fatigue and distraction when viewing information, and can also achieve more intuitive information presentation, enhanced spatial perception, improved clarity of navigation and route instructions, enhanced driving experience and other effects, thereby improving the driver's driving safety, comfort and fun.
[0064] As a specific embodiment of the present invention, Figure 4 As shown, the driver monitoring system includes an acquisition device (a first acquisition unit 10 and a second acquisition unit 40), a data processing unit 20 and a display unit 30. The first acquisition unit 10 includes a light field camera 11, and the second acquisition unit 40 includes an infrared camera 41. The data processing unit 20 is composed of six parts: a control module 25, a decoding module 21, a data buffer module 26, a preprocessing module 22, a feature extraction and fusion module 23, and a feature classification and detection module 24.
[0065] The light field camera 11 and the infrared camera 41 are deployed at the vehicle side to collect the driver's facial video in real time. The control module 25 communicates with the light field camera 11 and the infrared camera 41 interface, cyclically captures the video stream from the light field camera 11 and the infrared camera 41, waits for the data processing unit 20 to complete the processing, and uses the light field naked-eye 3D display technology to display a warning on the HUD display to remind the driver and improve driving safety.
[0066] The deep learning model used in the data processing unit 20 needs to collect a large amount of data obtained by the light field camera 11 and the infrared camera 41 in advance for model training. Among them, the light field image data mainly consists of the original data of the light field, including subviews and depth images. The infrared image data includes the collection information of various scenes such as day and night. These data should include various behaviors of the driver, such as normal driving, fatigue driving, making phone calls, smoking, etc. Then, these data are preprocessed, including extraction, normalization, and annotation of video frames, so that the model can recognize and understand them. For each image, the driver's behavior needs to be annotated, such as normal driving, fatigue driving, etc., which can be done using professional annotation tools.
[0067] In addition, a suitable deep learning model, such as a convolutional neural network or a recurrent neural network, needs to be selected so that the data processing unit 20 can process image and video data and extract useful features from them. During the model training process, the model will learn how to judge the driver's behavior based on the input video frames. This process may require a lot of computing resources and time. Use a test data set to evaluate the performance of the model, such as accuracy, recall, etc. If the performance of the model is not good, the model can be optimized by adjusting the model parameters, changing the model structure, or adding more training data. When the performance of the model meets the requirements, select a deep learning framework suitable for the vehicle side and deploy the environment.
[0068] The driver monitoring system proposed in this embodiment has the following advantages:
[0069] 1) Light field sensing technology can capture richer information, improve the accuracy of the driver monitoring system, and monitor the driver's status more accurately;
[0070] 2) Light field sensing technology can use fewer camera modules to obtain richer information, thus reducing the amount of wiring and reducing costs for automakers;
[0071] 3) By using deep learning models, driver monitoring systems can be continuously optimized and improved to adapt to different driving environments and driver needs;
[0072] 4) Using light field naked-eye 3D to display prompt information on the HUD can convey key information to the driver more intuitively and reduce driver distraction.
[0073] In summary, according to the driver monitoring system of the embodiment of the present invention, the first acquisition unit obtains the first monitoring video of the driver based on the light field sensing technology, the data processing unit determines the current state of the driver based on the first monitoring video, and generates target information according to the current state of the driver, so as to remind the driver based on the target information. Therefore, the monitoring system uses the first monitoring video obtained based on the light field sensing technology as input to analyze and monitor the driver's state, thereby improving the richness of the input information used for state analysis, and thus improving the accuracy of driver state monitoring.
[0074] Corresponding to the above embodiment, the present invention also provides a vehicle.
[0075] like Figure 5 As shown, the vehicle 100 according to the embodiment of the present invention includes the above-mentioned driver monitoring system 110 .
[0076] According to the vehicle of the embodiment of the present invention, based on the above-mentioned driver monitoring system, the driver's behavior is captured based on light field sensing technology to analyze the driver's state, realize real-time monitoring of the driver's state, improve driving safety and reduce the occurrence of traffic accidents.
[0077] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0078] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0079] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0080] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.
Claims
1. A driver monitoring system, characterized in that: Applied to a vehicle, the system comprises: A first acquisition unit, configured to acquire a first monitoring video of the driver based on light field perception technology; A data processing unit is used to determine the current state of the driver based on the first monitoring video, and generate target information according to the current state of the driver, so as to remind the driver based on the target information.
2. The driver monitoring system according to claim 1, characterized in that: Also includes: A display unit is used to display the target information in front of the driver based on the light field naked eye 3D display technology.
3. The driver monitoring system according to claim 1 or 2, characterized in that: Also includes: A second acquisition unit, used for acquiring a second monitoring video of the driver based on infrared radiation technology; The data processing unit is further used to determine the current state of the driver based on the first monitoring video and the second monitoring video.
4. The driver monitoring system according to claim 3, characterized in that: The first acquisition unit includes a light field camera, and the light field camera is used to acquire the first monitoring video; The second acquisition unit includes an infrared camera, and the infrared camera is used to acquire the second monitoring video.
5. The driver monitoring system according to claim 3, characterized in that: The data processing unit comprises: A decoding module, used for decoding the first surveillance video and the second surveillance video to generate target image data; A preprocessing module, used for preprocessing the target image data; The feature extraction and fusion module is used to extract and fuse the preprocessed target image data based on the deep learning model to generate target feature information; A feature classification and detection module, used for performing feature classification and detection on the target feature information and generating a state detection result, wherein the state detection result is used to characterize the current state of the driver; A control module is used to generate the target information according to the state detection result.
6. The driver monitoring system according to claim 5, characterized in that: The control module is also connected to the first acquisition unit, the second acquisition unit and the decoding module respectively, and is used to receive the first monitoring video and the second monitoring video, and send the first monitoring video and the second monitoring video to the decoding module.
7. The driver monitoring system according to claim 5, characterized in that: The control module is also connected to the preprocessing module and the feature extraction and fusion module respectively, and is used to respond to the notification instruction and generate the target work instruction; wherein, The preprocessing module is also used to generate the notification instruction when the preprocessing of the target image data is completed; The feature extraction and fusion module is also used to extract and fuse features of the preprocessed target image data based on the target work instruction.
8. The driver monitoring system according to claim 5, characterized in that: The data processing unit also includes: A data buffer module, the data buffer module is connected to the decoding module and the pre-processing module respectively, and is used to store the target image data.
9. The driver monitoring system according to claim 2, characterized in that: The display unit comprises: A HUD control module, used for establishing a HUD display view corresponding to the target information based on the light field naked eye 3D display technology, and rendering the HUD display view to generate a target display view; The HUD processing module is used to process the target display view based on a preset light field texture, so as to display the processed target display view based on the HUD display of the vehicle.
10. A vehicle, characterized in that: Comprising a driver monitoring system according to any one of claims 1-9.
Citation Information
Patent Citations
Real-time video stream fusion processing method and system based on light field camera
CN114173106A
Method for detecting fatigue driving of automobile driver based on neural network
CN115830581A
Driver state monitoring method, device, equipment and medium
CN116740684A
Real-time updating system for retaining wall of unmanned refuse dump of surface mine
CN117948990A
Self-driving-oriented human-machine collaborative perception method and system
WO2022095440A1
