Driver monitoring system and vehicle

Abstract

This invention discloses a driver monitoring system and a vehicle. The driver monitoring system, applied to a vehicle, includes: a first acquisition unit for acquiring a first monitoring video of the driver based on light field sensing technology; and a data processing unit for determining the driver's current state based on the first monitoring video and generating target information based on the driver's current state to provide reminders to the driver. Thus, this system uses the first monitoring video acquired based on light field sensing technology as input to analyze and monitor the driver's state, improving the richness of the input information used for state analysis and thereby enhancing the accuracy of driver state monitoring.

Description

Technical Field

[0001] This invention relates to the field of driver monitoring technology, and more particularly to a driver monitoring system and a vehicle. Background Technology

[0002] A driver monitoring system (DMS) primarily uses cameras to monitor the driver's condition, identify the driver, and determine whether the driver is driving while fatigued or engaging in other dangerous behaviors, thereby ensuring driving safety.

[0003] In related technologies, facial and eye information is analyzed and processed based on two-dimensional images of the driver captured by a camera to determine the driver's state and monitor it. However, this technical solution has low monitoring accuracy, which reduces the effectiveness of state monitoring. Summary of the Invention

[0004] This invention aims to at least partially solve one of the technical problems in related technologies. Therefore, the first objective of this invention is to propose a driver monitoring system that uses a first monitoring video acquired based on light field sensing technology as input to analyze and monitor the driver's state, thereby increasing the richness of the input information used for state analysis and thus improving the accuracy of driver state monitoring.

[0005] The second objective of this invention is to provide a vehicle.

[0006] To achieve the above objectives, a first aspect of the present invention provides a driver monitoring system applied to a vehicle. The system includes: a first acquisition unit for acquiring a first monitoring video of the driver based on light field sensing technology; and a data processing unit for determining the driver's current state based on the first monitoring video and generating target information based on the driver's current state, so as to remind the driver based on the target information.

[0007] According to an embodiment of the driver monitoring system of the present invention, a first acquisition unit acquires a first monitoring video of the driver based on light field sensing technology, a data processing unit determines the driver's current state based on the first monitoring video, and generates target information based on the driver's current state in order to remind the driver based on the target information. Thus, this monitoring system uses the first monitoring video acquired based on light field sensing technology as input to analyze and monitor the driver's state, improving the richness of the input information used for state analysis and thereby enhancing the accuracy of driver state monitoring.

[0008] In addition, the driver monitoring system according to the above embodiments of the present invention may also have the following additional technical features:

[0009] According to one embodiment of the present invention, the driver monitoring system further includes a display unit for displaying target information in front of the driver based on light field naked-eye 3D display technology.

[0010] According to one embodiment of the present invention, the driver monitoring system further includes: a second acquisition unit, configured to acquire a second monitoring video of the driver based on infrared radiation technology; and a data processing unit, configured to determine the driver's current state based on the first monitoring video and the second monitoring video.

[0011] According to one embodiment of the present invention, the first acquisition unit includes a light field camera for acquiring a first monitoring video; the second acquisition unit includes an infrared camera for acquiring a second monitoring video.

[0012] According to an embodiment of the present invention, the data processing unit includes: a decoding module for decoding a first surveillance video and a second surveillance video to generate target image data; a preprocessing module for preprocessing the target image data; a feature extraction and fusion module for extracting and fusing features from the preprocessed target image data based on a deep learning model to generate target feature information; a feature classification and detection module for classifying and detecting the target feature information and generating a state detection result, wherein the state detection result is used to characterize the current state of the driver; and a control module for generating target information based on the state detection result.

[0013] According to one embodiment of the present invention, the control module is also connected to the first acquisition unit, the second acquisition unit and the decoding module respectively, and is used to receive the first monitoring video and the second monitoring video, and send the first monitoring video and the second monitoring video to the decoding module.

[0014] According to one embodiment of the present invention, the control module is further connected to the preprocessing module and the feature extraction and fusion module respectively, and is used to generate a target working instruction in response to a notification instruction; wherein, the preprocessing module is further used to generate a notification instruction when the preprocessing of the target image data is completed; the feature extraction and fusion module is further used to perform feature extraction and fusion on the preprocessed target image data based on the target working instruction.

[0015] According to one embodiment of the present invention, the data processing unit further includes a data buffer module, which is connected to the decoding module and the preprocessing module respectively, and is used to store the target image data.

[0016] According to one embodiment of the present invention, the display unit includes: a HUD controller, configured to establish a HUD display view corresponding to target information based on light field naked-eye 3D display technology, and to render the HUD display view to generate a target display view; and a HUD processor, configured to process the target display view based on a preset light field texture so that the vehicle-based HUD display can display the processed target display view.

[0017] To achieve the above objectives, a second aspect of the present invention provides a vehicle including the driver monitoring system described above.

[0018] According to the vehicle of the present invention, based on the driver monitoring system described above, the driver's behavior is captured based on light field perception technology to analyze the driver's state, thereby achieving real-time monitoring of the driver's state, improving driving safety and reducing the occurrence of traffic accidents.

[0019] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0020] Figure 1 This is a connection diagram of a driver monitoring system according to an embodiment of the present invention;

[0021] Figure 2 This is a connection diagram of a driver monitoring system according to an embodiment of the present invention;

[0022] Figure 3 This is a connection diagram of a driver monitoring system according to another embodiment of the present invention;

[0023] Figure 4 This is a connection diagram of a driver monitoring system according to a specific embodiment of the present invention;

[0024] Figure 5 This is a block diagram of a vehicle according to an embodiment of the present invention. Detailed Implementation

[0025] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0026] The driver monitoring system and vehicle proposed in the embodiments of the present invention are described below with reference to the accompanying drawings.

[0027] In related technologies, driver status acquisition schemes for DMS (Driver Status Management System) are mainly divided into two categories:

[0028] The first type uses an RGB camera to capture a two-dimensional planar image of the face, combined with an infrared LED (Light-Emitting Diode) and corresponding computer vision algorithms, to accurately identify the driver's face under various lighting conditions.

[0029] The second type utilizes binocular recognition technology, 3D structured light, and TOF (Time of Flight) sensor technology to generate a "depth map" of the driver's head, thereby enabling accurate analysis of the driver's facial features.

[0030] While the second type of technical solution can provide depth information, it is costly and the sensor technology is not very mature, making it difficult to guarantee the quality of status monitoring. Therefore, most related technologies still rely on analyzing and processing facial and eye information based on two-dimensional images captured by cameras. The system collects less data when processing images, reducing the accuracy of status monitoring.

[0031] To address this, this application proposes a driver monitoring system that uses light field sensing technology to capture images of the driver, obtaining a first monitoring video. This first monitoring video is then used as input for real-time analysis and monitoring of the driver's state. Light field sensing technology can acquire multi-dimensional information about light rays in a single imaging process, achieving three-dimensional imaging. Simultaneously, it can record the direction and angle information of the light rays, and through post-processing focusing, obtain images focused on different planes. Furthermore, light field sensing technology can utilize the multi-dimensional information of light rays to achieve occlusion removal and depth estimation, significantly improving the richness of the input information used for state analysis and enhancing the accuracy of driver state monitoring.

[0032] Figure 1 This is a connection diagram of a driver monitoring system according to an embodiment of the present invention.

[0033] like Figure 1 As shown, the driver monitoring system of this embodiment of the invention is applied to a vehicle and may include: a first acquisition unit 10 and a data processing unit 20.

[0034] The first acquisition unit 10 is used to acquire a first monitoring video of the driver based on light field sensing technology. The data processing unit 20 is used to determine the driver's current state based on the first monitoring video and generate target information based on the driver's current state, so as to remind the driver based on the target information.

[0035] Specifically, the first acquisition unit 10, based on light field sensing technology, can obtain multi-dimensional information of light through a single imaging process to achieve three-dimensional imaging and acquire the corresponding first monitoring video, which can be a video of the driver's face. Furthermore, light field sensing technology can record the direction and angle information of light rays, and after post-focusing processing, obtain images focused on different planes. Additionally, light field sensing technology can utilize the multi-dimensional information of light rays to achieve occlusion removal and depth estimation.

[0036] The data processing unit 20 processes the first monitoring video to obtain high-quality monitoring data, thereby determining the driver's current state and improving the accuracy of state monitoring. The driver's state can be categorized based on actual conditions, such as smoking, fatigue, making a phone call, or normal status. After determining the driver's current state, the data processing unit 20 generates target information based on that state to provide reminders to the driver, thus improving driving safety.

[0037] The target information is a reminder message corresponding to the driving state. For example, when the driver is fatigued, the target information is a fatigue alert message; when the driver is on the phone, the target information is a "Do Not Make Phone Calls" reminder message. Furthermore, the target information can be displayed in the form of sound, light, or images, without specific limitations. For example, when the driver is fatigued, the target information could be a visual fatigue alert message displayed on a screen or other component; or it could be an audible fatigue reminder message, specifically a corresponding reminder music or volume adjustment information, implemented by controlling the vehicle's speakers; or it could be a warning light fatigue reminder message, specifically by illuminating warning lights on the dashboard or other locations to remind the driver.

[0038] This driver monitoring system can be applied in the field of intelligent assisted driving. It captures the driver's behavior through light field perception technology, thereby analyzing the driver's state to achieve real-time monitoring of the driver's state, thereby improving driving safety and reducing the occurrence of traffic accidents.

[0039] Combination Figure 2 As shown, in one embodiment of the present invention, the driver monitoring system further includes a display unit 30, which is used to display target information in front of the driver based on light field naked-eye 3D display technology to remind the driver.

[0040] Specifically, when the target information is a warning message in image form, the display unit 30 can perform rendering and other operations on the received target information based on light field naked-eye 3D display technology to display the target information in front of the driver, thereby reminding the driver. Light field naked-eye 3D display technology is closer to the natural visual perception of humans, so the driver does not need to make additional visual conversions or adjustments when viewing information, reducing eye fatigue and distraction. Furthermore, the application of light field naked-eye 3D display technology in conjunction with the vehicle's head-up display (HUD) system brings numerous benefits to the driver, including more intuitive information presentation, enhanced spatial perception, improved clarity of navigation and route instructions, enhanced driving experience, and reduced driver fatigue and distraction, further contributing to improved driving safety, comfort, and enjoyment.

[0041] It is understandable that the target information may include not only visual warnings but also audible and visual warnings. In other words, besides displaying the target information through the display unit 30, it can also be output simultaneously in the form of sound and warning lights to achieve a target reminder effect. For example, when the driver is fatigued, the fatigue reminder information output by the data processing unit 20 is sent to the display unit 30. Based on the received fatigue reminder information, the display unit 30 outputs a corresponding image in front of the driver, thus providing a visual reminder. Simultaneously, it is sent to the reminder speaker, which, based on the received fatigue reminder information, outputs corresponding energizing music or increases the volume of currently playing music to provide an auditory reminder to the driver.

[0042] Combination Figure 3 As shown, in one embodiment of the present invention, the driver monitoring system further includes: a second acquisition unit 40, used to acquire a second monitoring video of the driver based on infrared radiation technology; and a data processing unit 20, used to determine the driver's current state based on the first monitoring video and the second monitoring video.

[0043] Specifically, the second acquisition unit 40, based on infrared radiation technology, can remotely detect the position of a person's pupils, eyelid movements, and related driver alertness indicators without interference, outside the range of human visual perception, thereby acquiring a second monitoring video. Combined with the first monitoring video acquired by light field perception technology, it can obtain more comprehensive driver facial information, further improving the quality of driver status monitoring.

[0044] Combination Figure 4As shown, in one embodiment of the present invention, the first acquisition unit 10 includes a light field camera 11, which is used to acquire a first monitoring video; the second acquisition unit 40 includes an infrared camera 41, which is used to acquire a second monitoring video.

[0045] Specifically, the light field camera 11 can acquire the depth information of a scene with just one image. Employing a large aperture and large depth of field, the light field camera 11 can collect light from multiple angles at the same location, thus achieving better image quality and a wider depth range in low-light conditions. Furthermore, the light field camera 11 has the ability to focus after shooting and remove occlusions, providing greater flexibility for driver monitoring systems.

[0046] Infrared camera 41 can remotely detect the position of a person's pupils, eyelid movements and related driver alertness indicators without interference, outside the range of human visual perception. When used in conjunction with light field camera 11, it can obtain more comprehensive driver facial information.

[0047] In one embodiment of the present invention, the data processing unit 20 includes: a decoding module 21, used to decode the first monitoring video and the second monitoring video to generate target image data; a preprocessing module 22, used to preprocess the target image data; a feature extraction and fusion module 23, used to extract and fuse features from the preprocessed target image data based on a deep learning model to generate target feature information; a feature classification and detection module 24, used to classify and detect the target feature information and generate a state detection result, wherein the state detection result is used to characterize the current state of the driver; and a control module 25, used to generate target information based on the state detection result.

[0048] Specifically, the driver's facial monitoring video continuously captured by the light field camera 11 and the infrared camera 41 is processed by the data processing unit 20.

[0049] Since the video data acquired by the light field camera 11 and the infrared camera 41 is usually stored in a specific file format, it needs to be decoded into a format that can be processed later. First, the decoding module 21 decodes the first and second monitoring videos. Specifically, the decoding module 21 decodes the first monitoring video to output a light field image and decodes the second monitoring video to generate an infrared image. For example, the light field camera 11 records the light field information of a scene by capturing the direction of light propagation in space. The acquired light field data is usually stored in the form of a light field file, such as .lfp format. During video decoding, the light field data will be decoded into sub-views for storage, recording data from different angles and directions. This acquisition technology can provide rich light and depth information, thereby obtaining a more realistic and three-dimensional image of the driver's face. This helps the system more accurately identify the driver's facial expressions, eye contact, and head posture, providing more reliable data for subsequent driver state analysis and judgment.

[0050] The preprocessing module 22 processes the light field image and infrared image output by the decoding module 21 through steps such as denoising, contrast enhancement, image super-resolution, and normalization. The light field image may also include processing steps such as denoising, white balance adjustment, and exposure compensation.

[0051] The feature extraction and fusion module 23 is based on a pre-trained deep learning model, such as a convolutional neural network (CNN) or a recurrent neural network (RNN). These models can process image and video data and extract useful features from them, thereby automatically extracting features from the preprocessed target image data. In this stage, the feature extraction and fusion module 23 can obtain meaningful features from the target image data to determine whether the driver is closing their eyes, looking down, making a phone call, smoking, or looking around. It then fuses the features extracted from the video frames obtained by the infrared camera 41 and the light field camera 11 to obtain richer information and generate target feature information.

[0052] The feature classification and detection module 24 can classify and detect target feature information to determine the driver's current state and generate a state detection result. Specifically, the target feature information can be classified by a classifier, thereby classifying the driver's state into different categories based on the extracted and fused features, such as smoking, fatigue, making a phone call, or normal.

[0053] The control module 25 receives the status detection results and generates corresponding target information for the display module 30. The target information is then displayed in front of the driver based on light field naked-eye 3D display technology to remind the driver and improve driving safety.

[0054] In one embodiment of the present invention, the control module 25 is also connected to the first acquisition unit 10, the second acquisition unit 40 and the decoding module 21 respectively, for receiving the first monitoring video and the second monitoring video, and sending the first monitoring video and the second monitoring video to the decoding module 21.

[0055] In other words, besides the decoding module 21 being directly connected to the first acquisition unit 10 and the second acquisition unit 40 to acquire the first and second monitoring videos, the first and second monitoring videos can also be forwarded through the control module 25. That is, the control module 25 acquires the video streams from the first acquisition unit 10 and the second acquisition unit 40 and sends them to the decoding module 21, i.e., the video decoder, for decoding.

[0056] In one embodiment of the present invention, the control module 25 is also connected to the preprocessing module 22 and the feature extraction and fusion module 23 respectively, and is used to generate a target working instruction in response to a notification instruction; wherein, the preprocessing module 22 is also used to generate a notification instruction when the preprocessing of the target image data is completed; the feature extraction and fusion module 23 is also used to perform feature extraction and fusion on the preprocessed target image data based on the target working instruction.

[0057] In other words, when the preprocessing module 23 completes the preprocessing of the target image data, it generates a notification instruction to notify the control module 25. At this time, the control module 25 issues a feature extraction and fusion instruction, i.e., a target working instruction. The feature extraction and fusion module 23 performs feature extraction and fusion processing based on the target working instruction using a deep learning model.

[0058] In one embodiment of the present invention, the data processing unit 20 further includes a data buffer module 26, which is connected to the decoding module 21 and the preprocessing module 22 respectively, and is used to store the target image data.

[0059] In other words, the target image data decoded by the decoding module 21 is stored in the data buffer module 26, and the preprocessing module 22 retrieves the target image data from the data buffer module 26 for image processing. Thus, decoding and preprocessing can work in parallel; the decoding module 21 is responsible for decoding new video frames, while the preprocessing module 22 processes the already decoded target image data in the data buffer module 26, thereby improving efficiency.

[0060] In one embodiment of the present invention, the display unit 30 includes: a HUD controller 31, configured to establish a HUD display view corresponding to the target information based on light field naked-eye 3D display technology, and to render the HUD display view to generate a target display view; and a HUD processor 32, configured to process the target display view based on a preset light field texture so that the vehicle-based HUD display can display the processed target display view.

[0061] In other words, the HUD controller 31 receives the target information from the data processing unit 20, processes and renders the content to be displayed on the HUD display, generates the target display view, and then the HUD processor 32 is responsible for sending the generated target display view to the HUD display for display.

[0062] Specifically, after receiving target information from the data processing unit 20, the HUD controller 31 displays the target information on the HUD using light field naked-eye 3D display technology. Then, it renders the pre-calculated light field data (including multi-view information and depth information) of the target information. The HUD controller 31 creates the HUD display view and sets the vertex shader, fragment shader, and rendering parameters in the OpenGL rendering pipeline. The fragment shader is used to process the rendering of the light field naked-eye 3D data. The fragment shader reads the multi-view image data and depth information and synthesizes the final image based on this information. Texture sampling is used to access the multi-view image data, and depth information is used to determine how to synthesize these images. Once the light field data is rendered in the HUD controller 31, the rendering result can be stored in a frame buffer object (FBO), and then the FBO can be passed as a texture to the HUD processor 32. Of course, the HUD controller can receive data from different sources, which may include sensor information, vehicle status, user input, etc.

[0063] The HUD processor 32 applies the received light field texture to the HUD display area. The rendered light field image will be an image with a sense of depth. Compared with traditional 2D display, light field naked-eye 3D display technology can reduce eye fatigue and distraction for drivers when viewing information. At the same time, it can achieve more intuitive information presentation, enhanced spatial perception, improved clarity of navigation and route instructions, and enhanced driving experience, thereby improving the driver's driving safety, comfort, and enjoyment.

[0064] As a specific embodiment of the present invention, such as Figure 4 As shown, the driver monitoring system includes acquisition devices (first acquisition unit 10 and second acquisition unit 40), a data processing unit 20, and a display unit 30. The first acquisition unit 10 includes a light field camera 11, and the second acquisition unit 40 includes an infrared camera 41. The data processing unit 20 consists of six parts: a control module 25, a decoding module 21, a data buffer module 26, a preprocessing module 22, a feature extraction and fusion module 23, and a feature classification and detection module 24.

[0065] The light field camera 11 and infrared camera 41 are deployed on the vehicle's infotainment system to capture real-time video of the driver's face. The control module 25 communicates with the light field camera 11 and infrared camera 41 through an interface, continuously capturing video streams from the light field camera 11 and infrared camera 41, waiting for the data processing unit 20 to complete the processing, and then displaying warnings on the HUD display with the help of light field naked-eye 3D display technology to remind the driver and improve driving safety.

[0066] The deep learning model used in the data processing unit 20 requires the prior collection of a large amount of data from the light field camera 11 and the infrared camera 41 for model training. The light field image data mainly consists of raw light field data, including sub-views and depth images. The infrared image data includes information collected in various scenes, such as daytime and nighttime. This data should contain various driver behaviors, such as normal driving, fatigued driving, making phone calls, and smoking. Then, this data undergoes preprocessing, including video frame extraction, normalization, and annotation, so that the model can recognize and understand it. For each image, the driver's behavior, such as normal driving or fatigued driving, needs to be labeled, which can be done using specialized annotation tools.

[0067] Additionally, a suitable deep learning model, such as a convolutional neural network or a recurrent neural network, needs to be selected so that the data processing unit 20 can process image and video data and extract useful features from it. During model training, the model learns how to judge the driver's behavior based on the input video frames. This process may require significant computational resources and time. The model's performance, such as accuracy and recall, is evaluated using a test dataset. If the model's performance is unsatisfactory, it can be optimized by adjusting the model's parameters, changing the model structure, or adding more training data. Once the model's performance meets the requirements, a suitable deep learning framework for the vehicle's infotainment system is selected and deployed in the environment.

[0068] The driver monitoring system proposed in this embodiment has the following advantages:

[0069] 1) Light field sensing technology can capture richer information, improve the accuracy of driver monitoring systems, and more accurately monitor driver status;

[0070] 2) Light field sensing technology can acquire more information with fewer camera modules, thus reducing wiring and lowering costs for automakers;

[0071] 3) By using deep learning models, the driver monitoring system can be continuously optimized and improved to adapt to different driving environments and driver needs;

[0072] 4) Using light field naked-eye 3D display on the HUD to display prompts can convey key information to the driver more intuitively and reduce driver distraction.

[0073] In summary, the driver monitoring system according to embodiments of the present invention acquires a first monitoring video of the driver using light field sensing technology through a first acquisition unit, determines the driver's current state based on the first monitoring video, and generates target information based on the driver's current state to provide reminders to the driver. Therefore, this monitoring system uses the first monitoring video acquired based on light field sensing technology as input to analyze and monitor the driver's state, improving the richness of the input information used for state analysis and thus enhancing the accuracy of driver state monitoring.

[0074] Corresponding to the above embodiments, the present invention also proposes a vehicle.

[0075] like Figure 5 As shown, the vehicle 100 of this embodiment includes the driver monitoring system 110 described above.

[0076] According to the vehicle of the present invention, based on the driver monitoring system described above, the driver's behavior is captured based on light field perception technology to analyze the driver's state, thereby achieving real-time monitoring of the driver's state, improving driving safety and reducing the occurrence of traffic accidents.

[0077] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0078] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0079] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0080] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A driver monitoring system, characterized in that, Applied to vehicles, the system includes: The first acquisition unit is used to acquire the first monitoring video of the driver based on light field sensing technology; The data processing unit is configured to determine the driver's current state based on the first surveillance video, and generate target information based on the driver's current state, so as to provide a reminder to the driver based on the target information; wherein, The system further includes: a second acquisition unit, used to acquire a second monitoring video of the driver based on infrared radiation technology; The data processing unit is further configured to determine the current state of the driver based on the first surveillance video and the second surveillance video; The first acquisition unit includes a light field camera, which is used to acquire the first monitoring video; The second acquisition unit includes an infrared camera, which is used to acquire the second surveillance video; The data processing unit includes: The decoding module is used to decode the first surveillance video and the second surveillance video to generate target image data; The preprocessing module is used to preprocess the target image data; The feature extraction and fusion module is used to extract and fuse features from preprocessed target image data based on a deep learning model to generate target feature information. The feature classification and detection module is used to classify and detect the target feature information and generate a state detection result, wherein the state detection result is used to characterize the current state of the driver. The control module is used to generate the target information based on the state detection results.

2. The driver monitoring system according to claim 1, characterized in that, Also includes: The display unit is used to display the target information in front of the driver based on light field naked-eye 3D display technology.

3. The driver monitoring system according to claim 1, characterized in that, The control module is also connected to the first acquisition unit, the second acquisition unit, and the decoding module respectively, and is used to receive the first monitoring video and the second monitoring video, and send the first monitoring video and the second monitoring video to the decoding module.

4. The driver monitoring system according to claim 1, characterized in that, The control module is also connected to the preprocessing module and the feature extraction and fusion module, respectively, and is used to respond to notification commands and generate target working commands; wherein... The preprocessing module is further configured to generate the notification instruction when the preprocessing of the target image data is completed; The feature extraction and fusion module is also used to extract and fuse features from the preprocessed target image data based on the target working instructions.

5. The driver monitoring system according to claim 1, characterized in that, The data processing unit further includes: A data buffer module, which is connected to both the decoding module and the preprocessing module, is used to store the target image data.

6. The driver monitoring system according to claim 2, characterized in that, The display unit includes: The HUD control module is used to establish a HUD display view corresponding to the target information based on light field naked-eye 3D display technology, and to render the HUD display view to generate the target display view; The HUD processing module is used to process the target display view based on a preset light field texture so that the processed target display view can be displayed on the vehicle's HUD display.

7. A vehicle, characterized in that, Including the driver monitoring system according to any one of claims 1-6.

Images