Driver state monitoring method and device, electronic equipment and medium

Abstract

The invention provides a driver state monitoring method and device, electronic equipment and a medium, and relates to the field of vehicle-mounted intelligent monitoring, and the method comprises the steps: obtaining image data of a driver collected by image collection equipment, the image data comprising facial expressions, eye states and body movements; preprocessing the image data to obtain initial image data; processing the initial image data by adopting a super-resolution enhancement model to obtain target image data; performing feature extraction on the target image data to obtain state features; and if the state feature is an abnormal state, generating early warning information corresponding to the abnormal state. According to the method, details of key areas such as eyes and mouth corners of a driver are enhanced through a super-resolution technology, the abnormal state recognition accuracy is greatly improved, the problems of false detection and missing detection of a traditional system are solved, detection tasks are completed with lower hardware requirements, deployment and landing costs are effectively reduced, detection precision and practicability are considered, and the method is suitable for popularization and application. And the suitability and the popularization value of the scheme are greatly improved.

Description

Technical Field

[0001] This application relates to the field of vehicle-mounted intelligent monitoring, and more specifically, to a method, device, electronic device, and medium for monitoring driver status. Background Technology

[0002] With the rapid development of intelligent driving technology, driver status monitoring has become a core element in ensuring driving safety. Current driver monitoring systems on the market mainly consist of image acquisition, basic image processing, simple status recognition, storage, and early warning modules. They acquire images through ordinary cameras, process them simply, and then use traditional algorithms to determine the driver's status and issue warnings. However, existing technologies have significant shortcomings: images acquired by ordinary cameras have low resolution and lack super-resolution technology support, resulting in blurred key details such as the degree of eye closure and micro-expressions around the mouth, leading to a high rate of false alarms; in complex lighting conditions such as backlighting and nighttime, image contrast is low and noise is high, making it difficult for existing algorithms to effectively optimize; the separation of traditional image processing and super-resolution algorithms results in cumbersome computations and insufficient real-time performance, failing to meet the real-time monitoring needs of driving scenarios; and the lack of optimized storage strategies leads to a large amount of invalid data consuming resources and low efficiency in retrieving key data. Summary of the Invention

[0003] The purpose of this application is to provide a method, device, electronic device and medium for monitoring driver status, so as to solve the above-mentioned problems existing in the prior art and improve the accuracy of key area detail recognition and abnormal status detection rate.

[0004] Firstly, a method for monitoring driver status is provided, which may include: Image acquisition devices collect image data of the driver, including facial expressions, eye states, and body movements; The image data is preprocessed to obtain initial image data; The initial image data is processed using a super-resolution enhancement model to obtain the target image data; Feature extraction is performed on the target image data to obtain state features; If the state characteristic is an abnormal state, then a warning message corresponding to the abnormal state is generated.

[0005] In one possible implementation, the preprocessing includes noise reduction, illumination equalization, and core region preservation, performed sequentially. The noise reduction uses a Gaussian filtering algorithm, the illumination equalization uses a preset algorithm to adjust the brightness distribution, and the core region preservation process preserves the core monitoring area of ​​the driver's face through edge cropping.

[0006] In one possible implementation, a super-resolution enhancement model is used to process the initial image data to obtain target image data, including: The super-resolution enhancement model is used to adaptively divide the initial image data into regions, resulting in the eye region, mouth region, and facial texture region. Enhancement processing is performed on the eye area, the corner of the mouth area, and the facial texture area to obtain corresponding enhanced eye areas, enhanced corner of the mouth areas, and enhanced facial texture areas; The enhanced eye area, the enhanced corner of the mouth area, and the enhanced facial texture area are fused together to obtain the target image data.

[0007] In one possible implementation, the super-resolution enhancement model dynamically adjusts the enhancement strategy based on image content; the enhancement strategy is to adjust processing parameters according to the image parameters of the initial image data.

[0008] In one possible implementation, feature extraction is performed on the target image data to obtain state features, including: The target image is subjected to feature extraction using a multi-channel parallel extraction method of convolutional neural network to obtain eye features, facial features, and limb features; The eye features, facial features, and limb features are fused to obtain a comprehensive feature vector; The state features are determined based on the comprehensive feature vector, and the state features include at least one of eye opening and closing, pupil diameter, head turning angle, and facial expression features.

[0009] In one possible implementation, determining whether a state characteristic is an abnormal state includes: A configured classification model is used to determine the state features and vehicle driving data, and a determination result is obtained; the vehicle driving data includes vehicle speed and steering signals.

[0010] In one possible implementation, the abnormal state includes at least one of a fatigue state, a distracted state, and a violation of making or receiving phone calls.

[0011] Secondly, a driver status monitoring device is provided, which may include: The acquisition unit is used to acquire image data of the driver collected by the image acquisition device, the image data including facial expressions, eye status and body movements; The processing unit is used to preprocess the image data to obtain initial image data; Furthermore, the initial image data is processed using a super-resolution enhancement model to obtain the target image data; The extraction unit is used to extract features from the target image data to obtain state features; The generation unit is used to generate early warning information corresponding to the abnormal state if the state feature is an abnormal state.

[0012] Thirdly, an electronic device is provided, which includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements any of the steps described in the first aspect above.

[0013] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the steps of any of the methods described in the first aspect above.

[0014] This application provides a method, device, electronic device, and medium for monitoring driver status. The method includes: acquiring image data of the driver using an image acquisition device, the image data including facial expressions, eye status, and body movements; preprocessing the image data to obtain initial image data; processing the initial image data using a super-resolution enhancement model to obtain target image data; extracting features from the target image data to obtain status features; and generating a warning message corresponding to the abnormal status if the status features indicate an abnormal status. This method enhances the details of key areas such as the driver's eyes and mouth using super-resolution technology, significantly improving the accuracy of abnormal status recognition and solving the problems of false positives and false negatives in traditional systems. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A system architecture diagram of a driver status monitoring method provided in this application embodiment; Figure 2 A flowchart illustrating a driver status monitoring method provided in this application embodiment; Figure 3 A schematic diagram illustrating the processing of the initial image data by the super-resolution enhancement model provided in this embodiment of the application; Figure 4A schematic diagram of the structure of a driver status monitoring device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0018] The driver status monitoring method provided in this application embodiment can be applied to... Figure 1 In the system architecture shown, such as Figure 1 As shown, the system may include a server and a terminal. The server can be a physical server, a server cluster consisting of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal may be a user equipment (UE) such as a mobile phone, smartphone, laptop, digital radio receiver, personal digital assistant (PDA), tablet computer (PAD), handheld device, in-vehicle device, wearable device, computing device, or other processing device connected to a wireless modem, mobile station (MS), mobile terminal, etc. The terminal and server can be directly or indirectly connected via wired or wireless communication methods, which is not limited herein.

[0019] The terminal is used to acquire image data of the driver collected by the image acquisition device and send the image data to the server; The server is used to acquire image data of the driver collected by the image acquisition device in order to execute the driver status monitoring method provided in this application.

[0020] With the rapid development of intelligent driving technology, driver status monitoring has become a core element in ensuring driving safety. Current driver monitoring systems on the market mainly consist of image acquisition, basic image processing, simple status recognition, storage, and early warning modules. They acquire images through ordinary cameras, process them simply, and then use traditional algorithms to determine the driver's status and issue warnings. However, existing technologies have significant shortcomings: images acquired by ordinary cameras have low resolution and lack super-resolution technology support, resulting in blurred key details such as the degree of eye closure and micro-expressions around the mouth, leading to a high rate of false alarms; in complex lighting conditions such as backlighting and nighttime, image contrast is low and noise is high, making it difficult for existing algorithms to effectively optimize; the separation of traditional image processing and super-resolution algorithms results in cumbersome computations and insufficient real-time performance, failing to meet the real-time monitoring needs of driving scenarios; and the lack of optimized storage strategies leads to a large amount of invalid data consuming resources and low efficiency in retrieving key data.

[0021] Therefore, this application provides a method for monitoring driver status, which solves the above-mentioned problems existing in the prior art and can improve the accuracy of key area detail recognition and abnormal status detection rate.

[0022] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0023] Figure 2 This is a flowchart illustrating a method for monitoring driver status provided in an embodiment of this application. Figure 2 As shown, the method may include: Step S210: Acquire image data of the driver from the image acquisition device.

[0024] The image data includes facial expressions, eye states, and body movements; The image acquisition device uses a 2-megapixel CMOS (Complementary Metal Oxide Semiconductor) image sensor, equipped with two infrared illumination units. Its field of view is 38° horizontally (H) and 58° vertically (V). The image output resolution is no less than 1920*1200. The default acquisition frame rate is 30FPS, and it supports a dynamic adjustment range of 30-60 frames / second, which can adapt to the imaging needs of low-light scenes such as night and tunnels.

[0025] After the vehicle starts, the system automatically powers on and completes a 3-second self-test (checking the core hardware such as the high-definition acquisition module and the super-resolution processing module, as well as the connectivity of the communication link). After the self-test is passed, it enters the image acquisition stage. During the acquisition process, the image acquisition device adopts an adaptive acquisition strategy: it automatically controls the opening and closing of the infrared supplementary light unit according to the ambient light intensity. When the light intensity is lower than the preset threshold (such as at night or in backlit scenes), the infrared supplementary light unit is activated to supplement a uniform infrared light source to suppress image noise and improve contrast. At the same time, it dynamically adjusts the acquisition frame rate in combination with the real-time vehicle speed. When driving at high speed, the frame rate is increased to capture fast limb movements, and when driving at low speed or stationary, the frame rate is appropriately reduced to balance performance and power consumption.

[0026] The acquired image data is directly connected to the subsequent super-resolution processing module via the LVDS (Low-Voltage Differential Signaling) interface, enabling uncompressed real-time transmission and avoiding image quality loss during data transmission. The image data comprehensively covers three core monitoring dimensions: driver facial expressions, eye status, and body movements. Facial expression data includes micro-expression information such as the curvature of the corners of the mouth and the state of facial muscle contraction; eye status data covers key details such as the degree of eyelid opening and closing, pupil diameter, and eye movement trajectory; and body movement data includes features such as head tilt angle, torso posture, hand placement, and range of motion, ensuring that the subsequent feature extraction stage can obtain the complete data support required for driver status assessment.

[0027] The acquisition solution, through the collaborative design of high-definition sensors, adaptive lighting, and frame rate adjustment, not only ensures high resolution and detail integrity of image data, but also achieves adaptability to complex driving scenarios such as nighttime, backlighting, and tunnel entrances and exits, effectively solving the problem of poor data quality of traditional acquisition equipment in low-light and dynamic scenarios.

[0028] Step S220: Preprocess the image data to obtain initial image data.

[0029] The preprocessing includes noise reduction, illumination equalization, and core area preservation, which are performed sequentially.

[0030] A Gaussian filtering algorithm is used to denoise the original image data. The core principle is based on the random distribution characteristics of image noise, achieving a balance between noise suppression and detail preservation by adaptively adjusting the filter kernel parameters. The filter kernel size is dynamically adapted according to the noise intensity of the original image (e.g., a 5×5 filter kernel is used for high-noise images in low-light nighttime scenes; a 3×3 filter kernel is used for low-noise images in well-lit scenes). A weighted average is calculated for each pixel and its neighboring pixels using the weighting rules of the Gaussian function. This maximizes the filtering of random noise and salt-and-pepper noise while avoiding blurring and distortion of image edge details and texture information. This processing effectively solves the problem of misjudgment in state recognition caused by high image noise in traditional systems, laying a clear image foundation for subsequent processing.

[0031] Based on a preset algorithm, the denoised image undergoes illumination equalization processing. Addressing the issues of uneven brightness distribution and low contrast in complex lighting scenarios such as backlighting, nighttime conditions, and tunnel entrances / exits, brightness optimization is achieved by decomposing the image's illuminance and reflectance components. First, the illuminance component, determined by ambient light, and the reflectance component, determined by the object's inherent characteristics, are separated. Then, the illuminance component is dynamically stretched and normalized to adjust brightness differences in different areas, improving detail visibility in dark areas and suppressing overexposure in bright areas. The final output is an image with uniform brightness and moderate contrast. This processing, combined with the infrared illumination function of the high-definition acquisition module, significantly enhances the image signal-to-noise ratio in complex lighting scenarios, ensuring clear rendering of the driver's face and key details in various driving environments.

[0032] Edge cropping technology is used to preserve the core area while removing redundant pixel data irrelevant to driver status monitoring. First, a deep learning-based facial detection algorithm is employed to accurately locate the core facial region of the driver in the image (covering key monitoring areas such as the eyes, face, and corners of the mouth), and the cropping boundary is determined based on a preset monitoring range threshold. Then, edge cropping is performed on the image to remove non-core areas such as the vehicle interior and background environment, retaining only the core facial region and the image portion corresponding to key upper body movements. This processing not only reduces the amount of computational data required for subsequent super-resolution enhancement and feature extraction, improving the system's real-time performance, but also avoids interference from redundant data in key feature extraction, and provides a data optimization foundation for implementing differentiated storage strategies.

[0033] The initial image data obtained after the above three preprocessing steps has the characteristics of low noise, uniform brightness, prominent key areas, and concise data volume. It not only solves the limitations of traditional image preprocessing that only performs simple operations, but also accurately matches the technical requirements of subsequent super-resolution enhancement and state feature extraction, providing core data support for the system to achieve accurate and real-time monitoring of the driver's state.

[0034] Step S230: Use a super-resolution enhancement model to process the initial image data to obtain the target image data.

[0035] Combination Figure 3 As shown, the super-resolution enhancement model is deployed on a dedicated image processing chip with powerful computing capabilities. The model architecture integrates a feature extraction network, residual dense blocks, and an adaptive upsampling module. It can achieve a step-by-step increase in image resolution while meeting real-time requirements (processing time ≤15ms), and finally output target image data adapted to feature extraction needs. Its core processing flow consists of three stages: adaptive region segmentation, region-specific enhancement, and region fusion. It also supports dynamic adjustment of enhancement strategies based on image content to ensure processing effects and scene adaptability.

[0036] Specifically, step 1 involves adaptively segmenting the initial image data using a super-resolution enhancement model to obtain the eye region, mouth region, and facial texture region. This step can be understood as the super-resolution enhancement model calling a pre-trained lightweight region detection sub-model to perform pixel-level content parsing and precise region localization on the initial image data. This detection sub-model is trained based on a convolutional neural network (CNN) and can automatically identify and segment three core monitoring regions: the eye region (covering the eyelids, pupils, and periocular skin), the mouth region (including the lip contour and surrounding micro-expression related regions), and the facial texture region (covering facial muscle distribution, skin texture, and other state-related regions), while also marking non-critical background regions. During the segmentation process, the model calculates the correlation weight between each region and the driver's state assessment (the eye region has the highest weight, followed by the mouth region, and then the facial texture region), allocating computational resources for subsequent region enhancement. This ensures the accuracy of key region processing while avoiding unnecessary computational consumption in non-critical regions, meeting the real-time design requirements of the lightweight model.

[0037] Step 2: Enhance the eye area, mouth area, and facial texture area to obtain corresponding enhanced eye areas, mouth area areas, and facial texture areas. Eye area enhancement: Focus on strengthening subtle features such as eyelid opening and closing edges and pupil contours. Increase the number of stacked residual dense blocks (2-3 more layers than other areas) to improve detail reproduction. Simultaneously, use an adaptive noise reduction algorithm to suppress noise interference in low-light scenes, ensuring accurate extraction of features such as eye opening and closing angles and pupil diameter. Mouth area enhancement: Focus on micro-expression details such as changes in mouth corner curvature and lip muscle contraction. Adjust edge sharpening parameters and contrast gain to highlight micro-expression differences, providing clear data support for facial expression feature recognition. Facial texture area enhancement: Balance detail clarity and image quality fidelity. Through appropriate upsampling and texture restoration, preserve the texture associated with facial muscle distribution and skin folds, while avoiding artifacts caused by over-enhancement, ensuring the integrity of overall facial features. During the enhancement process in each region, the model synchronously calls the color correction module to correct the color deviation of the initial image, ensuring color consistency in each region under different lighting conditions.

[0038] Step 3: Fuse the enhanced eye area, the enhanced mouth area, and the enhanced facial texture area to obtain the target image data. After the regional enhancement is completed, the model uses a weighted fusion algorithm to integrate the enhanced eye area, the enhanced mouth area, and the enhanced facial texture area. The fusion weight is based on the preset correlation weight of each region (the weight of the enhanced eye area is not less than 40%, and the weight of the enhanced mouth area is not less than 30%), ensuring that the key regional features are still prominent after fusion. To avoid abruptness at the boundaries of the region fusion, the model uses an edge smoothing transition algorithm to perform pixel-level optimization on the boundaries of each region, eliminating stitching marks and ensuring the overall coordination of the target image data. At the same time, the adaptive upsampling module achieves the final improvement of image resolution, so that the target image data resolution reaches 3840*2160P (4K) and the peak signal-to-noise ratio (PSNR) is ≥37dB.

[0039] In some embodiments, the super-resolution enhancement model dynamically adjusts the enhancement strategy based on image content; the enhancement strategy involves adjusting processing parameters according to the image parameters of the initial image data. In some embodiments, the super-resolution enhancement model supports dynamically adjusting the enhancement strategy based on the image parameters of the initial image data, further improving scene adaptability. The image parameters include, but are not limited to, noise level, brightness uniformity, contrast, and blur degree. The model adaptively adjusts the processing parameters by analyzing these parameters in real time: for example, when the noise level of the initial image is detected to be higher than a preset threshold (such as at night or in backlit scenes), the model automatically increases the noise reduction processing intensity of each region while maintaining the enhancement strength of key details; when the image brightness is detected to be low, the local brightness is moderately increased while enhancing details to avoid loss of details in dark areas; when the image is clear overall and the noise is low, the proportion of noise reduction operations is reduced, and the focus is on optimizing the detail sharpening effect. This dynamic adjustment mechanism enables the super-resolution enhancement model to adapt to the image data characteristics under different driving scenarios, ensuring that high-quality target image data can be output regardless of complex scenes such as at night, backlight, and rain, or normal scenes with sufficient light.

[0040] This approach, through a collaborative design of adaptive region segmentation, precise enhancement of different regions, and dynamic strategy adjustment, not only solves the problems of traditional super-resolution algorithms that do not segment regions and lack enhancement of key details, but also ensures real-time processing (model inference speed ≥ 50 frames / second) through a lightweight architecture and reasonable allocation of computing power. Ultimately, it yields target image data with clear details in key regions, stable overall image quality, and adaptability to subsequent feature extraction requirements, providing core technical support for accurate judgment of driver status.

[0041] Step S240: Extract features from the target image data to obtain state features.

[0042] Specifically, step 1 involves using a multi-channel parallel extraction method based on a convolutional neural network to extract features from the target image, obtaining eye features, facial features, and limb features. This process can be understood as follows: using a multi-channel parallel extraction scheme based on a CNN architecture, relying on a vehicle-mounted intelligent chip with powerful computing capabilities, the super-resolution enhanced 4K target image data is transmitted in blocks in parallel and synchronously to the three dedicated channels of the feature extraction module (eye feature extraction channel, facial feature extraction channel, and limb feature extraction channel). Each channel operates independently and synchronously, significantly improving feature extraction efficiency and ensuring that the total time for feature extraction of a single frame image is ≤15ms, meeting the system's real-time requirements (total latency ≤100ms). Eye Feature Extraction Channel: Focusing on the eye region in the target image, this channel accurately captures subtle features such as eyelid opening and closing edges, pupil contours, and eye movement trajectories through multi-scale convolutional layers and edge detection sub-networks. It outputs high-dimensional feature data including grayscale distribution, contour coordinates, and dynamic change trends of the eye region, laying the foundation for subsequent analysis of key state features such as eye opening and closing degree and pupil diameter. Facial Feature Extraction Channel: Targeting the core facial region (covering the corners of the mouth, cheek muscles, forehead, etc.), this channel extracts features such as facial muscle contraction state, changes in the curvature of the corners of the mouth, and skin texture details through a texture feature extraction network and micro-expression recognition sub-model. It focuses on capturing facial micro-expression changes related to fatigue and distraction. Limb Feature Extraction Channel: Focusing on the driver's upper body limb movements, this channel extracts feature data such as head tilt angle, torso tilt posture, hand placement, and movement amplitude through a pose estimation network. The head tilt angle is calculated using keypoint coordinates, and hand movements are achieved through contour recognition and dynamic trajectory tracking. This parallel extraction scheme works synergistically with super-resolution enhancement processing. The sharpness of key region details in the super-resolution target image (PSNR≥37dB) ensures the accuracy of feature extraction for each channel. At the same time, the channel-specific design avoids mutual interference between different types of features and improves the purity of the feature data.

[0043] Step 2: Fuse eye features, facial features, and limb features to obtain a comprehensive feature vector. This process may include: First, standardizing the original feature data output from each channel to eliminate fusion interference caused by differences in feature dimensions and numerical ranges across different channels, ensuring that each feature has an equal weight in the fusion process; Second, adopting a weighted concatenation and feature interaction fusion strategy, pre-setting fusion weights based on the contribution of each feature to the abnormal state determination, with eye features having a weight of no less than 40%, facial features no less than 30%, and limb features no less than 20%, and the remaining weights allocated to interaction terms between features to strengthen the correlation information between different types of features; Finally, concatenating and integrating the standardized channel features and interaction features through a fully connected layer to output a comprehensive feature vector with a fixed dimension of 512. During the fusion process, vehicle driving data (such as vehicle speed and steering signals) and environmental sensor data (such as light intensity and road conditions) transmitted via the CAN bus are introduced simultaneously. These data are quantified into low-dimensional features and then embedded into a 512-dimensional comprehensive feature vector. This allows the feature vector to not only contain image visual features but also integrate driving scene context information, thereby improving the comprehensiveness and accuracy of subsequent state determination.

[0044] Step 3: Determine the state features based on the comprehensive feature vector. The state features include at least one of the following: eye opening and closing degree, pupil diameter, head turning angle, and facial expression features. This step can specifically include: Eye opening / closing degree: By parsing the eyelid contour coordinate data output from the eye feature channel, the ratio of the distance between the upper and lower eyelids to the vertical length of the eye region is calculated. Combined with the changing trend within a dynamic time window, the real-time eye opening / closing degree parameter is obtained. This parameter can be directly used for fatigue state judgment (e.g., eye closure time ≥ 2 seconds is judged as severe fatigue); Pupil diameter: Based on the pupil contour coordinates in the eye features, the actual pupil diameter is calculated using an ellipse fitting algorithm. At the same time, ambient light intensity data is used for correction to eliminate the interference of light changes on pupil diameter detection; Head deflection angle: By using the head key point coordinates output from the limb feature channel, the horizontal deflection angle and vertical pitch angle of the head relative to the reference coordinate system (based on the driver looking straight ahead) are calculated for distraction state judgment (e.g., continuous head deflection angle ≥ 30° is judged as distraction); Facial expression features: By parsing the micro-expression data output from the facial feature channel, combined with a pre-trained expression classification sub-model, facial states are mapped to expression categories such as calm, fatigue, irritability, and focus, providing supplementary basis for intelligent state analysis.

[0045] The process of determining state characteristics strictly follows the preset accuracy standards to ensure that the detection error of eye opening and closing is ≤5%, the detection error of head deflection angle is ≤3°, and the detection error of pupil diameter is ≤0.1mm.

[0046] In summary, this feature extraction step employs a three-stage design of multi-channel parallel extraction, multi-dimensional feature fusion, and precise feature parsing. This approach fully leverages the advantages of high-definition image data enhanced by super-resolution and improves the comprehensiveness of features by fusing driving and environmental data. The final output state features are characterized by high precision and high effectiveness, providing core data support for the accurate determination of abnormal driver states. At the same time, it meets the real-time requirements of the system and solves the technical defects of traditional feature extraction methods, such as low precision, single features, and susceptibility to interference.

[0047] Step S250: Determine whether the status characteristics are abnormal, and determine the warning information based on the determination result.

[0048] Specifically, a configured classification model is used to determine the state features and vehicle driving data, and to obtain the determination result; the vehicle driving data includes vehicle speed and steering signals. This step may include: the classification model is a multi-feature fusion classification model trained based on deep learning, deployed on an onboard intelligent chip with powerful computing capabilities, and establishing a real-time data interaction link with the feature extraction module, vehicle CAN bus and environmental sensors to ensure efficient response of the determination process (single frame determination time ≤15ms). The model's input data contains two types of core information, and the accuracy of the judgment is ensured through a synchronous temporal alignment mechanism: The first type is a 512-dimensional comprehensive feature vector output from the feature extraction step. This vector integrates core driver status information such as eye opening and closing, pupil diameter, head deflection angle, and facial expression features. Its data accuracy meets the following requirements: eye opening and closing detection error ≤5%, head deflection angle detection error ≤3°, and pupil diameter detection error ≤0.1mm, providing accurate visual feature support for abnormal state judgment. The second type is vehicle driving data and environmental sensor data. The vehicle driving data is acquired in real time through the CAN bus, including vehicle speed and steering signals (left turn / right turn / straight ahead). The environmental sensor data includes current light intensity and driving conditions (highway / urban road / rural road), which are used to dynamically adjust the judgment threshold in combination with the scene to improve the judgment adaptability under different driving scenarios. The classification model employs a three-dimensional judgment logic of feature threshold determination, time-series trend analysis, and scenario adaptation adjustment to comprehensively analyze the input data and accurately identify at least one abnormal state among fatigue, distraction, and phone call violations. The overall abnormal state detection rate is ≥95%, accuracy is ≥95%, and false alarm rate is ≤3%. Specific judgment rules include: Fatigue status determination: Based on the characteristics of eye opening and closing, combined with time-series trend analysis, when the driver's eye closing time is ≥2 seconds, or the average eye opening and closing value is lower than the preset threshold (such as 30% of the normal opening and closing value) within 30 consecutive seconds, and combined with vehicle speed data (the threshold is appropriately tightened when driving at high speed), it is determined to be a fatigue state. Distraction status determination: Based on the joint determination of head turning angle and facial expression features, when a continuous head turning angle ≥30° and a duration of more than 5 seconds is detected (such as looking out of the car or looking down at the device), or the facial expression features show the non-focus category and are accompanied by unstable head posture, combined with the steering signal (the determination is more stringent when there is no steering operation), it is determined to be a distraction state; among them, short-term distraction (such as a duration ≤2 seconds) is determined to be mild abnormality, and continuous distraction (such as a duration ≥5 seconds) is determined to be moderate abnormality; Determination of illegal phone call status: Based on the correlation analysis of body features and facial regions, when it is detected that the driver's hand is continuously placed on the side of the face (near the ear), accompanied by a slightly fixed head posture and eyes deviating from the direction of travel, and excluding interference actions such as normal adjustment of sitting posture, it is determined to be an illegal phone call status, which is classified as moderate abnormality by default.

[0049] During the judgment process, the model supports dynamic adjustment of thresholds according to the driving scenario: when driving at high speed (vehicle speed ≥ 60km / h), the judgment thresholds for fatigue and distraction are appropriately tightened (e.g., judgment is triggered when eyes are closed for ≥ 1.5 seconds) to adapt to the safety requirements in high-speed scenarios; when driving on urban roads (vehicle speed ≤ 40km / h), the thresholds are appropriately relaxed to reduce the false alarm rate caused by minor actions in scenarios such as traffic congestion, and to achieve accurate scenario-based judgment.

[0050] If an alarm needs to be triggered, the target image data is compressed to reduce memory usage. The compressed image retains details of key monitoring areas and does not affect the tracking of abnormal states.

[0051] If the state characteristic is an abnormal state, then a warning message corresponding to the abnormal state is generated; the abnormal state includes at least one of the following: fatigue state, distraction state, and illegal phone call state.

[0052] In some embodiments, based on the model's judgment results, the system automatically generates corresponding warning information and adopts a tiered warning strategy to ensure timely alerts for abnormal states while avoiding excessive interference with the driver's normal driving. Specific tiering rules and output methods include: Mild Abnormality Warning: For mild abnormal states such as brief distraction, only text warning information (such as "Please pay attention to driving") is output through the vehicle display controller (supports 1080P output), without voice prompts, to reduce interference with driving; Moderate Abnormality Warning: For moderate abnormal states such as continuous distraction and illegal phone calls, a combination of text and voice prompts is triggered. The in-vehicle display shows the abnormality type (such as distracted driving and illegal phone calls), and the speaker plays a clear voice prompt (such as please focus on driving immediately and do not make or receive phone calls). The voice volume is adapted to the ambient noise in the vehicle to ensure that the driver can clearly perceive it. Severe Abnormality Warning: In response to severe abnormal conditions such as fatigue, in addition to text and voice prompts, the frequency of voice prompts is increased (repeated once every 3 seconds), and the warning information is continuously highlighted on the in-vehicle display screen until the driver's condition returns to normal or the vehicle decelerates to a safe speed, thereby enhancing the warning effect.

[0053] The vehicle display controller supports 1080P output, and users can set system parameters such as warning sensitivity and anomaly detection threshold through the touch screen.

[0054] The judgment results are synchronously transmitted to the differentiated storage module, forming an associated storage of the judgment results, image data, and timestamps: When an abnormal state is determined, a differentiated storage strategy is triggered, automatically storing the 4K super-resolution target image data at the time of the abnormality, and marking the abnormality type (fatigue / distraction / making a phone call), judgment timestamp, and scene information such as current vehicle speed and road conditions; when a normal state is determined, only 720P low-resolution non-critical frame images are stored, ensuring that critical abnormal data is traceable while optimizing storage resource usage (daily storage ≤ 16GB). A 256GB high-speed eMMC flash memory is used with a tiered storage controller, supporting fast retrieval by abnormal type and timestamp (location time ≤ 3 seconds). When storage overflows, non-critical data is automatically overwritten, prioritizing the retention of 4K keyframes corresponding to abnormal states within the last 30 days.

[0055] In some embodiments, the system supports OTA remote upgrades, which can update the super-resolution model and classification model algorithms; it has a built-in fault self-checking module that regularly checks the operating status of each module and prompts maintenance information through the vehicle display screen when an abnormality occurs.

[0056] This approach, through the collaborative design of precise feature input, scenario-based judgment logic, and hierarchical early warning output, not only solves the problem of traditional judgment algorithms relying on only a single feature and having a high false judgment rate, but also achieves accurate identification, timely warning, and efficient traceability of abnormal states through deep collaboration with driving scenarios and storage modules, providing full-process protection for driving safety, while meeting the core requirements of system real-time performance (total latency ≤100ms) and user experience optimization.

[0057] This application provides a method for monitoring driver status. The method includes: acquiring image data of the driver using an image acquisition device, the image data including facial expressions, eye state, and body movements; preprocessing the image data to obtain initial image data; processing the initial image data using a super-resolution enhancement model to obtain target image data; extracting features from the target image data to obtain state features; and generating a warning message corresponding to the abnormal state if the state features indicate an abnormal state. This method enhances the details of key areas such as the driver's eyes and mouth using super-resolution technology, significantly improving the accuracy of abnormal state recognition and solving the problems of false positives and false negatives in traditional systems. Furthermore, it completes the detection task with lower hardware requirements, effectively reducing deployment and implementation costs, balancing detection accuracy and practicality, and greatly improving the adaptability and promotional value of the solution.

[0058] Corresponding to the above method, embodiments of this application also provide a driver status monitoring device, such as... Figure 4 As shown, the device includes: The acquisition unit 410 is used to acquire image data of the driver collected by the image acquisition device, the image data including facial expressions, eye state and body movements; Processing unit 420 is used to preprocess the image data to obtain initial image data; Furthermore, the initial image data is processed using a super-resolution enhancement model to obtain the target image data; Extraction unit 430 is used to extract features from the target image data to obtain state features; The generation unit 440 is used to generate early warning information corresponding to the abnormal state if the state feature is an abnormal state.

[0059] The functions of each functional unit of the driver status monitoring device provided in the above embodiments of this application can be implemented through the above methods and steps. Therefore, the specific working process and beneficial effects of each unit in the driver status monitoring device provided in the embodiments of this application will not be repeated here.

[0060] This application also provides an electronic device, such as... Figure 5 As shown, it includes a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540.

[0061] Memory 530 is used to store computer programs; When the processor 510 executes the program stored in the memory 530, it performs the following steps: Image acquisition devices collect image data of the driver, including facial expressions, eye states, and body movements; The image data is preprocessed to obtain initial image data; The initial image data is processed using a super-resolution enhancement model to obtain the target image data; Feature extraction is performed on the target image data to obtain state features; If the state characteristic is an abnormal state, then a warning message corresponding to the abnormal state is generated.

[0062] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0063] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0064] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0065] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0066] The implementation methods and beneficial effects of the various components of the electronic device in the above embodiments for solving the problem can be found in [reference needed]. Figure 2The steps in the illustrated embodiments are used to implement the electronic device. Therefore, the specific working process and beneficial effects of the electronic device provided in this application will not be repeated here.

[0067] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform a driver state monitoring method as described in any of the above embodiments.

[0068] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the driver state monitoring methods described in the above embodiments.

[0069] Those skilled in the art will understand that the embodiments in this application can be provided as methods, systems, or computer program products. Therefore, the embodiments in this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments in this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0071] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0072] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0073] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected," "coupled," or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0074] Although preferred embodiments have been described in this application, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the embodiments in this application are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments in this application.

[0075] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the embodiments of this application and their equivalents, then these modifications and variations are also intended to be included in the embodiments of this application.

Claims

1. A method for monitoring driver status, characterized in that, The method includes: Image acquisition devices collect image data of the driver, including facial expressions, eye states, and body movements; The image data is preprocessed to obtain initial image data; The initial image data is processed using a super-resolution enhancement model to obtain the target image data; Feature extraction is performed on the target image data to obtain state features; If the state characteristic is an abnormal state, then a warning message corresponding to the abnormal state is generated.

2. The method as described in claim 1, characterized in that, The preprocessing includes noise reduction, illumination equalization, and core area preservation, which are performed sequentially. The noise reduction uses a Gaussian filtering algorithm, the illumination equalization uses a preset algorithm to adjust the brightness distribution, and the core area preservation process preserves the core monitoring area of ​​the driver's face through edge cropping.

3. The method as described in claim 1, characterized in that, The initial image data is processed using a super-resolution enhancement model to obtain target image data, including: The super-resolution enhancement model is used to adaptively divide the initial image data into regions, resulting in the eye region, mouth region, and facial texture region. Enhancement processing is performed on the eye area, the corner of the mouth area, and the facial texture area to obtain corresponding enhanced eye areas, enhanced corner of the mouth areas, and enhanced facial texture areas; The enhanced eye area, the enhanced corner of the mouth area, and the enhanced facial texture area are fused together to obtain the target image data.

4. The method as described in claim 3, characterized in that, The super-resolution enhancement model is based on a dynamic adjustment strategy for image content; the enhancement strategy is to adjust the processing parameters according to the image parameters of the initial image data.

5. The method as described in claim 1, characterized in that, Feature extraction is performed on the target image data to obtain state features, including: The target image is subjected to feature extraction using a multi-channel parallel extraction method of convolutional neural network to obtain eye features, facial features, and limb features; The eye features, facial features, and limb features are fused to obtain a comprehensive feature vector; The state features are determined based on the comprehensive feature vector, and the state features include at least one of eye opening and closing, pupil diameter, head turning angle, and facial expression features.

6. The method as described in claim 1, characterized in that, Determining whether a state characteristic is an abnormal state includes: A configured classification model is used to determine the state features and vehicle driving data, and a determination result is obtained; the vehicle driving data includes vehicle speed and steering signals.

7. The method as described in claim 1, characterized in that, The abnormal state includes at least one of the following: fatigue state, distracted state, and illegal phone call state.

8. A driver status monitoring device, characterized in that, The device includes: The acquisition unit is used to acquire image data of the driver collected by the image acquisition device, the image data including facial expressions, eye status and body movements; The processing unit is used to preprocess the image data to obtain initial image data; Furthermore, the initial image data is processed using a super-resolution enhancement model to obtain the target image data; The extraction unit is used to extract features from the target image data to obtain state features; The generation unit is used to generate early warning information corresponding to the abnormal state if the state feature is an abnormal state.

9. An electronic device, characterized in that, The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-7.

Images