Driver attention level warning method, device, equipment and program product

Abstract

The application discloses a driver attention grading early warning method, device, equipment and program product, comprising: acquiring brain wave data, facial image data, operation behavior data of a target driver and a current driving scene of a target vehicle; extracting brain wave distribution features according to the brain wave data, extracting facial key point features according to the facial image data, and extracting operation sequence features according to the operation behavior data; determining corresponding fusion weight parameters according to the current driving scene, and performing feature fusion on the brain wave distribution features, the facial key point features and the operation sequence features according to the fusion weight parameters to obtain multi-modal fusion features; inputting the multi-modal fusion features into a pre-trained driver attention recognition model to obtain a current attention score of the target driver, and performing grading early warning according to the current attention score. The application improves the driving safety of the vehicle and can be applied to the technical field of vehicle monitoring.

Description

Technical Field

[0001] This invention relates to the field of vehicle monitoring technology, and in particular to a driver attention level warning method, device, equipment and program product. Background Technology

[0002] Current driver monitoring systems mostly detect driver fatigue through facial images, which is easily affected by facial obstruction and ambient lighting, and cannot accurately reflect the driver's level of concentration. Furthermore, existing vehicle warning mechanisms are simplistic, and intelligent driving takeover strategies are disconnected from the driver's real-time attention status, easily triggering takeover panic or misoperation, thus failing to guarantee vehicle driving safety.

[0003] The above problems urgently need to be addressed. Summary of the Invention

[0004] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0005] Therefore, one objective of this invention is to provide a driver attention grading and early warning method. This method extracts the driver's brainwave distribution features, facial key point features, and operation sequence features, performs feature fusion based on the driving scenario to obtain multimodal fusion features, and combines these features with a pre-trained driver attention recognition model to obtain the driver's current attention score. This allows for grading and early warning based on the driver's real-time attention state, thereby improving vehicle driving safety.

[0006] Another objective of this invention is to provide a driver attention level warning device.

[0007] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of the present invention include: On one hand, embodiments of the present invention provide a driver attention level warning method, comprising the following steps: Acquire the target driver's brainwave data, facial image data, operational behavior data, and the target vehicle's current driving scenario; Brainwave distribution features are extracted from the brainwave data, facial key point features are extracted from the facial image data, and operation sequence features are extracted from the operation behavior data. Based on the current driving scenario, the corresponding fusion weight parameters are determined, and feature fusion is performed on the EEG distribution features, facial key point features, and operation sequence features based on the fusion weight parameters to obtain multimodal fusion features; The multimodal fusion features are input into a pre-trained driver attention recognition model to obtain the current attention score of the target driver, and a graded warning is given based on the current attention score.

[0008] Furthermore, in one embodiment of the present invention, the step of extracting brainwave distribution features based on the brainwave data specifically includes: The EEG data is filtered to obtain multiple EEG component signals, including alpha wave band signals, beta wave band signals, gamma wave band signals, delta wave band signals, and theta wave band signals. The power spectral density of each of the brainwave component signals is determined, the frequency domain energy of the corresponding brainwave component signal is determined based on the power spectral density, and the energy ratio of each brainwave component signal is calculated based on the frequency domain energy. Sliding window analysis was performed on the EEG data to obtain the standard deviation, power spectrum peak position, and power spectrum peak size for each window. The temporal stability of the EEG data is determined based on the fluctuation of the standard deviation of each window, and the frequency domain stability of the EEG data is determined based on the offset of the peak position of the power spectrum of each serial port and the fluctuation of the peak value of the power spectrum. The EEG distribution features are generated based on the energy ratio, the temporal stability, and the frequency stability.

[0009] Furthermore, in one embodiment of the present invention, the step of extracting facial key point features based on the facial image data specifically includes: Key point detection is performed on the facial image data to obtain the real-time key point poses of multiple facial key points; Based on the real-time key point pose, the eye rotation angle, eye closure degree, mouth opening degree, and head pose angle are determined to generate the facial key point features.

[0010] Furthermore, in one embodiment of the present invention, the step of extracting operation sequence features based on the operation behavior data specifically includes: Based on the operational behavior data, multiple operational actions and corresponding operational times are extracted. The operational actions include turning the steering wheel, pressing the accelerator pedal, and pressing the brake pedal. The operation actions are sorted according to the operation time to obtain an operation action sequence; Based on the operational behavior data, the rate of change of steering wheel angle, the rate of change of accelerator pedal opening, or the rate of change of brake pedal opening corresponding to each operational action is statistically analyzed to obtain the action amplitude characteristics corresponding to each operational action. The operation sequence feature is generated based on the operation action sequence and the action amplitude feature.

[0011] Furthermore, in one embodiment of the present invention, the driver attention recognition model is trained through the following steps: Acquire brainwave samples, facial image samples, operational behavior samples of the test driver, and sample driving scenarios of the test vehicle in the test scenario. Multimodal fusion feature samples are determined based on the EEG samples, facial image samples, operational behavior samples, and sample driving scenarios, and corresponding driver attention labels are determined through manual annotation. The multimodal fusion feature samples are input into a pre-constructed convolutional neural network to obtain a predicted attention score; The loss value is determined based on the predicted attention score and the driver attention label; The parameters of the convolutional neural network are updated based on the loss value to obtain the trained driver attention recognition model.

[0012] Furthermore, in one embodiment of the present invention, the step of providing graded warnings based on the current attention score specifically includes: When the current attention score is greater than or equal to a preset first threshold, it is determined that the target driver is in a state of focused attention, and driving reminders are given at preset time intervals; When the current attention score is greater than or equal to a preset second threshold and less than the first threshold, it is determined that the target driver is in a state of slight inattention, and a real-time reminder is given through the dashboard lights; When the current attention score is less than the second threshold, it is determined that the target driver is in a state of severe inattention, and a real-time reminder is given through seat vibration and / or voice broadcast.

[0013] Furthermore, in one embodiment of the present invention, the driver attention level warning method further includes the following steps: When the target driver is in a state of severe inattention for a preset period of time, the intelligent driving system will automatically take over and determine the corresponding intelligent driving mode based on the current driving scenario. When the target driver is detected to have regained focus, the target driver is alerted to take over.

[0014] On the other hand, embodiments of the present invention provide a driver attention level warning device, comprising: The data acquisition module is used to acquire the target driver's electroencephalogram data, facial image data, operational behavior data, and the current driving scenario of the target vehicle; The feature extraction module is used to extract brainwave distribution features based on the brainwave data, extract facial key point features based on the facial image data, and extract operation sequence features based on the operation behavior data. The feature fusion module is used to determine the corresponding fusion weight parameters according to the current driving scenario, and to perform feature fusion on the EEG distribution features, the facial key point features and the operation sequence features according to the fusion weight parameters to obtain multimodal fusion features; The graded early warning module is used to input the multimodal fusion features into a pre-trained driver attention recognition model to obtain the current attention score of the target driver, and to issue graded early warnings based on the current attention score.

[0015] On the other hand, embodiments of the present invention provide an electronic device, including: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the above-described driver attention level warning method.

[0016] On the other hand, embodiments of the present invention also provide a computer-readable storage medium storing a processor-executable computer program that, when executed by a processor, implements the above-described driver attention level warning method.

[0017] On the other hand, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the above-described driver attention level warning method.

[0018] The advantages and beneficial effects of the present invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention: This invention acquires the target driver's electroencephalogram (EEG) data, facial image data, operational behavior data, and the current driving scenario of the target vehicle. It extracts EEG distribution features from the EEG data, facial key point features from the facial image data, and operational sequence features from the operational behavior data. Based on the current driving scenario, it determines corresponding fusion weight parameters and performs feature fusion on the EEG distribution features, facial key point features, and operational sequence features to obtain multimodal fusion features. These multimodal fusion features are then input into a pre-trained driver attention recognition model to obtain the target driver's current attention score. A tiered warning is then issued based on this current attention score. This invention extracts the driver's EEG distribution features, facial key point features, and operational sequence features, performs feature fusion based on the driving scenario to obtain multimodal fusion features, and combines these with a pre-trained driver attention recognition model to obtain the driver's current attention score. This allows for tiered warnings based on the driver's real-time attention state, improving vehicle driving safety. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments of the present invention are described below. It should be understood that the drawings described below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating the steps of a driver attention level warning method provided in an embodiment of the present invention; Figure 2 This is a structural block diagram of a driver attention level warning device provided in an embodiment of the present invention; Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0023] The driver attention level warning method provided in this invention can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application that implements the driver attention level warning method, but is not limited to the above forms.

[0024] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0025] It should be noted that in various specific embodiments of the present invention, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user parking space location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of the present invention require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of the present invention acquired.

[0026] Reference Figure 1 This invention provides a driver attention level warning method, which specifically includes the following steps: S101. Obtain the target driver's brainwave data, facial image data, operational behavior data, and the current driving scenario of the target vehicle; S102. Extract brainwave distribution features from brainwave data, extract facial key point features from facial image data, and extract operation sequence features from operation behavior data. S103. Determine the corresponding fusion weight parameters based on the current driving scenario, and perform feature fusion on EEG distribution features, facial key point features and operation sequence features based on the fusion weight parameters to obtain multimodal fusion features; S104. Input the multimodal fusion features into the pre-trained driver attention recognition model to obtain the current attention score of the target driver, and issue a graded warning based on the current attention score.

[0027] This invention extracts the driver's brainwave distribution features, facial key point features, and operation sequence features, and performs feature fusion based on the driving scenario to obtain multimodal fusion features. Combined with a pre-trained driver attention recognition model, the driver's current attention score is obtained, thereby enabling graded warnings based on the driver's real-time attention state and improving vehicle driving safety.

[0028] First, the target driver's electroencephalogram (EEG) data, facial image data, operational behavior data, and the current driving scenario of the target vehicle are collected, as detailed below: 1) EEG data acquisition Choose a dry electrode head-mounted EEG device and wear it on the driver's forehead, temporal lobe, or other areas with active brain waves. Set the sampling frequency to 256Hz-512Hz to collect raw EEG signals in different frequency bands such as theta waves, alpha waves, and beta waves, and record the timestamps simultaneously. Calibrate the device in advance to eliminate environmental electromagnetic interference and device baseline noise to ensure that the data signal-to-noise ratio meets the standards.

[0029] 2) Facial image data acquisition A high-definition camera is installed directly in front of the cockpit, with a frame rate of 30fps, to capture the driver's face in real time, focusing on key areas such as the eyes, eyebrows, and mouth; the camera's low-light compensation function is enabled to ensure image clarity in low-light scenarios such as nighttime and tunnels.

[0030] 3) Operational behavior data collection The system acquires vehicle operation data such as steering wheel angle, accelerator pedal opening, brake pedal travel, and gear shifting actions via the vehicle's CAN bus interface. The acquisition frequency is synchronized with the vehicle's CAN bus (usually 10Hz-50Hz). An additional touch sensor is installed to record the timing of the driver's touches on the center console and steering wheel buttons.

[0031] 4) Driving scenario data collection The system utilizes onboard millimeter-wave radar, lidar, and monocular cameras to collect data on the vehicle's surrounding environment, including distance to the vehicle in front, lane line position, and distribution of pedestrians and obstacles. It also obtains scene tags such as road type (highway, urban road, rural road), weather conditions, and time information (day / night) through onboard GPS and navigation systems.

[0032] As a further optional implementation, brainwave distribution features are extracted from the brainwave data, specifically including: S201. Filter the EEG data to obtain multiple EEG component signals, including alpha wave frequency band signal, beta wave frequency band signal, gamma wave frequency band signal, delta wave frequency band signal and theta wave frequency band signal; S202. Determine the power spectral density of each EEG component signal, determine the frequency domain energy of the corresponding EEG component signal based on the power spectral density, and calculate the energy proportion of each EEG component signal based on the frequency domain energy. S203. Perform sliding window analysis on the EEG data to obtain the standard deviation, power spectrum peak position, and power spectrum peak size of each window. S204. Determine the temporal stability of EEG data based on the fluctuation of the standard deviation of each window, and determine the frequency stability of EEG data based on the offset of the peak position of the power spectrum of each serial port and the fluctuation of the peak value of the power spectrum. S205. Generate EEG distribution characteristics based on energy proportion, temporal stability, and frequency stability.

[0033] Specifically, the raw EEG signal is preprocessed by separating the alpha, beta, gamma, delta, and theta frequency bands using a Butterworth filter to remove artifacts such as electrooculography (EOG) and electromyography (EMG). The power spectral density of each frequency band is calculated to statistically determine the energy proportion of different frequency bands. The standard deviation, peak position, and peak size of each window are analyzed using a sliding window to statistically determine the temporal and frequency domain stability of the EEG data. Finally, based on the energy proportion of different frequency bands and the distribution characteristics of the temporal and frequency domain stability of the EEG data, an EEG feature vector is formed.

[0034] As a further optional implementation, facial key point features are extracted from facial image data, specifically including: S301. Perform key point detection on the facial image data to obtain the real-time key point poses of multiple facial key points. S302. Based on the real-time key point pose, determine the eye rotation angle, eye closure degree, mouth opening degree, and head pose angle to generate facial key point features.

[0035] Specifically, a facial key point detection algorithm is used to locate key points such as eyes (eyelids, pupils), eyebrows, corners of the mouth, and jawline. The real-time coordinate changes of these points are calculated, and static features such as eyeball rotation angle, eye closure degree, mouth opening degree, and head posture angle are extracted to form facial key point features.

[0036] As a further optional implementation, operation sequence features are extracted based on the operation behavior data, specifically including: S401. Extract multiple operation actions and corresponding operation times based on operation behavior data. The operation actions include turning the steering wheel, pressing the accelerator pedal, and pressing the brake pedal. S402. Sort the operation actions according to the operation time to obtain the operation action sequence; S403. Based on the operation behavior data, the change rate of steering wheel angle, the change rate of accelerator pedal opening, or the change rate of brake pedal opening corresponding to each operation action is statistically analyzed to obtain the action amplitude characteristics corresponding to each operation action. S404. Generate operation sequence features based on the operation action sequence and action amplitude features.

[0037] Specifically, multiple operation actions and their corresponding operation times are extracted from the operation behavior data, and then sorted according to the operation time order to obtain the operation action sequence; the continuous operation data is segmented by sliding window, and the frequency of change of steering wheel angle and the rate of change of accelerator / brake pedal opening within the window are statistically analyzed as the action amplitude features of the corresponding operation actions. The operation action sequence is then upgraded based on the action amplitude features to obtain the operation sequence features.

[0038] After extracting the EEG distribution features, facial key point features, and operation sequence features, the corresponding fusion weight parameters are determined according to the current driving scenario. Then, the EEG distribution features, facial key point features, and operation sequence features are fused according to the fusion weight parameters to obtain multimodal fusion features.

[0039] Specifically, a scene weight mapping table is constructed based on driving scene labels. For example, in highway scenarios, EEG and operational behavior features have higher weights (40% each), while facial features account for 20%. In urban congestion scenarios, the weight of facial features increases to 40%, while EEG and operational behavior each account for 30%. A fuzzy logic algorithm is used to dynamically adjust the weight parameters based on real-time scene data to ensure that the fusion adapts to the current driving environment. EEG distribution features, facial key point features, and operational sequence features are normalized to unify the numerical range of the feature vectors. Based on the scene-specific fusion weight parameters, a weighted summation method is used for feature fusion to obtain a multimodal fusion feature vector containing multi-dimensional information.

[0040] As an optional implementation, the driver attention recognition model is trained through the following steps: S501. Obtain the test driver's brainwave samples, facial image samples, operation behavior samples, and sample driving scenarios of the test vehicle in the test scenario. S502. Determine multimodal fusion feature samples based on EEG samples, facial image samples, operational behavior samples, and sample driving scenarios, and determine the corresponding driver attention labels through manual annotation; S503. Input the multimodal fusion feature samples into a pre-built convolutional neural network to obtain the predicted attention score; S504. Determine the loss value based on the predicted attention score and the driver attention label; S505. Update the parameters of the convolutional neural network based on the loss value to obtain the trained driver attention recognition model.

[0041] Specifically, driver data from different test scenarios is collected, and the corresponding real attention scores are determined by manual annotation as driver attention labels. Convolutional neural networks are used for sample training and validation, and finally a trained driver attention recognition model is obtained.

[0042] During the prediction phase, multimodal fusion features are input into the trained driver attention recognition model, which ultimately outputs an attention score of 0-100, with a higher score indicating more focused attention.

[0043] As a further optional implementation, a tiered warning system is provided based on the current attention score, which specifically includes: S1041. When the current attention score is greater than or equal to the preset first threshold, determine that the target driver is in a state of focused attention, and provide driving reminders at preset time intervals. S1042. When the current attention score is greater than or equal to the preset second threshold and less than the first threshold, it is determined that the target driver is in a state of slight distraction, and a real-time reminder is given through the dashboard lights. S1043. When the current attention score is less than the second threshold, it is determined that the target driver is in a state of severe inattention, and a real-time reminder is given through seat vibration and / or voice broadcast.

[0044] Specifically, attention score thresholds are preset: 80-100 points indicate focused attention, triggering warnings only at regular fixed intervals; 60-79 points indicate slightly distracted attention, triggering a Level 1 warning (instrument panel light alert, gentle voice reminder); 0-59 points indicate severely distracted attention, triggering a Level 2 warning (rapid voice warning, seat vibration alert). After a warning is triggered, the driver's status is continuously monitored, and the corresponding level of warning is automatically deactivated when the attention score rises above the threshold.

[0045] It should be noted that the setting of each threshold can be made according to the actual situation, and the embodiments of the present invention do not limit this.

[0046] As an optional implementation, the driver attention level warning method further includes the following steps: S105. When the target driver is in a state of severe inattention for a preset time, the intelligent driving system will automatically take over and determine the corresponding intelligent driving mode based on the current driving scenario. S106. When it is detected that the target driver has returned to a state of focused attention, remind the target driver to take over.

[0047] Specifically, when the target driver is in a state of severe inattention for a preset duration, the intelligent driving takeover is automatically triggered, and the intelligent driving mode is adjusted according to the current driving scenario. During the takeover process, the driver's state is monitored in real time, and once the driver returns to a state of focused attention, the driver is prompted to take over the vehicle.

[0048] The method steps of the embodiments of the present invention have been described above. It can be understood that the embodiments of the present invention extract the driver's electroencephalogram distribution features, facial key point features, and operation sequence features, perform feature fusion based on the driving scenario to obtain multimodal fusion features, and combine them with a pre-trained driver attention recognition model to identify the driver's current attention score. Thus, graded warnings can be given based on the driver's real-time attention state, thereby improving vehicle driving safety.

[0049] Reference Figure 2This invention provides a driver attention level warning device, comprising: The data acquisition module is used to acquire the target driver's electroencephalogram data, facial image data, operational behavior data, and the current driving scenario of the target vehicle; The feature extraction module is used to extract brainwave distribution features from brainwave data, facial key point features from facial image data, and operation sequence features from operation behavior data. The feature fusion module is used to determine the corresponding fusion weight parameters according to the current driving scenario, and to perform feature fusion on EEG distribution features, facial key point features and operation sequence features according to the fusion weight parameters to obtain multimodal fusion features; The graded early warning module is used to input multimodal fusion features into a pre-trained driver attention recognition model to obtain the current attention score of the target driver, and to issue graded early warnings based on the current attention score.

[0050] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0051] Reference Figure 3 This invention provides an electronic device, comprising: At least one processor; At least one memory for storing at least one program; When the above-mentioned at least one program is executed by the above-mentioned at least one processor, the above-mentioned at least one processor implements the above-mentioned driver attention level warning method.

[0052] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0053] This invention also provides a computer-readable storage medium storing a processor-executable computer program that, when executed by a processor, implements the aforementioned driver attention level warning method.

[0054] This invention provides a computer-readable storage medium that can execute a driver attention level warning method provided in the method embodiment of this invention. It can execute any combination of the implementation steps of the method embodiment and has the corresponding functions and beneficial effects of the method.

[0055] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described driver attention level warning method.

[0056] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0057] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0058] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0059] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0060] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the aforementioned blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0061] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0062] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0063] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0064] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the aforementioned program can be printed, because the aforementioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0065] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0066] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0067] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0068] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A driver attention level classification pre-warning method, characterized in that, Includes the following steps: Acquire the target driver's brainwave data, facial image data, operational behavior data, and the target vehicle's current driving scenario; Brainwave distribution features are extracted from the brainwave data, facial key point features are extracted from the facial image data, and operation sequence features are extracted from the operation behavior data. Based on the current driving scenario, the corresponding fusion weight parameters are determined, and feature fusion is performed on the EEG distribution features, facial key point features, and operation sequence features based on the fusion weight parameters to obtain multimodal fusion features; The multimodal fusion features are input into a pre-trained driver attention recognition model to obtain the current attention score of the target driver, and a graded warning is given based on the current attention score.

2. The method of claim 1, wherein, The step of extracting brainwave distribution features based on the brainwave data specifically includes: The EEG data is filtered to obtain multiple EEG component signals, including alpha wave band signals, beta wave band signals, gamma wave band signals, delta wave band signals, and theta wave band signals. The power spectral density of each of the brainwave component signals is determined, the frequency domain energy of the corresponding brainwave component signal is determined based on the power spectral density, and the energy ratio of each brainwave component signal is calculated based on the frequency domain energy. Sliding window analysis was performed on the EEG data to obtain the standard deviation, power spectrum peak position, and power spectrum peak size for each window. The temporal stability of the EEG data is determined based on the fluctuation of the standard deviation of each window, and the frequency domain stability of the EEG data is determined based on the offset of the peak position of the power spectrum of each serial port and the fluctuation of the peak value of the power spectrum. The EEG distribution features are generated based on the energy ratio, the temporal stability, and the frequency stability.

3. The method of claim 1, wherein the method further comprises: The step of extracting facial key point features based on the facial image data specifically includes: Key point detection is performed on the facial image data to obtain the real-time key point poses of multiple facial key points; Based on the real-time key point pose, the eye rotation angle, eye closure degree, mouth opening degree, and head pose angle are determined to generate the facial key point features.

4. The method of claim 1, wherein the method further comprises: The step of extracting operation sequence features based on the operation behavior data specifically includes: Based on the operational behavior data, multiple operational actions and corresponding operational times are extracted. The operational actions include turning the steering wheel, pressing the accelerator pedal, and pressing the brake pedal. The operation actions are sorted according to the operation time to obtain an operation action sequence; Based on the operational behavior data, the rate of change of steering wheel angle, the rate of change of accelerator pedal opening, or the rate of change of brake pedal opening corresponding to each operational action is statistically analyzed to obtain the action amplitude characteristics corresponding to each operational action. The operation sequence feature is generated based on the operation action sequence and the action amplitude feature.

5. The driver attention level warning method according to claim 1, characterized in that, The driver attention recognition model is trained through the following steps: Acquire brainwave samples, facial image samples, operational behavior samples of the test driver, and sample driving scenarios of the test vehicle in the test scenario. Multimodal fusion feature samples are determined based on the EEG samples, facial image samples, operational behavior samples, and sample driving scenarios, and corresponding driver attention labels are determined through manual annotation. The multimodal fusion feature samples are input into a pre-constructed convolutional neural network to obtain a predicted attention score; The loss value is determined based on the predicted attention score and the driver attention label; The parameters of the convolutional neural network are updated based on the loss value to obtain the trained driver attention recognition model.

6. A driver attention level warning method according to any one of claims 1 to 5, characterized in that, The step of providing graded warnings based on the current attention score specifically includes: When the current attention score is greater than or equal to a preset first threshold, it is determined that the target driver is in a state of focused attention, and driving reminders are given at preset time intervals; When the current attention score is greater than or equal to a preset second threshold and less than the first threshold, it is determined that the target driver is in a state of slight inattention, and a real-time reminder is given through the dashboard lights; When the current attention score is less than the second threshold, it is determined that the target driver is in a state of severe inattention, and a real-time reminder is given through seat vibration and / or voice broadcast.

7. The driver attention level warning method according to claim 6, characterized in that, The driver attention level warning method also includes the following steps: When the target driver is in a state of severe inattention for a preset period of time, the intelligent driving system will automatically take over and determine the corresponding intelligent driving mode based on the current driving scenario. When the target driver is detected to have regained focus, the target driver is alerted to take over.

8. A driver attention level warning device, characterized in that, include: The data acquisition module is used to acquire the target driver's electroencephalogram data, facial image data, operational behavior data, and the current driving scenario of the target vehicle; The feature extraction module is used to extract brainwave distribution features based on the brainwave data, extract facial key point features based on the facial image data, and extract operation sequence features based on the operation behavior data. The feature fusion module is used to determine the corresponding fusion weight parameters according to the current driving scenario, and to perform feature fusion on the EEG distribution features, the facial key point features and the operation sequence features according to the fusion weight parameters to obtain multimodal fusion features; The graded early warning module is used to input the multimodal fusion features into a pre-trained driver attention recognition model to obtain the current attention score of the target driver, and to issue graded early warnings based on the current attention score.

9. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a driver attention level warning method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements a driver attention level warning method as described in any one of claims 1 to 7.

Images