Driving state detection method and device, storage medium and computer program product
By extracting face characteristic information from images using a convolutional neural network with a specific structure, the problem of poor real-time driving state detection in the prior art is solved, and efficient and accurate driving state detection is achieved.
Patent Information
- Application Number
- CN202411885858.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-19
- Publication Date
- 2025-04-11
AI Technical Summary
The existing driving state detection method based on deep learning has poor real-time performance, which leads to the inability to detect the driver's fatigue status in time, increasing the risk of traffic accidents.
A convolutional neural network with a specific structure performs a single-stage detection method, and directly extracts face characteristic information from the image, including the backbone network composed of the convolutional SCConv module using alternately connected reserve networks and the vision transformer RMT module, space and channel reconstruction.
It realizes driving state detection with fast speed and high accuracy, which can meet the real-time requirements of driving state detection and reduces the risk of traffic accidents.
Smart Images
Figure CN119810803A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of electronic communication technology, and in particular, to a driving status detection method, device, storage medium and computer program product. Background Art
[0002] At present, the driving status detection method based on deep learning is mainly used to detect fatigue driving behavior and realize the evaluation of the driver's fatigue status.
[0003] The driving state detection method based on deep learning usually has good accuracy, but due to the problems of more model parameters and large amount of calculation, the real-time performance of the detection method is poor. If the driver's fatigue state cannot be detected in real time, the driver cannot be reminded in time or appropriate safety measures cannot be taken, which increases the risk of traffic accidents and potential safety hazards. In addition, it may not be possible to accurately judge the driver's state, resulting in misjudgment or omission, and there may be problems such as inaccurate and incomplete data, which affects the comprehensive understanding of the driver's state. Summary of the invention
[0004] The embodiments of the present application provide a driving status detection method, device, storage medium and computer program product, which use a convolutional neural network with a specific structure to execute a single-stage detection method, directly extract facial feature information from an image, have the characteristics of fast speed and high accuracy, and can meet the real-time requirements of driving status detection.
[0005] The technical solution of the embodiment of the present application is implemented as follows:
[0006] The present application provides a driving state detection method, including:
[0007] A frame of the driver's face image is collected at each first time interval to obtain N frames of face images; N is a natural number greater than 1;
[0008] Extracting facial feature information of the driver from the N frames of facial images using a pre-trained convolutional neural network; wherein the convolutional neural network comprises an input end, a backbone network and a head network connected in sequence, and the backbone network is composed of an alternately connected retention network and a visual transformer RMT module, and a space and channel reconstruction convolution SCConv module;
[0009] The facial feature information is used to determine whether the driver is in an abnormal driving state.
[0010] In the above method, collecting a frame of the driver's face image at each first time interval to obtain N frames of face images includes:
[0011] capturing a video stream containing the driver;
[0012] A frame of video image is extracted from the video stream at intervals of the first time length, and a facial region of the driver is intercepted from each frame of the video image to determine a frame of facial image, thereby obtaining the N frames of facial images.
[0013] In the above method, the extracting the facial feature information of the driver from the N frames of facial images using a pre-trained convolutional neural network includes:
[0014] Receiving the N frames of face images through the input end, and transferring the N frames of face images to the backbone network after pre-processing;
[0015] Through the backbone network, image features are extracted from the preprocessed N frames of face images and transmitted to the head network;
[0016] The feature attributes of the image features extracted by the backbone network are determined through the head network to generate the facial feature information.
[0017] In the above method, the SCConv module includes a spatial reconstruction unit SRU and a channel reconstruction unit CRU;
[0018] The SRU and CRU have the ability to limit feature redundancy and integrate geometric information into the convolution weights to capture the spatial structure of the scene;
[0019] The SRU,adopts a separation-reconstruction strategy to suppress spatial redundancy;
[0020] The CRU adopts a segmentation-conversion-fusion strategy to reduce channel redundancy.
[0021] In the above method, the backbone network further includes an efficient multi-scale attention EMA module, and one of the EMA modules is connected between one of the RMT modules and the head network;
[0022] The EMA module has the capabilities of channel dimensionality reduction, channel grouping, global information encoding and cross-dimensional interaction.
[0023] In the above method, the head network includes a maximum pooling convolution MPConv module, and the maximum pooling layer in the MPConv module is an importance-based pooling LIP layer;
[0024] The LIP layer has the ability to automatically enhance recognition features during the sampling process.
[0025] In the above method, the use of the facial feature information to determine whether the driver is in an abnormal driving state includes:
[0026] Analyzing changes in at least one of the driver's eyes, mouth, head, and expression using the facial feature information;
[0027] The analysis result is used to determine whether the driver is in at least one abnormal driving state of fatigue driving, distracted driving, and physical discomfort.
[0028] In the above method, after using the facial feature information to determine whether the driver is in an abnormal driving state, the method further includes:
[0029] When it is determined that the driver is in an abnormal driving state, collecting a frame of the driver's face image at every second time interval to obtain M frames of face images; M is a natural number greater than 1, and the second time interval is less than the first time interval;
[0030] The M frames of face images are uploaded to a cloud server, so that the cloud server uses the M frames of face images to determine whether the driver is in an abnormal driving state.
[0031] In the above method, after uploading the M frames of face images to the cloud server, the method further includes:
[0032] When receiving the warning signal sent by the cloud server, outputting a prompt message indicating that the driver is in an abnormal driving state;
[0033] Wherein, the warning signal is generated by the cloud server when it is determined that the driver is in an abnormal driving state.
[0034] The present application provides a driving state detection device, including:
[0035] An image acquisition module is used to acquire a frame of the driver's face image at each first time interval to obtain N frames of face images, where N is a natural number greater than 1;
[0036] A feature extraction module, used to extract facial feature information of the driver from the N frames of facial images using a pre-trained convolutional neural network; wherein the convolutional neural network comprises an input end, a backbone network and a head network connected in sequence, and the backbone network is composed of an alternately connected retention network and a visual transformer RMT module, and a space and channel reconstruction convolution SCConv module;
[0037] The information processing module is used to use the facial feature information to determine whether the driver is in an abnormal driving state.
[0038] The embodiment of the present application provides a driving state detection device, including: a processor, a memory, and a communication bus;
[0039] The communication bus is used to realize the communication connection between the processor and the memory;
[0040] The processor is used to execute one or more computer programs stored in the memory to implement the driving state detection method.
[0041] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements a driving state detection method when the computer program is executed by a processor.
[0042] An embodiment of the present application provides a computer program product, including a computer program, which implements a driving state detection method when the computer program is executed.
[0043] The embodiments of the present application provide a driving state detection method, device, storage medium and computer program product, the method comprising: collecting a frame of facial image of the driver at each first time interval to obtain N frames of facial images; N is a natural number greater than 1; using a pre-trained convolutional neural network to extract facial feature information of the driver from the N frames of facial images; wherein the convolutional neural network comprises a sequentially connected input end, a backbone network and a head network, and the backbone network is composed of an alternately connected retention network and a visual transformer RMT module, and a space and channel reconstruction convolution SCConv module; using facial feature information to determine whether the driver is in an abnormal driving state, the technical solution provided by the embodiments of the present application uses a convolutional neural network with a specific structure to execute a single-stage detection method to directly extract facial feature information from the image, which has the characteristics of fast speed and high accuracy, and can meet the real-time requirements of driving state detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 A flowchart of a driving state detection method provided in an embodiment of the present application;
[0045] Figure 2 A schematic diagram of the structure of an exemplary convolutional neural network provided in an embodiment of the present application;
[0046] Figure 3 A schematic diagram of the structure of an exemplary backbone network provided in an embodiment of the present application;
[0047] Figure 4 A schematic diagram of the structure of an exemplary RMT module provided in an embodiment of the present application;
[0048] Figure 5 A schematic diagram of the structure of an exemplary MPConv module provided in an embodiment of the present application;
[0049] Figure 6 A schematic diagram of a driving state detection device provided in an embodiment of the present application Figure 1 ;
[0050] Figure 7 A schematic diagram of a driving state detection device provided in an embodiment of the present application Figure 2 . DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0052] The following will specifically describe the technical solution of the present application and how the technical solution of the present application solves the above technical problems through embodiments and in conjunction with the accompanying drawings. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.
[0053] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.
[0054] The embodiment of the present application provides a driving state detection method, which is implemented by a driving state detection device, which may include a camera for collecting videos and images. The driving state detection device can be deployed in front of the cockpit of the vehicle to appropriately alarm when an abnormal driving state occurs, reminding the driver to take a rest, stay focused, or change drivers.
[0055] Figure 1 A flowchart of a driving state detection method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, in an embodiment of the present application, the driving state detection method mainly includes the following steps:
[0056] S101. Collect a frame of the driver's face image at every first time interval to obtain N frames of face images, where N is a natural number greater than 1.
[0057] In an embodiment of the present application, the driving status detection device can collect a frame of facial image of the driver at every first time interval to obtain N frames of facial images; N is a natural number greater than 1.
[0058] It should be noted that in the embodiments of the present application, the values of the first duration and N can be set according to actual needs and application scenarios, and are not limited in the embodiments of the present application.
[0059] In an embodiment of the present application, a facial image frame of the driver is collected at every first time interval to obtain N facial image frames, including: collecting a video stream containing the driver; extracting a video image frame from the video stream at every first time interval, and intercepting the driver's facial area from each video image frame to determine it as a facial image frame, to obtain N facial image frames.
[0060] It can be understood that in the embodiments of the present application, the driving status detection device can use a camera to collect a video stream containing the driver, that is, complete video information, and extract a video image from it at intervals of a first time length, and then further locate the specific position of the driver's face therein, that is, the face area, and cut it out from the video image as a face image.
[0061] It should be noted that, in the embodiments of the present application, the above-mentioned method of extracting N frames of facial images from the video stream is only an optional method. Of course, other methods can also be used to achieve N frames of facial images. For example, the camera can be directly controlled to capture a frame of driving image of the driver every first time interval, and then the facial area is extracted from it to be determined as a facial image. The embodiments of the present application are not limited to this.
[0062] S102. Extract the driver's facial feature information from N frames of facial images using a pre-trained convolutional neural network; wherein the convolutional neural network includes a sequentially connected input end, a backbone network, and a head network, and the backbone network is composed of an alternately connected retention network and a visual transformer RMT module, and a spatial and channel reconstruction convolution SCConv module.
[0063] In an embodiment of the present application, after collecting N frames of facial images, the driving status detection device uses a pre-trained convolutional neural network to extract the driver's facial feature information from the N frames of facial images.
[0064] It should be noted that, in an embodiment of the present application, the driving device detection can input N frames of face images into a convolutional neural network in sequence, and the convolutional neural network extracts the driver's facial feature information therefrom.
[0065] In the examples of this application, see Figure 2 The convolutional neural network is composed of a sequentially connected input terminal, a backbone network, and a head network. The driving state detection device uses a pre-trained convolutional neural network to extract the driver's facial feature information from N frames of facial images, including: receiving N frames of facial images through the input terminal, pre-processing the N frames of facial images and passing them to the backbone network; extracting image features from the pre-processed N frames of facial images through the backbone network and passing them to the head network; determining the feature attributes of the image features extracted by the backbone network through the head network to generate facial feature information.
[0066] It should be noted that in the embodiment of the present application, the function of the input end is to receive data not input from the outside and pass it to the next layer of the network, that is, the backbone network. The input end can align, adjust, denoise and enhance N frames of face images before passing them. At the same time, in order to solve the problem that the detection accuracy may be reduced in scenes where the light intensity changes too quickly, High Dynamic Range Imaging (HDR) technology is used to ensure that the light is not too strong or too weak, thereby improving the image quality and detection accuracy. Figure 2 and Figure 3 The backbone network is mainly composed of alternately connected Retentive Networks Meet Vision Transformer (RMT) modules and Spatial and Channel reconstruction Convolution (SCConv) modules. Different RMT modules are used in different stages to extract image features. L1 and other numbers represent specific quantities. The head network is composed of multiple modules such as Spatial Pyramid Pooling (SPPCSPC), Extended Efficient Layer Aggregation Network (E-ELAN), Planned re-parameterized convolution (RepConv) and upsampling (UPSample).
[0067] It should be noted that the Transformer network first appeared in the field of natural language processing and also has excellent performance in visual tasks. Inspired by the retention network (RetNet), the RMT module of the backbone network can be a combination of RetNet and Transformer, introducing explicit attenuation and introducing prior knowledge related to spatial distance. The RMT module is expanded from unidirectional to bidirectional based on RetNet. Considering the characteristics of two-dimensional space, an attenuation matrix based on two-dimensional distance is also designed. In order to solve the problem of high computational load in the early stages of the visual backbone (Backbone), the RMT module decomposes the two-dimensional calculation process into two coordinate axes along the image. In this application, the RMT module is used as the backbone network of the convolutional neural network to enhance the feature extraction capability of the network. The overall structure of the backbone network is as shown in the figure. Figure 2 and Figure 3 As shown, the structure of the RMT module is shown in Figure 4, consists of 3×3 depthwise convolution (DWConv), normalization (Layer Norm, LN), retention self-attention (ReSA) mechanism, and feedforward neural network (FFN).
[0068] It should be noted that in deep learning, feature redundancy may cause the model to be overly sensitive to certain specific features, so that a small change in the input data may lead to a huge difference in the output result, thereby affecting the stability of the convolutional neural network. Therefore, in an embodiment of the present application, the SCConv module is introduced in the backbone network to further enhance its feature extraction capability. This module utilizes two components: the spatial reconstruction unit (Spatial Reconstruction Unit, SRU) and the channel reconstruction unit (Channel Reconstruction Unit, CRU). SRU suppresses spatial redundancy through a separation-reconstruction method. CRU adopts a segmentation-conversion-fusion strategy to reduce channel redundancy. It can not only effectively limit feature redundancy, but also establish the connection between the potential spatial relationship between the convolution weights and their corresponding pixels, and integrate geometric information into the convolution weights to better capture the spatial structure of the scene.
[0069] It should be noted that in various computer vision tasks, the channel attention mechanism has been shown to be significantly effective in producing clearer feature representations. This mechanism focuses better on task-related features by assigning different weights to input channels to emphasize or suppress specific types of information. However, a possible side effect is that modeling cross-channel relationships through channel dimensionality reduction may have a negative impact on extracting deep visual representations. To address this problem, in an embodiment of the present application, an efficient multi-scale attention (EMA) module can also be added to the backbone network, an EMA module connected to an RMT module, and between the head networks. The main goal of this module is to retain information on each channel and reduce computational overhead. To achieve this goal, the EMA module adopts a unique channel reshaping method. It not only retains the information of the original channel but also reduces the computational complexity by reshaping part of the channel into batch dimensions. In addition, the channel dimension is grouped into multiple sub-features, which makes the spatial semantic features evenly distributed in each feature group. The EMA module achieves the goals of reducing computational complexity, retaining information richness, improving spatial information capture capabilities, and enhancing model generalization capabilities through technologies such as channel dimensionality reduction, channel grouping, global information encoding, and cross-dimensional interaction.
[0070] It should be noted that the maximum pooling layer has some shortcomings and deficiencies. For example, the maximum pooling may destroy the structure of the input data to some extent, resulting in the loss of feature information. In addition, if the window size of the maximum pooling is not selected properly, it may reduce the robustness of the model to the input features and make the model more susceptible to noise and outliers. Therefore, in the embodiment of the present application, it is proposed to use the local importance-based pooling (Local Importance-based Pooling) for the maximum pooling convolution (Max PoolingConvolution, MPConv) in the head network. ， LIP) layer to replace its original maximum pooling layer. It can automatically enhance the recognition features during the sampling process by learning adaptive importance weights based on the input. It proposes a unified downsampling layer framework called local aggregation and normalization from the perspective of local importance. For a given input feature map I, a set of kernel indices Ω containing relative sampling positions (Δx, Δy) in a sliding window, the upper left corner position coordinates (x, y) of the sliding window corresponding to the feature map I and its corresponding output position (x 1 ,y 1 ), the formula for the LAN framework is as follows: Where F(I) represents the importance map, the size is the same as the input feature map I, and F(I) ≥ 0. Pooling can be regarded as the weighted sum of each window, where the weight represents the importance of local normalization. The importance function can be expressed as: in is a learnable network, also known as a logarithmic module, in order to make the importance weights non-negative and easy to optimize, so Add exp(·) operation. From the above formula, it can be concluded that the LIP layer can be expressed as The MPConv module structure of this application is as follows Figure 5 As shown, it consists of a LIP layer, a convolution (Convolution), a batch normalization (BatchNorm), an activation (SiLU) layer, that is, a CBS layer.
[0071] It should be noted that in the embodiments of the present application, the specific structure and connection relationship of the input end, backbone network, and head network in the above-mentioned convolutional neural network can be set according to actual needs and application scenarios. The relevant drawings are only exemplary illustrations and are not limited to the specific embodiments of the present application.
[0072] S103: Using facial feature information, determine whether the driver is in an abnormal driving state.
[0073] In an embodiment of the present application, after the driving state detection device uses a convolutional neural network to extract the driver's facial feature information from N frames of facial images, it can use the facial feature information to determine whether the driver is in an abnormal driving state.
[0074] In an embodiment of the present application, a driving state detection device uses facial feature information to determine whether the driver is in an abnormal driving state, including: using facial feature information to analyze changes in at least one of the driver's eyes, mouth, head, and expressions; using the analysis results to determine whether the driver is in at least one abnormal driving state of fatigue driving, distracted driving, and physical discomfort.
[0075] It should be noted that in the embodiments of the present application, the determination of the abnormal driving state of the driver is mainly obtained by analyzing the facial feature information. The changes in at least one of the driver's eyes, mouth, head and expression can be used as the basis for determining the abnormal driving state. Of course, it can also be combined with other characteristics of the driver, such as body movements, body postures, etc. Abnormal driving states may include: fatigue driving, distracted driving, and physical discomfort. For example, by analyzing the driver's facial features, such as the degree of eye closure, head posture, etc., such as the number of yawns and frequent blinks per unit time, it can be detected whether the driver is in fatigue driving; by analyzing the driver's eye direction, head posture, combined with hand movements, such as holding a mobile phone, adjusting the audio, etc., it can be determined whether the driver is in a distracted driving state. By analyzing facial features, body postures and head postures, it can be detected whether the driver is in physical discomfort. For example, the indicators of the judgment standard can be quantified as:
[0076] The number of blinks counted in one minute is greater than 30;
[0077] The number of blinks counted in one minute is less than 6;
[0078] The number of yawns counted in one minute is greater than 2 times;
[0079] In addition, the driver's facial expression can be used to determine whether the driver is in an abnormal driving state. If the driver shows expressions such as frowning or pain, and his hands are off the steering wheel and pressed on the head or torso, it can be determined that the driver is in a state of physical discomfort and timely warnings or interventions should be given to stop abnormal driving behavior.
[0080] In an embodiment of the present application, after the driving state detection device uses facial feature information to determine whether the driver is in an abnormal driving state, the following steps may be performed: when it is determined that the driver is in an abnormal driving state, a frame of facial image of the driver is collected at every second time interval to obtain M frames of facial images; M is a natural number greater than 1, and the second time interval is less than the first time interval; the M frames of facial images are uploaded to the cloud server, so as to use the M frames of facial images through the cloud server to determine whether the driver is in an abnormal driving state.
[0081] It is understandable that in the embodiment of the present application, if the driving state detection device determines that the driver is in an abnormal driving state, the time for collecting facial images can be shortened, and facial images can be collected at intervals of a second time length. In this way, the abnormal driving state can be judged more accurately. The second time length is less than the first time length, and the specific second time length is not limited in the embodiment of the present application. In addition, the value of M is also not limited. The method for collecting M frames of facial images is similar to the method for collecting N frames of facial images described above, and will not be repeated here.
[0082] It is understandable that in the embodiments of the present application, the cloud server usually has strong computing power, the model specifications deployed by it are larger, and the accuracy of abnormal driving state judgment is higher. The driving state detection can upload the collected M frames of face images to the cloud server, and the cloud server performs intensive detection. Among them, the judgment method of the cloud server, and the model or neural network used can be similar to the convolutional neural network used by the driving state detection device, but its specifications and structure are usually more complex and more accurate. Of course, the cloud server can also use other more accurate algorithms or models to judge abnormal driving states, which is not limited by the embodiments of the present application.
[0083] In an embodiment of the present application, after the driving status detection device uploads M frames of facial images to the cloud server, the following steps can also be performed: upon receiving an alarm signal sent by the cloud server, outputting a prompt message to remind the driver that he is in an abnormal driving state; wherein, the alarm signal is generated by the cloud server when it is determined that the driver is in an abnormal driving state.
[0084] It is understandable that in the embodiment of the present application, when the cloud server determines that the driver is in an abnormal driving state, it can send an alarm signal to the driving state detection device to notify the driving device detection device that the driver is indeed in an abnormal driving state. Based on this, the driving device detection device can input prompt information to reduce the incidence of accidents. The role of introducing the cloud server here is to ensure the accuracy of the abnormal driving state judgment. Of course, the driving state detection device can also directly output prompt information when it determines that the driver is in an abnormal driving state, and the embodiment of the present application is not limited.
[0085] The technical solution provided in the embodiment of the present application detects the driver's abnormal driving state by setting a driving state detection device in front of the vehicle's cockpit, and pushes reminders or alarms in time, thereby protecting the driver's physical health and life safety. The convolutional neural network used therein can execute a single-stage detection method, directly extract feature positions and identify feature attributes from the image, and the detection speed is faster than the two-stage detection algorithm, and the real-time performance is also higher, which can effectively help the driver reduce the occurrence of abnormal driving states during driving.
[0086] An embodiment of the present application also provides a driving status detection device. Figure 6 A schematic diagram of a driving state detection device provided in an embodiment of the present application Figure 1 .like Figure 6 As shown, in an embodiment of the present application, the driving state detection device 1 includes:
[0087] The image acquisition module 11 is used to acquire a frame of the driver's face image at each first time interval to obtain N frames of face images, where N is a natural number greater than 1;
[0088] A feature extraction module 12, used to extract the facial feature information of the driver from the N frames of facial images using a pre-trained convolutional neural network; wherein the convolutional neural network includes an input end, a backbone network and a head network connected in sequence, and the backbone network is composed of an alternately connected retention network and a visual transformer RMT module, and a space and channel reconstruction convolution SCConv module;
[0089] The information processing module 13 is used to use the facial feature information to determine whether the driver is in an abnormal driving state.
[0090] In one embodiment of the present application, the image acquisition module 11 is used to acquire a video stream containing the driver; extract a frame of video image from the video stream at each interval of the first time length, and capture the driver's face area from each frame of the video image to determine it as a frame of face image, thereby obtaining the N frames of face images.
[0091] In one embodiment of the present application, the feature extraction module 12 is used to receive the N frames of facial images through the input end, pre-process the N frames of facial images and pass them to the backbone network; extract image features from the pre-processed N frames of facial images through the backbone network and pass them to the head network; determine the feature attributes of the image features extracted by the backbone network through the head network to generate the facial feature information.
[0092] In one embodiment of the present application, the SCConv module includes a spatial reconstruction unit SRU and a channel reconstruction unit CRU;
[0093] The SRU and CRU have the ability to limit feature redundancy and integrate geometric information into the convolution weights to capture the spatial structure of the scene;
[0094] The SRU,adopts a separation-reconstruction strategy to suppress spatial redundancy;
[0095] The CRU adopts a segmentation-conversion-fusion strategy to reduce channel redundancy.
[0096] In one embodiment of the present application, the backbone network further includes an efficient multi-scale attention EMA module, and one of the EMA modules is connected between one of the RMT modules and the head network;
[0097] The EMA module has the capabilities of channel dimensionality reduction, channel grouping, global information encoding and cross-dimensional interaction.
[0098] In one embodiment of the present application, the head network includes a maximum pooling convolution MPConv module, and the maximum pooling layer in the MPConv module is an importance-based pooling LIP layer;
[0099] The LIP layer has the ability to automatically enhance recognition features during the sampling process.
[0100] In one embodiment of the present application, the information processing module 13 is used to use the facial feature information to analyze changes in at least one of the driver's eyes, mouth, head, and expressions; and use the analysis results to determine whether the driver is in at least one abnormal driving state of fatigue driving, distracted driving, and physical discomfort.
[0101] In one embodiment of the present application, the information processing module 13 is also used to collect a frame of facial image of the driver at every second time interval to obtain M frames of facial images when it is determined that the driver is in an abnormal driving state; M is a natural number greater than 1, and the second time interval is less than the first time interval; and the M frames of facial images are uploaded to a cloud server, so as to determine whether the driver is in an abnormal driving state by using the M frames of facial images through the cloud server.
[0102] In one embodiment of the present application, the information processing module 13 is also used to output a prompt message indicating that the driver is in an abnormal driving state when receiving an alarm signal sent by the cloud server; wherein, the alarm signal is generated by the cloud server when it is determined that the driver is in an abnormal driving state.
[0103] Based on the same inventive concept, Figure 7 A schematic diagram of a driving state detection device provided in an embodiment of the present application Figure 2.like Figure 7 As shown, in the embodiment of the present application, the driving state detection device 1 includes: a processor 14, a memory 15 and a communication bus 16;
[0104] The communication bus 16 is used to realize the communication connection between the processor 14 and the memory 15;
[0105] The processor 14 is used to execute one or more computer programs stored in the memory 15 to implement the driving state detection method.
[0106] An embodiment of the present application provides a computer program product, including a computer program, which implements a driving state detection method when the computer program is executed.
[0107] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and the computer program implements a driving state detection method when executed by a processor. The computer-readable storage medium can be a volatile memory (volatile memory), such as a random access memory (Random-Access Memory, RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (Read-Only Memory, ROM), a flash memory (flash memory), a hard disk (Hard Disk Drive, HDD) or a solid-state drive (SSD); or it can be a respective device including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc.
[0108] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.
[0109] The present application is described with reference to implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0110] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing the steps in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0112] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the utility application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A driving state detection method, characterized in that: The method comprises: A frame of the driver's face image is collected at each first time interval to obtain N frames of face images; N is a natural number greater than 1; Extracting facial feature information of the driver from the N frames of facial images using a pre-trained convolutional neural network; wherein the convolutional neural network comprises an input end, a backbone network and a head network connected in sequence, and the backbone network is composed of an alternately connected retention network and a visual transformer RMT module, and a space and channel reconstruction convolution SCConv module; The facial feature information is used to determine whether the driver is in an abnormal driving state.
2. The method according to claim 1, characterized in that The step of collecting a frame of the driver's face image at each first time interval to obtain N frames of face images includes: capturing a video stream containing the driver; A frame of video image is extracted from the video stream at intervals of the first time length, and a facial region of the driver is intercepted from each frame of the video image to determine a frame of facial image, thereby obtaining the N frames of facial images.
3. The method according to claim 1, characterized in that The method of extracting facial feature information of the driver from the N frames of facial images using a pre-trained convolutional neural network includes: Receiving the N frames of face images through the input end, and transferring the N frames of face images to the backbone network after pre-processing; Through the backbone network, image features are extracted from the preprocessed N frames of face images and transmitted to the head network; The feature attributes of the image features extracted by the backbone network are determined through the head network to generate the facial feature information.
4. The method according to claim 1, characterized in that: The SCConv module includes a spatial reconstruction unit SRU and a channel reconstruction unit CRU; The SRU and CRU have the ability to limit feature redundancy and integrate geometric information into the convolution weights to capture the spatial structure of the scene; The SRU,adopts a separation-reconstruction strategy to suppress spatial redundancy; The CRU adopts a segmentation-conversion-fusion strategy to reduce channel redundancy.
5. The method according to claim 1, characterized in that The backbone network also includes an efficient multi-scale attention EMA module, one of the EMA modules is connected between one of the RMT modules and the head network; The EMA module has the capabilities of channel dimensionality reduction, channel grouping, global information encoding and cross-dimensional interaction.
6. The method according to claim 1, characterized in that The head network includes a maximum pooling convolution MPConv module, and the maximum pooling layer in the MPConv module is an importance-based pooling LIP layer; The LIP layer has the ability to automatically enhance recognition features during the sampling process.
7. The method according to claim 1, characterized in that The using the facial feature information to determine whether the driver is in an abnormal driving state includes: Analyzing changes in at least one of the driver's eyes, mouth, head, and expression using the facial feature information; The analysis result is used to determine whether the driver is in at least one abnormal driving state of fatigue driving, distracted driving, and physical discomfort.
8. The method according to claim 1, characterized in that After using the facial feature information to determine whether the driver is in an abnormal driving state, the method further includes: When it is determined that the driver is in an abnormal driving state, collecting a frame of the driver's face image at every second time interval to obtain M frames of face images; M is a natural number greater than 1, and the second time interval is less than the first time interval; The M frames of face images are uploaded to a cloud server, so that the cloud server uses the M frames of face images to determine whether the driver is in an abnormal driving state.
9. The method according to claim 8, characterized in that After uploading the M frames of face images to the cloud server, the method further includes: When receiving the warning signal sent by the cloud server, outputting a prompt message to prompt the driver that he is in an abnormal driving state; Wherein, the warning signal is generated by the cloud server when it is determined that the driver is in an abnormal driving state.
10. A driving state detection device, characterized in that: include: An image acquisition module is used to acquire a frame of the driver's face image at each first time interval to obtain N frames of face images, where N is a natural number greater than 1; A feature extraction module, used to extract facial feature information of the driver from the N frames of facial images using a pre-trained convolutional neural network; wherein the convolutional neural network comprises an input end, a backbone network and a head network connected in sequence, and the backbone network is composed of an alternately connected retention network and a visual transformer RMT module, and a space and channel reconstruction convolution SCConv module; The information processing module is used to use the facial feature information to determine whether the driver is in an abnormal driving state.
11. A driving state detection device, characterized in that: include: processor, memory, and communications bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute one or more computer programs stored in the memory to implement the driving state detection method described in any one of claims 1-10.
12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the driving state detection method as described in any one of claims 1-10 is implemented.
13. A computer program product, comprising a computer program, characterized in that When the computer program is executed, the driving state detection method as described in any one of claims 1-10 is implemented.