Driver driving state detection method and device, storage medium, equipment and vehicle

Abstract

The invention discloses a driver driving state detection method and device, a storage medium, equipment and a vehicle. The method comprises the steps that DMS signal features and image features in multiple frames of face images are detected based on a DMS signal feature detection model; splicing and coding the plurality of DMS signal features to obtain an initial DMS signal feature matrix, and splicing and coding the plurality of image features to obtain an image feature matrix; sequentially executing the plurality of attention fusion models to obtain a target attention fusion feature, and determining a driving state based on the fusion feature; the execution process of the attention fusion model comprises the steps of obtaining a current self-attention fusion feature matrix based on a self-attention mechanism and a current DMS signal feature matrix; obtaining a current cross attention fusion feature matrix based on a cross attention mechanism, the current self-attention signal feature matrix and the image feature matrix; and obtaining a new current DMS signal feature matrix based on the feedforward neural network and the current cross attention fusion feature matrix.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, in particular to a driver driving state detection method and device, a storage medium, equipment and a vehicle. BACKGROUND

[0002] DMS (Driver Monitoring System) is a real-time system based on driver face image processing to study the state of the driver, including fatigue detection, distraction detection, expression recognition, gesture recognition, dangerous action recognition, gaze tracking, etc.

[0003] When detecting that the driver has abnormal driving behaviors such as closing eyes, yawning, and lowering head, the DMS system gives a voice alarm to remind the driver to pay attention to driving safety and correct the abnormal driving behavior or park for rest. However, the current DMS system mainly determines the state of the driver based on the DMS basic signals (such as head posture, eyelid distance, and mouth distance) contained in a single frame of face image, which is easy to cause errors in state detection due to the blocking of the camera by obstacles in front of the face, or the state detection error due to the inconsistency between the instantaneous state and the real state of the driver. SUMMARY

[0004] The present application provides a driver driving state detection method, device, storage medium, equipment and vehicle, which can solve the problem of errors in detecting the state of the driver based on a single frame of face image.

[0005] The specific technical solutions are as follows:

[0006] In a first aspect, the present application provides a driver driving state detection method, which comprises:

[0007] detecting at least one DMS signal feature of each frame of face image and at least one image feature of each frame of face image in multiple frames of face image based on a driver monitoring system DMS signal feature detection model;

[0008] splicing multiple DMS signal features of multiple frames of face image, encoding the splicing result to obtain an initial DMS signal feature matrix, and splicing multiple image features of multiple frames of face image, and encoding the splicing result to obtain an image feature matrix;

[0009] sequentially executing multiple attention fusion models to obtain a target attention fusion feature;

[0010] determining the driving state of the driver based on the target attention fusion feature;

[0011] The execution process of the attention fusion model includes:

[0012] The current self-attention fusion feature matrix is obtained based on the self-attention mechanism to analyze the correlation between the DMS signal features in the current DMS signal feature matrix, wherein the current DMS signal feature matrix input when the first attention fusion model is executed is the initial DMS signal feature matrix;

[0013] The current cross-attention fusion feature matrix is obtained based on the cross-attention mechanism to analyze the correlation between the current self-attention signal feature matrix and the image feature matrix;

[0014] The current cross-attention fusion feature matrix is subjected to convolution calculation based on the feedforward neural network, and the convolution calculation result is taken as a new current DMS signal feature matrix.

[0015] It can be known from the above solution that the embodiments of the present application can first detect at least one DMS signal feature of each frame of face image in multiple frames of face image and at least one image feature of each frame of face image based on a DMS signal feature detection model, then splice multiple DMS signal features of the multiple frames of face image, encode the spliced result to obtain an initial DMS signal feature matrix, splice multiple image features of the multiple frames of face image, encode the spliced result to obtain an image feature matrix, sequentially execute multiple attention fusion models containing self-attention mechanism, cross-attention mechanism and feedforward neural network to obtain a target attention fusion feature, and finally determine the driving state of the driver based on the target attention fusion feature. It can be known that the embodiments of the present application can model the correlation between the time-series DMS basic signals through the self-attention mechanism, model the correlation between the time-series DMS basic signals and the high-dimensional image features extracted from the image through the cross-attention mechanism, and the fusion of the two will make the detected driver state more rich and accurate.

[0016] In a first possible implementation manner of the first aspect, when the DMS signal feature detection model includes a convolutional neural network model and a three-dimensional face reconstruction model, the at least one DMS signal feature of each frame of face image in the multiple frames of face image and the at least one image feature of each frame of face image are detected based on a driver monitoring system DMS signal feature detection model, and the detection includes:

[0017] The two-dimensional face key points and the first DMS signal feature of each frame of face image in the multiple frames of face image are detected based on the convolutional neural network model, and at least one image feature after the convolutional neural network model is subjected to convolution calculation on the multiple frames of face image is obtained, wherein the first DMS signal feature includes head posture and / or line-of-sight vector;

[0018] obtaining a second DMS signal feature based on the three-dimensional face reconstruction model and the two-dimensional face key points, wherein the second DMS signal feature comprises at least one of a face mouth opening distance, an eyelid distance, an expression coefficient, and an identity coefficient, the face mouth opening distance and the eyelid distance are calculated based on the three-dimensional face key points determined based on the three-dimensional face reconstruction model and the two-dimensional face key points, and the first DMS signal feature and the second DMS signal feature are DMS signal features in the at least one DMS signal feature.

[0019] According to the above scheme, the embodiment of the present application first detects the two-dimensional face key points, the head posture, and the line-of-sight vector of the face image through the convolutional neural network model, then obtains the three-dimensional face key points, the expression coefficient, and the identity coefficient based on the three-dimensional face reconstruction model and the two-dimensional face key points, and at least one image feature after convolutional calculation of the convolutional neural network model on the plurality of face images, and finally obtains the DMS signal features such as the face mouth opening distance and the eyelid distance based on the three-dimensional face key points, so that various three-dimensional DMS signal features can be obtained, and the accuracy of the DMS signal features can be improved.

[0020] In a second possible implementation manner of the first aspect, splicing the plurality of DMS signal features of the plurality of face images comprises: respectively splicing each DMS signal feature in each face image to obtain a DMS signal splicing feature of each face image, and splicing the DMS signal splicing features of the plurality of face images in time sequence to obtain a splicing result of the plurality of DMS signal features of the plurality of face images.

[0021] And / or, splicing the plurality of image features of the plurality of face images comprises: respectively splicing each image feature in each face image to obtain an image splicing feature of each face image, and splicing the image splicing features of the plurality of face images in time sequence to obtain a splicing result of the plurality of image features of the plurality of face images.

[0022] In a third possible implementation manner of the first aspect, the Query, the Key, and the Value in the self-attention mechanism are all derived from the DMS signal feature matrix.

[0023] And / or, the Query in the cross-attention mechanism is derived from the self-attention fusion feature matrix, and the Key and the Value in the cross-attention mechanism are both derived from the image feature matrix.

[0024] In a fourth possible implementation manner of the first aspect, determining the driving state of the driver based on the target attention fusion feature comprises:

[0025] performing a first pooling operation on the target attention fusion feature of T*D dimensions to obtain a target attention fusion feature of D dimensions, where T represents a time length of the multiple frames of the face images, and D represents a feature length of a single frame of the face images;

[0026] processing the target attention fusion feature of D dimensions based on a multi-layer perception (MLP) to determine whether the driver is in a fatigue state; and / or processing the target attention fusion feature of D dimensions based on the MLP to determine an expression of the driver.

[0027] It can be learned from the above solution that the embodiments of the present application can convert the target attention fusion feature of T*D dimensions into the target attention fusion feature of D dimensions through the first pooling operation, so that the target attention fusion feature is more focused on the feature itself, and then whether the driver is in a fatigue state and / or the expression of the driver can be determined.

[0028] In a fifth possible implementation manner of the first aspect, determining the driving state of the driver based on the target attention fusion feature comprises:

[0029] performing a second pooling operation on the target attention fusion feature of T*D dimensions to obtain a target attention fusion feature of T dimensions, where T represents a time length of the multiple frames of the face images, and D represents a feature length of a single frame of the face images;

[0030] processing the target attention fusion feature of T dimensions based on an MLP to obtain a blood volume pulse (BVP) waveform, and determining a heart rate and / or a respiration rate of the driver by analyzing the BVP waveform.

[0031] It can be learned from the above solution that the embodiments of the present application can convert the target attention fusion feature of T*D dimensions into the target attention fusion feature of T dimensions through the second pooling operation, so that the target attention fusion feature is more focused on the time dimension, and then the change of the driver in the time dimension can be determined, that is, the heart rate and / or the respiration rate of the driver can be determined.

[0032] In a second aspect, the embodiments of the present application provide a driver driving state detection device, which comprises:

[0033] a detection unit configured to detect at least one DMS signal feature of each frame of the face images and at least one image feature of each frame of the face images based on a driver monitoring system (DMS) signal feature detection model.

[0034] The encoding unit is configured to splice the plurality of DMS signal features of the plurality of frames of the face images and encode the spliced result to obtain an initial DMS signal feature matrix, and splice the plurality of image features of the plurality of frames of the face images and encode the spliced result to obtain an image feature matrix;

[0035] The execution unit is configured to sequentially execute a plurality of attention fusion models to obtain a target attention fusion feature.

[0036] The determination unit is configured to determine a driving state of the driver based on the target attention fusion feature.

[0037] The execution unit includes:

[0038] The self-attention analysis module is configured to analyze, based on a self-attention mechanism, a correlation between the DMS signal features in the current DMS signal feature matrix to obtain a current self-attention fusion feature matrix, wherein the current DMS signal feature matrix input when the first attention fusion model is executed is the initial DMS signal feature matrix.

[0039] The cross-attention analysis module is configured to analyze, based on a cross-attention mechanism, a correlation between the current self-attention signal feature matrix and the image feature matrix to obtain a current cross-attention fusion feature matrix.

[0040] The convolution calculation module is configured to perform convolution calculation on the current cross-attention fusion feature matrix based on a feedforward neural network, and take the convolution calculation result as a new current DMS signal feature matrix.

[0041] In a first possible implementation manner of the second aspect, the detection unit includes:

[0042] The detection module is configured to, when the DMS signal feature detection model includes a convolutional neural network model and a three-dimensional face reconstruction model, detect, based on the convolutional neural network model, a two-dimensional face key point and a first DMS signal feature of each of the plurality of frames of the face images, and obtain at least one image feature after the convolutional neural network model performs convolution calculation on the plurality of frames of the face images, wherein the first DMS signal feature includes a head pose and / or a line-of-sight vector.

[0043] The first determining module is configured to obtain a second DMS signal feature based on the three-dimensional face reconstruction model and the two-dimensional face key points, wherein the second DMS signal feature comprises at least one of a face mouth opening distance, an eyelid distance, an expression coefficient and an identity coefficient, the face mouth opening distance and the eyelid distance are calculated based on three-dimensional face key points determined based on the three-dimensional face reconstruction model and the two-dimensional face key points, and the first DMS signal feature and the second DMS signal feature are DMS signal features in the at least one DMS signal feature.

[0044] In a second possible implementation manner of the second aspect, the encoding unit comprises a first splicing module and / or a second splicing module.

[0045] The first splicing module is configured to splice the DMS signal features of the plurality of face images in a plurality of frames, including respectively splicing each DMS signal feature in each face image to obtain a DMS signal splicing feature of each face image, and splicing the DMS signal splicing features of the plurality of face images in a plurality of frames in a time sequence to obtain a splicing result of the DMS signal features of the plurality of face images in a plurality of frames.

[0046] The second splicing module is configured to splice the image features of the plurality of face images in a plurality of frames, including respectively splicing each image feature in each face image to obtain an image splicing feature of each face image, and splicing the image splicing features of the plurality of face images in a plurality of frames in a time sequence to obtain a splicing result of the image features of the plurality of face images in a plurality of frames.

[0047] In a third possible implementation manner of the second aspect, the Query, the Key and the Value in the self-attention mechanism are all derived from a DMS signal feature matrix.

[0048] And / or, the Query in the cross-attention mechanism is derived from a self-attention fusion feature matrix, and the Key and the Value in the cross-attention mechanism are both derived from an image feature matrix.

[0049] In a fourth possible implementation manner of the second aspect, the determining unit comprises:

[0050] A first pooling module is configured to perform a first pooling operation on the target attention fusion feature of T*D dimensions to obtain a target attention fusion feature of D dimensions, wherein T represents a time length of the plurality of face images in a plurality of frames, and D represents a feature length of a single face image.

[0051] a second determining module, configured to determine whether the driver is in a fatigue state based on processing the target attention fusion feature of the D dimension by a multi-layer perception (MLP), and / or determine the expression of the driver based on processing the target attention fusion feature of the D dimension by the MLP.

[0052] In a fifth possible implementation manner of the second aspect, the determining unit comprises:

[0053] a second pooling module, configured to perform a second pooling operation on the target attention fusion feature of the T*D dimension to obtain a target attention fusion feature of a T dimension, where T represents a length of the plurality of frames of the face images, and D represents a feature length of a single frame of the face images;

[0054] a processing module, configured to obtain a blood volume pulse (BVP) waveform by processing the target attention fusion feature of the T dimension based on an MLP;

[0055] a third determining module, configured to determine a heart rate and / or a respiration rate of the driver by analyzing the BVP waveform.

[0056] According to the above scheme, the embodiments of the present application can first detect at least one DMS signal feature of each frame of face image in the plurality of frames of face images and at least one image feature of each frame of face image based on a DMS signal feature detection model, then splice the plurality of DMS signal features of the plurality of frames of face images, encode the spliced result to obtain an initial DMS signal feature matrix, splice the plurality of image features of the plurality of frames of face images, encode the spliced result to obtain an image feature matrix, and sequentially execute a plurality of attention fusion models containing a self-attention mechanism, a cross-attention mechanism and a feedforward neural network to obtain a target attention fusion feature, and finally determine the driving state of the driver based on the target attention fusion feature. As can be seen, the embodiments of the present application can model the association between the time-series DMS basic signals through the self-attention mechanism, and model the association between the time-series DMS basic signals and the high-dimensional image features extracted from the images through the cross-attention mechanism. The fusion of the two will make the detected driver state more rich and accurate.

[0057] In a third aspect, the embodiments of the present application provide a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any possible implementation manner of the first aspect.

[0058] In a fourth aspect, the embodiments of the present application provide an electronic device, which comprises:

[0059] one or more processors;

[0060] The processor is coupled with a storage device for storing one or more programs.

[0061] When the one or more programs are executed by one or more processors, the electronic device implements the method as described in any possible implementation manner of the first aspect.

[0062] In a fifth aspect, an embodiment of the present application provides a vehicle, which comprises the apparatus as described in any possible implementation manner of the second aspect, or comprises the electronic device as described in the fourth aspect.

[0063] In a sixth aspect, an embodiment of the present application provides a computer program product, which comprises instructions, when the instructions are executed on a computer or a processor, causing the computer or the processor to execute the method as described in any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application. Those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0065] Figure 1 A flowchart of a driver driving state detection method provided by an embodiment of the present application is shown in FIG. 1;

[0066] Figure 2 A flowchart of another driver driving state detection method provided by an embodiment of the present application is shown in FIG. 2;

[0067] Figure 3 A flowchart of a self-attention mechanism processing method provided by an embodiment of the present application is shown in FIG. 3;

[0068] Figure 4 A flowchart of a cross-attention mechanism processing method provided by an embodiment of the present application is shown in FIG. 4;

[0069] Figure 5 A block diagram of a driver driving state detection apparatus provided by an embodiment of the present application is shown in FIG. 5;

[0070] Figure 6 A structural diagram of an electronic device or a computer device provided by an embodiment of the present application is shown in FIG. 6. DETAILED DESCRIPTION

[0071] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0072] It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The terms "including" and "having" in the embodiments of this application and the accompanying drawings, as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0073] Figure 1 This is a flowchart of a method for detecting a driver's driving status. The method can be applied to electronic devices or computer devices, specifically to a terminal or server, where the terminal can be a vehicle. The method includes the following steps:

[0074] S110: Detecting at least one DMS signal feature of each face image frame and at least one image feature of each face image frame in the plurality of face image frames based on a DMS signal feature detection model.

[0075] Among them, the DMS signal features include at least one of the following signal features: head posture, sight vector, mouth opening distance, eyelid distance, expression coefficient, identity coefficient, etc. Figure 2 As shown, the image features include image features obtained when performing convolution calculations on each layer of the convolutional neural network in the DMS signal feature detection model. The image features in the embodiment of the present application include at least the image features of the last layer of the convolutional neural network, that is, the deep convolutional neural network, and may also include image features of other layers of the convolutional neural network, such as shallow and intermediate layers of the convolutional neural network. The identity coefficient and the expression coefficient are respectively a high-dimensional vector parsed by the DMS signal feature detection model. When the DMS signal feature detection model includes a three-dimensional face reconstruction model, the identity coefficient and the expression coefficient can be respectively a high-dimensional vector parsed by the three-dimensional face reconstruction model. Figure 2 The pooling in represents pooling.

[0076] The DMS signal feature detection model can be a three-dimensional detection model trained based on a convolutional neural network. It can directly detect head posture, gaze vector, three-dimensional facial key points, expression coefficient and identity coefficient, and obtain the mouth opening distance, eyelid distance, etc. based on the three-dimensional facial key points.

[0077] The DMS signal feature detection model can further include a CNN (Convolutional Neural Network) model and a three-dimensional face reconstruction model. As shown in Figure 2 the first DMS signal feature, and obtain at least one image feature after convolution calculation of the multi-frame face images by the CNN model, wherein the first DMS signal feature includes a head pose and / or a line-of-sight vector; and then based on the three-dimensional face reconstruction model and the two-dimensional face key points, obtain a second DMS signal feature, wherein the second DMS signal feature includes at least one of a face mouth opening distance, an eyelid distance, an expression coefficient, and an identity coefficient, the face mouth opening distance and the eyelid distance being calculated based on the three-dimensional face reconstruction model and the two-dimensional face key points, that is, after the two-dimensional face key points are input into the three-dimensional face reconstruction model, the three-dimensional face key points, the expression coefficient, and the identity coefficient are output first, and then the face mouth opening distance and the eyelid distance can be calculated based on the three-dimensional face key points. The first DMS signal feature and the second DMS signal feature are DMS signal features in the at least one DMS signal feature. The three-dimensional face reconstruction model includes a 3DMM (3D morph model). The face key points include eye, nose, mouth, ear, and the like.

[0078] The face image is an image including only a face detected from an original image including a face captured by a camera in the DMS by using a target detection technology such as Faster-RCNN (Faster Region based Convolutional Neural Network), YOLO (You Look once), SCRFD, and the like. When the face image extracted from the original image is tilted, a pre-trained face feature point positioning depth model is usually used to obtain two-dimensional face key points, and then the face is corrected and cropped based on the two-dimensional face key points to obtain a corrected face image.

[0079] S120: splice the multiple DMS signal features of the multi-frame face images, encode the spliced result to obtain an initial DMS signal feature matrix, and splice the multiple image features of the multi-frame face images, and encode the spliced result to obtain an image feature matrix.

[0080] The plurality of DMS signal features of the plurality of face images are spliced, including: splicing each DMS signal feature in each face image respectively to obtain a DMS signal spliced feature of each face image, and splicing the DMS signal spliced features of the plurality of face images in time sequence to obtain a spliced result of the plurality of DMS signal features of the plurality of face images.

[0081] The splicing sequence of each DMS signal feature is not limited when each DMS signal feature in each face image is spliced respectively, as long as the splicing sequence remains consistent.

[0082] The plurality of image features of the plurality of face images are spliced, including: splicing each image feature in each face image respectively to obtain an image spliced feature of each face image, and splicing the image spliced features of the plurality of face images in time sequence to obtain a spliced result of the plurality of image features of the plurality of face images.

[0083] The splicing sequence of each image feature is not limited when each image feature in each face image is spliced respectively, as long as the splicing sequence remains consistent.

[0084] In addition, the encoding method for encoding the spliced result includes but is not limited to MLP (Multi-Layer Perceptron) encoding, and in order to improve the quality of the model, position encoding can also be added on the basis of MLP encoding. The initial DMS signal feature matrix and the image feature matrix can both be T*D dimension matrices, where T represents the time length of the plurality of face images, and D represents the feature length of a single face image.

[0085] S130: sequentially execute a plurality of attention fusion models to obtain a target attention fusion feature.

[0086] The sequential execution of the plurality of attention fusion models means that the output result of the first attention fusion model is used as the input data of the second attention fusion model, and the output result of the second attention fusion model is used as the input data of the third attention fusion model. Figure 2 As shown in FIG. 8, each attention fusion model includes a self-attention mechanism, a cross-attention mechanism, and a feedforward neural network (FeedForward).

[0087] The execution process of each attention fusion model includes: analyzing the correlation between each DMS signal feature in the current DMS signal feature matrix based on a self-attention mechanism to obtain a current self-attention fusion feature matrix, wherein the current DMS signal feature matrix input when the first attention fusion model is executed is an initial DMS signal feature matrix; analyzing the correlation between the current self-attention signal feature matrix and the image feature matrix based on a cross-attention mechanism to obtain a current cross-attention fusion feature matrix; and performing convolution calculation on the current cross-attention fusion feature matrix based on a feedforward neural network, and taking the convolution calculation result as a new current DMS signal feature matrix.

[0088] The number of the plurality of attention fusion models is determined by the computing power and specific speed requirement that can be provided by the chip of the electronic device. The attention mechanism includes three elements of Query, Key and Value, and the basic principle is: given a Query, the correlation between the Query and the Key is calculated, and then the most suitable Value is found according to the correlation between the Query and the Key. The attention fusion model in the embodiments of the present application can be a Transformer model, and supports a multi-head attention mechanism.

[0089] In the embodiments of the present application, the Query, the Key and the Value in the self-attention mechanism are all derived from the DMS signal feature matrix; and / or, the Query in the cross-attention mechanism is derived from the self-attention fusion feature matrix, and the Key and the Value in the cross-attention mechanism are both derived from the image feature matrix. The sources of the Key and the Value are consistent, and the sources of the Key and the Value determine whether it is a self-attention mechanism or a cross-attention mechanism.

[0090] As Figure 3As shown, the specific process of obtaining the current self-attention fusion feature matrix based on the analysis of the correlation between the DMS signal features in the current DMS signal feature matrix includes: inputting the current DMS signal feature matrix as input information into the self-attention mechanism model, first performing calculation based on a 1*1 convolution layer and a BN (Batch Normalization) layer respectively to obtain Query, Key and Value; after transposing the Key, calculating the product of Query and Key by using a Matmul function to obtain a fusion feature; inputting the fusion feature into a Softmax function (a normalized exponential function) to obtain Scores; calculating the product of Scores and Value by using a Matmul function to obtain new Value; and performing fusion calculation on the new Value through a 1*1 convolution layer and a BN layer to obtain the final Value; and finally superimposing the final Value and the input information to obtain output information, i.e., the current self-attention fusion feature matrix.

[0091] As shown, Figure 4 As shown, the specific process of obtaining the current cross-attention fusion feature matrix based on the analysis of the correlation between the current self-attention signal feature matrix and the image feature matrix includes: inputting the current self-attention signal feature matrix as first input information, obtaining Query through calculation of a 1*1 convolution layer and a BN layer, inputting the image feature matrix as second input information, and performing calculation through different 1*1 convolution layers and BN layers respectively to obtain Key and Value; after transposing the Key, calculating the product of Query and Key by using a Matmul function to obtain a fusion feature; inputting the fusion feature into a Softmax function (a normalized exponential function) to obtain Scores; calculating the product of Scores and Value by using a Matmul function to obtain new Value; and performing fusion calculation on the new Value through a 1*1 convolution layer and a BN layer to obtain the final Value; and finally superimposing the final Value and the first input information to obtain output information, i.e., the current cross-attention fusion feature matrix.

[0092] The feedforward neural network is a neural network structure, which is composed of multiple interconnected hidden layers, and each layer is obtained by processing through linear transformation, activation function and dropout (random inactivation) operation, etc. The feedforward neural network can improve the expression ability of the whole attention fusion model.

[0093] S140: determining the driving state of the driver based on the target attention fusion feature.

[0094] After obtaining the target attention fusion feature, different kinds of driving states of the driver can be determined based on the target attention fusion feature according to different requirements. The driving states include whether in a fatigue state, specific expressions, heart rate, respiration rate and the like of the driver.

[0095] As shown in Figure 2 When it is required to detect whether the driver is in a fatigue state, a first pooling operation can be performed on the target attention fusion feature of T*D dimensions to obtain a target attention fusion feature of D dimensions, where T represents the time length of the plurality of facial images, and D represents the feature length of a single facial image; the target attention fusion feature of D dimensions is processed based on the MLP to determine whether the driver is in a fatigue state. When it is required to detect specific expressions of the driver, the target attention fusion feature of D dimensions is processed based on the MLP to determine the expressions of the driver, including any one of expressions 1-expression n, such as joy, anger, sadness, happiness and the like. The specific expression classification can be divided according to actual requirements.

[0096] When it is required to detect whether the driver is in a fatigue state, the DMS signal features detected from each facial image at least include a mouth opening distance and an eyelid distance, and other DMS signal features can be used as optional additional features to further improve the accuracy of fatigue state detection; when it is required to detect specific expressions of the driver, the DMS signal features detected from each facial image at least include expression coefficients, and other DMS signal features can be used as optional additional features to further improve the accuracy of expression detection.

[0097] As can be known from the above scheme, the embodiments of the present application can convert the target attention fusion feature of T*D dimensions into a target attention fusion feature of D dimensions through a first pooling operation, so that the target attention fusion feature is more focused on the feature itself, and then whether the driver is in a fatigue state and / or the expression of the driver can be determined.

[0098] As shown in Figure 2 When it is required to detect the heart rate and / or respiration rate of the driver, a second pooling operation can be performed on the target attention fusion feature of T*D dimensions to obtain a target attention fusion feature of T dimensions, where T represents the time length of the plurality of facial images, and D represents the feature length of a single facial image; the target attention fusion feature of T dimensions is processed based on the MLP to obtain a BVP (blood volume pulse wave) waveform, and the heart rate and / or respiration rate of the driver is determined by analyzing the BVP waveform.

[0099] When it is required to detect the heart rate and / or respiration rate of the driver, the image features detected from each facial image are essential, and the DMS signal features can be used as optional additional features to further improve the accuracy of the heart rate and / or respiration rate.

[0100] Through the above scheme, it can be known that the target attention fusion feature of T*D dimensions can be converted into a target attention fusion feature of T dimensions through the second pooling operation, so that the target attention fusion feature is more focused on the time dimension, and then the change of the driver in the time dimension can be determined, that is, the heart rate and / or the respiration rate of the driver can be determined.

[0101] The driver driving state detection method provided by the embodiment of the application can first detect at least one DMS signal feature of each frame of face image in the plurality of frames of face images and at least one image feature of each frame of face image based on a DMS signal feature detection model, then splice a plurality of DMS signal features of the plurality of frames of face images, encode the spliced result to obtain an initial DMS signal feature matrix, splice a plurality of image features of the plurality of frames of face images, encode the spliced result to obtain an image feature matrix, sequentially execute a plurality of attention fusion models containing self-attention mechanism, cross-attention mechanism and feedforward neural network to obtain a target attention fusion feature, and finally determine the driving state of the driver based on the target attention fusion feature. It can be known that the embodiment of the application can model the association between the time sequence DMS basic signals through the self-attention mechanism, model the association between the time sequence DMS basic signals and the high-dimensional image features extracted from the images through the cross-attention mechanism, and the fusion of the two will make the detected driver state more rich and accurate.

[0102] Based on the above method embodiment, as shown in Figure 5 Another embodiment of the application provides a driver driving state detection device, which comprises:

[0103] The detection unit 210 is configured to detect at least one DMS signal feature of each frame of face image in the plurality of frames of face images and at least one image feature of each frame of face image based on a DMS signal feature detection model;

[0104] The encoding unit 220 is configured to splice a plurality of DMS signal features of the plurality of frames of face images, encode the spliced result to obtain an initial DMS signal feature matrix, splice a plurality of image features of the plurality of frames of face images, and encode the spliced result to obtain an image feature matrix;

[0105] The execution unit 230 is configured to sequentially execute a plurality of attention fusion models to obtain a target attention fusion feature;

[0106] The determination unit 240 is configured to determine the driving state of the driver based on the target attention fusion feature;

[0107] The execution unit 230 comprises:

[0108] The self-attention analysis module 231 is configured to analyze the correlation between the DMS signal features in the current DMS signal feature matrix based on a self-attention mechanism, and obtain a current self-attention fusion feature matrix.

[0109] The cross-attention analysis module 232 is configured to analyze the correlation between the current self-attention signal feature matrix and the image feature matrix based on a cross-attention mechanism, and obtain a current cross-attention fusion feature matrix.

[0110] The convolution calculation module 233 is configured to perform convolution calculation on the current cross-attention fusion feature matrix based on a feedforward neural network, and take the convolution calculation result as a new current DMS signal feature matrix.

[0111] In a possible implementation, the detection unit 210 includes:

[0112] The detection module is configured to, when the DMS signal feature detection model includes a convolutional neural network model and a three-dimensional face reconstruction model, detect two-dimensional face key points and a first DMS signal feature of each of the plurality of face images based on the convolutional neural network model, and obtain at least one image feature after the convolutional neural network model performs convolution calculation on the plurality of face images, wherein the first DMS signal feature includes head posture and / or gaze vector.

[0113] The first determination module is configured to obtain a second DMS signal feature based on the three-dimensional face reconstruction model and the two-dimensional face key points, wherein the second DMS signal feature includes at least one of face mouth opening distance, eyelid distance, expression coefficient, and identity coefficient, the face mouth opening distance and the eyelid distance are calculated based on the three-dimensional face key points determined based on the three-dimensional face reconstruction model and the two-dimensional face key points, and the first DMS signal feature and the second DMS signal feature are DMS signal features in the at least one DMS signal feature.

[0114] In a possible implementation, the encoding unit 220 includes a first splicing module and / or a second splicing module.

[0115] The first splicing module is configured to splice the DMS signal features of the plurality of frames of the face images, including: splicing each DMS signal feature in each frame of the face images respectively to obtain a DMS signal splicing feature of each frame of the face images, and splicing the DMS signal splicing features of the plurality of frames of the face images in time sequence to obtain a splicing result of the plurality of DMS signal features of the plurality of frames of the face images.

[0116] The second splicing module is configured to splice the image features of the plurality of frames of the face images, including: splicing each image feature in each frame of the face images respectively to obtain an image splicing feature of each frame of the face images, and splicing the image splicing features of the plurality of frames of the face images in time sequence to obtain a splicing result of the plurality of image features of the plurality of frames of the face images.

[0117] In a possible implementation, the Query, the Key and the Value in the self-attention mechanism are all derived from the DMS signal feature matrix.

[0118] And / or, the Query in the cross-attention mechanism is derived from the self-attention fusion feature matrix, and the Key and the Value in the cross-attention mechanism are both derived from the image feature matrix.

[0119] In a possible implementation, the determination unit 240 includes:

[0120] A first pooling module is configured to perform a first pooling operation on the target attention fusion feature of the T*D dimension to obtain a target attention fusion feature of D dimension, where T represents the time length of the plurality of frames of the face images, and D represents the feature length of a single frame of the face images.

[0121] A second determination module is configured to determine whether the driver is in a fatigue state based on processing the target attention fusion feature of the D dimension by a multi-layer perception (MLP), and / or determine the expression of the driver based on processing the target attention fusion feature of the D dimension by the MLP.

[0122] In a possible implementation, the determination unit 240 includes:

[0123] A second pooling module is configured to perform a second pooling operation on the target attention fusion feature of the T*D dimension to obtain a target attention fusion feature of T dimension, where T represents the time length of the plurality of frames of the face images, and D represents the feature length of a single frame of the face images.

[0124] The processing module is configured to process the target attention fusion feature of the T-dimensional based on the MLP, and obtain a blood volume pulse (BVP) waveform;

[0125] The third determining module is configured to determine the heart rate and / or the respiration rate of the driver by analyzing the BVP waveform.

[0126] The driver driving state detection device provided by the embodiments of the present application can detect at least one DMS signal feature of each frame of face image in the plurality of frames of face images and at least one image feature of each frame of face image based on the DMS signal feature detection model first, splice the plurality of DMS signal features of the plurality of frames of face images, encode the spliced result to obtain an initial DMS signal feature matrix, splice the plurality of image features of the plurality of frames of face images, encode the spliced result to obtain an image feature matrix, sequentially execute a plurality of attention fusion models containing self-attention mechanism, cross-attention mechanism and feedforward neural network to obtain a target attention fusion feature, and finally determine the driving state of the driver based on the target attention fusion feature. Therefore, the embodiments of the present application can model the correlation between the time-series DMS basic signals through the self-attention mechanism, model the correlation between the time-series DMS basic signals and the high-dimensional image features extracted from the images through the cross-attention mechanism, and the fusion of the two will make the detected driver state more rich and accurate.

[0127] Based on the above method embodiments, another embodiment of the present application provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method according to any one of the above embodiments.

[0128] Based on the above method embodiments, another embodiment of the present application provides an electronic device or a computer device, as shown in the accompanying drawings, comprising: Figure 6

[0129] one or more processors 310;

[0130] The processor 310 is coupled with a storage device 320, and the storage device 320 is configured to store one or more programs;

[0131] When the one or more programs are executed by the one or more processors 310, the electronic device or the computer device implements the method according to any one of the above embodiments.

[0132] Based on the above method embodiments, another embodiment of the present application provides a vehicle, which comprises the device according to any one of the above embodiments, or comprises the electronic device as described above.

[0133] ​Based on the above embodiments, another embodiment of the present application provides a computer program product containing instructions, which, when executed on a computer or processor, cause the computer or processor to perform the method according to any one of the above embodiments.

[0134] The device embodiments correspond to the method embodiments and have the same technical effects. For details, refer to the method embodiments. The device embodiments are based on the method embodiments. For details, refer to the method embodiments, which will not be described here again. Those skilled in the art can understand that the drawings are only schematic diagrams of an embodiment, and the modules or flows in the drawings are not necessarily required for implementing the present application.

[0135] Those skilled in the art can understand that the modules in the device in the embodiments can be distributed in the device in the embodiments as described, or can be correspondingly changed and located in one or more devices different from the embodiments. The modules in the above embodiments can be combined into one module, or can be further split into multiple sub-modules.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features. Such modifications or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A driver driving state detection method characterized by comprising: The method comprises: detecting at least one DMS signal feature of each of the plurality of frames of face images and at least one image feature of each of the plurality of frames of face images based on a driver monitoring system (DMS) signal feature detection model; splicing a plurality of the DMS signal features of the plurality of frames of face images and encoding the spliced result to obtain an initial DMS signal feature matrix, and splicing a plurality of the image features of the plurality of frames of face images and encoding the spliced result to obtain an image feature matrix; sequentially executing a plurality of attention fusion models to obtain target attention fusion features; determining a driving state of a driver based on the target attention fusion features; wherein the execution process of the attention fusion model comprises: analyzing the correlation between each DMS signal feature in the current DMS signal feature matrix based on a self-attention mechanism to obtain a current self-attention fusion feature matrix, wherein the current DMS signal feature matrix input when the first attention fusion model is executed is the initial DMS signal feature matrix; analyzing the correlation between the current self-attention signal feature matrix and the image feature matrix based on a cross-attention mechanism to obtain a current cross-attention fusion feature matrix; performing convolution calculation on the current cross-attention fusion feature matrix based on a feedforward neural network, and taking the convolution calculation result as a new current DMS signal feature matrix.

2. The method of claim 1, wherein, When the DMS signal feature detection model comprises a convolutional neural network model and a three-dimensional face reconstruction model, detecting at least one DMS signal feature of each of the plurality of frames of face images and at least one image feature of each of the plurality of frames of face images based on a driver monitoring system (DMS) signal feature detection model comprises: detecting two-dimensional face key points and a first DMS signal feature of each of the plurality of frames of face images based on the convolutional neural network model, and obtaining at least one image feature after convolution calculation of the plurality of frames of face images by the convolutional neural network model, wherein the first DMS signal feature comprises head posture and / or line-of-sight vector; obtaining a second DMS signal feature based on the three-dimensional face reconstruction model and the two-dimensional face key points, wherein the second DMS signal feature comprises at least one of face mouth opening distance, eyelid distance, expression coefficient and identity coefficient, the face mouth opening distance and the eyelid distance are calculated based on the three-dimensional face key points determined by the three-dimensional face reconstruction model and the two-dimensional face key points, and the first DMS signal feature and the second DMS signal feature are both DMS signal features in the at least one DMS signal feature.

3. The method of claim 1, wherein, The splicing of the multiple DMS signal features of the multiple frames of the face images comprises: splicing each of the DMS signal features in each of the frames of the face images respectively to obtain DMS signal splicing features of each of the frames of the face images, and splicing the DMS signal splicing features of the multiple frames of the face images in time sequence to obtain a splicing result of the multiple DMS signal features of the multiple frames of the face images. The splicing of the multiple DMS signal features of the multiple frames of the face images comprises: splicing each of the DMS signal features in each of the frames of the face images respectively to obtain DMS signal splicing features of each of the frames of the face images, and splicing the DMS signal splicing features of the multiple frames of the face images in time sequence to obtain a splicing result of the multiple DMS signal features of the multiple frames of the face images.

4. The method of claim 1, wherein, The Query, the Key and the Value in the self-attention mechanism are all derived from the DMS signal feature matrix. The Query in the cross-attention mechanism is derived from the self-attention fusion feature matrix, and the Key and the Value in the cross-attention mechanism are both derived from the image feature matrix.

5. The method according to any one of claims 1-4, characterized in that, The determination of the driving state of the driver based on the target attention fusion feature comprises: performing a first pooling operation on the target attention fusion feature of T*D dimensions to obtain a target attention fusion feature of D dimensions, wherein T represents the time length of the multiple frames of the face images, and D represents the feature length of a single frame of the face images; processing the target attention fusion feature of D dimensions based on a multi-layer perception (MLP) to determine whether the driver is in a fatigue state, and / or processing the target attention fusion feature of D dimensions based on the MLP to determine the expression of the driver.

6. The method according to any one of claims 1-4, characterized in that, The determination of the driving state of the driver based on the target attention fusion feature comprises: performing a second pooling operation on the target attention fusion feature of T*D dimensions to obtain a target attention fusion feature of T dimensions, wherein T represents the time length of the multiple frames of the face images, and D represents the feature length of a single frame of the face images; processing the target attention fusion feature of T dimensions based on an MLP to obtain a blood volume pulse (BVP) waveform, and determining the heart rate and / or the respiration rate of the driver by analyzing the BVP waveform.

7. A driver driving state detection device characterized by comprising: The device comprises: a detection unit configured to detect at least one DMS signal feature of each of the multiple frames of the face images and at least one image feature of each of the frames of the face images based on a driver monitoring system (DMS) signal feature detection model; an encoding unit configured to splice multiple DMS signal features of the multiple frames of the face images, encode the splicing result to obtain an initial DMS signal feature matrix, and splice multiple image features of the multiple frames of the face images, encode the splicing result to obtain an image feature matrix; an execution unit configured to sequentially execute multiple attention fusion models to obtain a target attention fusion feature. The determining unit is configured to determine a driving state of the driver based on the target attention fusion feature; The execution unit comprises: The self-attention analysis module is configured to analyze the correlation between the DMS signal features in the current DMS signal feature matrix based on a self-attention mechanism, and obtain a current self-attention fusion feature matrix, wherein the current DMS signal feature matrix input when the first attention fusion model is executed is the initial DMS signal feature matrix. The cross-attention analysis module is configured to analyze the correlation between the current self-attention signal feature matrix and the image feature matrix based on a cross-attention mechanism, and obtain a current cross-attention fusion feature matrix. The convolution calculation module is configured to perform convolution calculation on the current cross-attention fusion feature matrix based on a feedforward neural network, and take the convolution calculation result as a new current DMS signal feature matrix.

8. The apparatus of claim 7, wherein, The detection unit comprises: The detection module is configured to, when the DMS signal feature detection model comprises a convolutional neural network model and a three-dimensional face reconstruction model, detect two-dimensional face key points and a first DMS signal feature of each of the plurality of frames of face images based on the convolutional neural network model, and obtain at least one image feature after the convolutional neural network model performs convolution calculation on the plurality of frames of face images, wherein the first DMS signal feature comprises a head posture and / or a line-of-sight vector. The first determination module is configured to obtain a second DMS signal feature based on the three-dimensional face reconstruction model and the two-dimensional face key points, wherein the second DMS signal feature comprises at least one of a face mouth opening distance, an eyelid distance, an expression coefficient, and an identity coefficient, the face mouth opening distance and the eyelid distance are calculated based on the three-dimensional face key points determined by the three-dimensional face reconstruction model and the two-dimensional face key points, and the first DMS signal feature and the second DMS signal feature are DMS signal features in the at least one DMS signal feature.

9. The apparatus of claim 7, wherein, The encoding unit comprises a first splicing module and / or a second splicing module. The first splicing module is configured to splice the plurality of DMS signal features of the plurality of frames of face images, including: splicing each DMS signal feature in each frame of face image respectively to obtain a DMS signal splicing feature of each frame of face image, and splicing the DMS signal splicing features of the plurality of frames of face images in time sequence to obtain a splicing result of the plurality of DMS signal features of the plurality of frames of face images. The second splicing module is configured to splice the plurality of image features of the plurality of frames of face images, including: splicing each image feature in each frame of face image respectively to obtain an image splicing feature of each frame of face image, and splicing the image splicing features of the plurality of frames of face images in time sequence to obtain a splicing result of the plurality of image features of the plurality of frames of face images.

10. The apparatus of claim 7, wherein, The Query, the Key and the Value in the self-attention mechanism are all derived from the DMS signal feature matrix; And / or, the Query in the cross-attention mechanism is derived from the self-attention fusion feature matrix, and the Key and the Value in the cross-attention mechanism are both derived from the image feature matrix.

11. The apparatus of any one of claims 7-10, wherein, The determining unit comprises: The first pooling module is configured to perform a first pooling operation on the target attention fusion feature of the T*D dimension to obtain a target attention fusion feature of a D dimension, wherein T represents the time length of the multiple frames of the face images, and D represents the feature length of a single frame of the face images. The second determining module is configured to determine whether the driver is in a fatigue state based on processing the target attention fusion feature of the D dimension by a multi-layer perception (MLP), and / or determine the expression of the driver based on processing the target attention fusion feature of the D dimension by the MLP.

12. The apparatus of any one of claims 7-10, wherein, The determining unit comprises: The second pooling module is configured to perform a second pooling operation on the target attention fusion feature of the T*D dimension to obtain a target attention fusion feature of a T dimension, wherein T represents the time length of the multiple frames of the face images, and D represents the feature length of a single frame of the face images. The processing module is configured to obtain a blood volume pulse (BVP) waveform by processing the target attention fusion feature of the T dimension based on an MLP. The third determining module is configured to determine the heart rate and / or the respiration rate of the driver by analyzing the BVP waveform.

13. A computer readable storage medium having stored thereon a computer program, characterized in that The program, when executed by a processor, implements the method of any one of claims 1-6.

14. An electronic device, comprising: The electronic device comprises: One or more processors; The processor is coupled with a storage device, and the storage device is configured to store one or more programs; When the one or more programs are executed by the one or more processors, the electronic device implements the method of any one of claims 1-6.

15. A vehicle characterized by comprising: The vehicle comprises the apparatus of any one of claims 7-12, or comprises the electronic device of claim 14.

Images