Driver state estimation method based on multi-task learning

Abstract

This invention discloses a driver state estimation method based on multi-task learning, comprising: using a shared pre-trained CNN model as the backbone network to extract features from an input target image to obtain shared features for the target tasks; wherein the target tasks include a keypoint prediction task, an occlusion probability estimation task, and a head pose estimation task; inputting the shared features into each task module to complete the corresponding task prediction, obtaining keypoint prediction results, occlusion probability estimation results, and head pose estimation results; and generating a driver state estimation result based on the keypoint prediction results, occlusion probability estimation results, and head pose estimation results. This invention efficiently solves the three problems of driver facial keypoint detection, occlusion probability estimation, and head pose estimation, improving the accuracy of the estimation results and can be widely applied in the field of deep learning technology.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a driver state estimation method based on multi-task learning. Background Technology

[0002] A driver's face contains a wealth of information, such as appearance, posture, and key points, which give rise to various driver state estimation tasks. Traditional methods treat these tasks as independent, neglecting the potential relationships between information from different tasks. In other words, within the same facial space, information from different modalities of different tasks may be closely interconnected due to the inherent rigid constraints of facial structure. To achieve artificial intelligence in complex open-world scenarios, a promising and necessary approach is to co-learn interrelated tasks to achieve task-specific knowledge transfer, thereby enabling mutual promotion and optimization of multiple tasks.

[0003] Based on the location of information interaction, multi-task learning can currently be divided into encoder-based and decoder-based methods. Encoder-based methods share an existing backbone network as the encoder, and then process the common features obtained by different task-specific modules. Decoder-based methods primarily involve information interaction within the decoder, simulating inter-task dependencies by interacting with features decoupled from specific task modules. While face-based multi-task learning has seen some development in recent years, they all share the characteristic of extracting common features from a shared underlying layer to simulate inter-task information interaction. In other words, they are all encoder-based methods and do not effectively explore the decoder stage, nor do they realistically embed the dependencies between these tasks into the network, thus failing to enable the model to jointly learn multiple tasks in a more human-like manner. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a joint and efficient driver state estimation method based on multi-task learning.

[0005] One aspect of this invention provides a driver state estimation method based on multi-task learning, comprising:

[0006] A shared pre-trained CNN model is used as the backbone network to extract features from the input target image to obtain shared features for the target task; wherein, the target task includes key point prediction task, occlusion probability estimation task and head pose estimation task.

[0007] The shared features are input into each task module to complete the corresponding task prediction, and the key point prediction results, occlusion probability estimation results and head pose estimation results are obtained.

[0008] Based on the key point prediction results, occlusion probability estimation results, and head pose estimation results, a driver state estimation result is generated.

[0009] Specifically, the step of inputting the shared features into each task module to complete the corresponding task prediction includes:

[0010] In the key point prediction module, the key point heatmap is predicted through the self-attention layer of the Transformer.

[0011] In the occlusion probability assessment module, the local feature block corresponding to each key point is extracted from the shared features using the key point location information. The local feature block and the current head pose information are input into the self-attention layer to predict the occlusion probability of each key point.

[0012] In the head pose estimation module, topological, local, and global appearance information are fused in a layer-by-layer fusion manner.

[0013] Optionally, the keypoint prediction module, which predicts the keypoint heatmap through the Transformer's self-attention layer, includes:

[0014] The shared features are input into the Transformer network for multiple iterative processing.

[0015] For any number of iterations, the following steps are executed:

[0016] The shared features are processed by a convolutional layer and then unfolded to obtain the first feature.

[0017] The head pose and occlusion probability transmitted at the current stage are transformed by two linear layers to obtain the second and third features.

[0018] The first feature, the second feature, and the third feature are concatenated to obtain the initial input features of the keypoint Transformer;

[0019] The initial input features are resized and transformed in dimension to obtain the key point heatmap of the corresponding pixels.

[0020] Optionally, in the occlusion probability evaluation module, the local feature block corresponding to each keypoint is extracted from the shared features using keypoint location information, and the local feature block and the current head pose information are input into the self-attention layer to predict the occlusion probability of each keypoint, including:

[0021] Based on the heat map of the key points, the location information of the key points is obtained;

[0022] Based on the key point location information, local feature blocks corresponding to each key point are extracted from the shared features;

[0023] The local feature blocks are subjected to a size transformation through a size convolution layer to obtain local appearance information;

[0024] The prior attitude information obtained in the previous stage is processed to obtain coarse attitude information;

[0025] The coarse pose information is combined with the local feature block to determine the dependency information between points under the influence of the current pose.

[0026] Based on the dependency information, calculate the occlusion probability corresponding to each key point.

[0027] Optionally, in the head pose estimation module, the topological, local, and global appearance information is fused in a layer-by-layer fusion manner, including:

[0028] Based on the key point heatmap, obtain the prior location information of the corresponding key points;

[0029] The prior location information is processed by a linear layer to obtain the initial input of the Transformer, and then the topological features corresponding to each key point are generated.

[0030] The extracted local features are fused with topological features based on the occlusion information of key points to obtain the local features corresponding to each point;

[0031] The local features are converted into one-dimensional global appearance features, and then the first dimension is expanded and copied N times to obtain the global appearance features.

[0032] After fusing the local features and the global appearance features, they are then processed interactively by a Transformer.

[0033] Another aspect of this invention provides a driver state estimation device based on multi-task learning, comprising:

[0034] The first module is used to extract features from the input target image using a shared pre-trained CNN model as the backbone network to obtain shared features for the target task; wherein, the target task includes a key point prediction task, an occlusion probability estimation task, and a head pose estimation task.

[0035] The second module is used to input the shared features into each task module to complete the corresponding task prediction, and obtain the key point prediction result, occlusion probability estimation result and head pose estimation result.

[0036] The third module is used to generate driver state estimation results based on the key point prediction results, occlusion probability estimation results, and head posture estimation results.

[0037] Specifically, the second module is used for:

[0038] In the key point prediction module, the key point heatmap is predicted through the self-attention layer of the Transformer.

[0039] In the occlusion probability assessment module, the local feature block corresponding to each key point is extracted from the shared features using the key point location information. The local feature block and the current head pose information are input into the self-attention layer to predict the occlusion probability of each key point.

[0040] In the head pose estimation module, topological, local, and global appearance information are fused in a layer-by-layer fusion manner.

[0041] Optionally, the key point prediction module is specifically used for:

[0042] The shared features are input into the Transformer network for multiple iterative processing.

[0043] For any number of iterations, the following steps are executed:

[0044] The shared features are processed by a convolutional layer and then unfolded to obtain the first feature.

[0045] The head pose and occlusion probability transmitted at the current stage are transformed by two linear layers to obtain the second and third features.

[0046] The first feature, the second feature, and the third feature are concatenated to obtain the initial input features of the keypoint Transformer;

[0047] The initial input features are resized and transformed in dimension to obtain the key point heatmap of the corresponding pixels.

[0048] Optionally, the occlusion probability assessment module is specifically used for:

[0049] Based on the heat map of the key points, the location information of the key points is obtained;

[0050] Based on the key point location information, local feature blocks corresponding to each key point are extracted from the shared features;

[0051] The local feature blocks are subjected to a size transformation through a size convolution layer to obtain local appearance information;

[0052] The prior attitude information obtained in the previous stage is processed to obtain coarse attitude information;

[0053] The coarse pose information is combined with the local feature block to determine the dependency information between points under the influence of the current pose.

[0054] Based on the dependency information, calculate the occlusion probability corresponding to each key point.

[0055] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0056] The memory is used to store programs;

[0057] The processor executes the program to implement the method described above.

[0058] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0059] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0060] In embodiments of the present invention, a shared pre-trained CNN model is used as the backbone network to extract features from the input target image, thereby obtaining shared features for the target task. The target task includes a keypoint prediction task, an occlusion probability estimation task, and a head pose estimation task. The shared features are input into each task module to complete the corresponding task prediction, resulting in keypoint prediction results, occlusion probability estimation results, and head pose estimation results. Based on the keypoint prediction results, occlusion probability estimation results, and head pose estimation results, a driver state estimation result is generated. Specifically, the step of inputting the shared features into each task module to complete the corresponding task prediction includes: in the keypoint prediction module, a keypoint heatmap is predicted using the self-attention layer of a Transformer; in the occlusion probability estimation module, local feature blocks corresponding to each keypoint are extracted from the shared features using keypoint location information, and the local feature blocks and current head pose information are input into the self-attention layer to predict the occlusion probability of each keypoint; in the head pose estimation module, topological, local, and global appearance information are fused layer by layer. This invention efficiently solves three problems: driver facial key point detection, occlusion probability estimation, and head pose estimation, thereby improving the accuracy of the estimation results. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a flowchart illustrating the overall steps of the present invention;

[0063] Figure 2 This is a schematic diagram of the overall architecture of the present invention;

[0064] Figure 3 This is a schematic diagram of the training sample extraction process in an embodiment of the present invention;

[0065] Figure 4 Examples of facial key points, occlusion probabilities, and head poses estimated for embodiments of the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0067] To address the problems existing in the prior art, one aspect of this invention provides a driver state estimation method based on multi-task learning, comprising:

[0068] A shared pre-trained CNN model is used as the backbone network to extract features from the input target image to obtain shared features for the target task; wherein, the target task includes key point prediction task, occlusion probability estimation task and head pose estimation task.

[0069] The shared features are input into each task module to complete the corresponding task prediction, and the key point prediction results, occlusion probability estimation results and head pose estimation results are obtained.

[0070] Based on the key point prediction results, occlusion probability estimation results, and head pose estimation results, a driver state estimation result is generated.

[0071] Specifically, the step of inputting the shared features into each task module to complete the corresponding task prediction includes:

[0072] In the key point prediction module, the key point heatmap is predicted through the self-attention layer of the Transformer.

[0073] In the occlusion probability assessment module, the local feature block corresponding to each key point is extracted from the shared features using the key point location information. The local feature block and the current head pose information are input into the self-attention layer to predict the occlusion probability of each key point.

[0074] In the head pose estimation module, topological, local, and global appearance information are fused in a layer-by-layer fusion manner.

[0075] Optionally, the keypoint prediction module, which predicts the keypoint heatmap through the Transformer's self-attention layer, includes:

[0076] The shared features are input into the Transformer network for multiple iterative processing.

[0077] For any number of iterations, the following steps are executed:

[0078] The shared features are processed by a convolutional layer and then unfolded to obtain the first feature.

[0079] The head pose and occlusion probability transmitted at the current stage are transformed by two linear layers to obtain the second and third features.

[0080] The first feature, the second feature, and the third feature are concatenated to obtain the initial input features of the keypoint Transformer;

[0081] The initial input features are resized and transformed in dimension to obtain the key point heatmap of the corresponding pixels.

[0082] Optionally, in the occlusion probability evaluation module, the local feature block corresponding to each keypoint is extracted from the shared features using keypoint location information, and the local feature block and the current head pose information are input into the self-attention layer to predict the occlusion probability of each keypoint, including:

[0083] Based on the heat map of the key points, the location information of the key points is obtained;

[0084] Based on the key point location information, local feature blocks corresponding to each key point are extracted from the shared features;

[0085] The local feature blocks are subjected to a size transformation through a size convolution layer to obtain local appearance information;

[0086] The prior attitude information obtained in the previous stage is processed to obtain coarse attitude information;

[0087] The coarse pose information is combined with the local feature block to determine the dependency information between points under the influence of the current pose.

[0088] Based on the dependency information, calculate the occlusion probability corresponding to each key point.

[0089] Optionally, in the head pose estimation module, the topological, local, and global appearance information is fused in a layer-by-layer fusion manner, including:

[0090] Based on the key point heatmap, obtain the prior location information of the corresponding key points;

[0091] The prior location information is processed by a linear layer to obtain the initial input of the Transformer, and then the topological features corresponding to each key point are generated.

[0092] The extracted local features are fused with topological features based on the occlusion information of key points to obtain the local features corresponding to each point;

[0093] The local features are converted into one-dimensional global appearance features, and then the first dimension is expanded and copied N times to obtain the global appearance features.

[0094] After fusing the local features and the global appearance features, they are then processed interactively by a Transformer.

[0095] Another aspect of this invention provides a driver state estimation device based on multi-task learning, comprising:

[0096] The first module is used to extract features from the input target image using a shared pre-trained CNN model as the backbone network to obtain shared features for the target task; wherein, the target task includes a key point prediction task, an occlusion probability estimation task, and a head pose estimation task.

[0097] The second module is used to input the shared features into each task module to complete the corresponding task prediction, and obtain the key point prediction result, occlusion probability estimation result and head pose estimation result.

[0098] The third module is used to generate driver state estimation results based on the key point prediction results, occlusion probability estimation results, and head posture estimation results.

[0099] Specifically, the second module is used for:

[0100] In the key point prediction module, the key point heatmap is predicted through the self-attention layer of the Transformer.

[0101] In the occlusion probability assessment module, the local feature block corresponding to each key point is extracted from the shared features using the key point location information. The local feature block and the current head pose information are input into the self-attention layer to predict the occlusion probability of each key point.

[0102] In the head pose estimation module, topological, local, and global appearance information are fused in a layer-by-layer fusion manner.

[0103] Optionally, the key point prediction module is specifically used for:

[0104] The shared features are input into the Transformer network for multiple iterative processing.

[0105] For any number of iterations, the following steps are executed:

[0106] The shared features are processed by a convolutional layer and then unfolded to obtain the first feature.

[0107] The head pose and occlusion probability transmitted at the current stage are transformed by two linear layers to obtain the second and third features.

[0108] The first feature, the second feature, and the third feature are concatenated to obtain the initial input features of the keypoint Transformer;

[0109] The initial input features are resized and transformed in dimension to obtain the key point heatmap of the corresponding pixels.

[0110] Optionally, the occlusion probability assessment module is specifically used for:

[0111] Based on the heat map of the key points, the location information of the key points is obtained;

[0112] Based on the key point location information, local feature blocks corresponding to each key point are extracted from the shared features;

[0113] The local feature blocks are subjected to a size transformation through a size convolution layer to obtain local appearance information;

[0114] The prior attitude information obtained in the previous stage is processed to obtain coarse attitude information;

[0115] The coarse pose information is combined with the local feature block to determine the dependency information between points under the influence of the current pose.

[0116] Based on the dependency information, calculate the occlusion probability corresponding to each key point.

[0117] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0118] The memory is used to store programs;

[0119] The processor executes the program to implement the method described above.

[0120] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0121] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0122] The specific implementation process of the present invention will now be described in detail with reference to the accompanying drawings:

[0123] like Figure 1 and Figure 2 As shown, a driver state estimation method based on multi-task learning includes the following steps:

[0124] In step S1, the common features of the three tasks need to be extracted from the image of the input model.

[0125] Specifically, for an input image, the image is first normalized and then input into the shared underlying network structure for the three tasks, namely the high-resolution network (HR-Net).

[0126] The high-resolution network consists of four sub-networks, each of which adds a branch compared to the previous one. The resolution of the new branch is halved compared to the branch in the previous layer, but the number of channels is doubled. The final shared feature F has strong semantic information while retaining accurate spatial information useful for key point prediction.

[0127] The sample extraction process of this invention is as follows: Figure 3 As shown.

[0128] In step S2, the shared features F extracted in step S1 are input into the separate modules for the three tasks: keypoint prediction, occlusion probability estimation, and head pose estimation.

[0129] In step S3, a keypoint heatmap needs to be predicted based on shared features and knowledge from other tasks:

[0130] The shared features of the keypoint prediction module LM_i at each stage are all the same, F∈R. C×H×W In the nth iteration, the shared feature F is expanded to obtain f after passing through a 1×1 convolution layer. shared ∈R (H·W)×D The head pose conveyed in the previous stage. i-1 (yaw n-1 pitch n-1 , roll n-1 |i∈[1,…,I])∈R 1×3 and occlusion probability The dimensions are obtained by performing dimensionality transformations through two linear layers respectively. and Then, these are concatenated together to obtain the initial input features of the keypoint Transformer. This allows the current head pose and keypoint occlusion information to be taken into account during the keypoint prediction process, which is the first explicit embedding of the task relationship.

[0131]

[0132] The Transformers in all three task modules only use the encoder structure. For the input x, the activation value of each pixel is calculated using multi-head attention (MSA), as shown in the following formula:

[0133]

[0134] MSA(x)=[SA1(x);SA2(x);…,SA m (x)]W M

[0135] MLP(x) = Relu(x W) L1 +b1)W L2 +b2

[0136] Therefore, for the key point Transformer module in the i-th stage, the formalization is as follows:

[0137]

[0138]

[0139]

[0140] After L lm The output after the layer is The first H×W tokens are truncated and reshaped to obtain an embedding of size D×H×W. A 1×1 (linear) convolution is then used for dimensionality transformation to obtain the heatmap H of the N keypoints predicted in the i-th stage. i ∈R N×H×W .

[0141] During the first iteration, pose 0 and op 0 Unknown, therefore When the number of iterations is greater than 1, the model needs to know the pose and occlusion information in the current head space to consider pose and occlusion information during keypoint prediction. Furthermore, when the number of iterations is greater than 1, the coarse location information of the keypoints (Prior) is also considered. i-1 (landmark|Image) is also known, as reflected in H. i-1 Therefore, the heatmap predicted in the previous stage is multiplied by the final result to refine the image; that is, the heatmap predicted by the key point module is:

[0142]

[0143] In step S4, the occlusion probability of key points needs to be predicted based on shared features and knowledge from other tasks:

[0144] Using the H output in step S3 i Obtaining key point location information in shared features F∈R C×H×W Extract the local feature block patch corresponding to each key point i ∈R N×C×h×w This allows the model to predict whether keypoints are occluded based on local appearance information, much like humans do. This is the second explicit embedding of the task relation. The extracted local feature blocks are then subjected to a size transformation through a convolution of size h×w (h=w).

[0145]

[0146] After obtaining local appearance information, if we want to further consider whether each point is occluded, there are interrelationships between them. For example, the occlusion probabilities of two adjacent points are interdependent; points on the back in a large pose and several adjacent points occluded by a certain object. Judging whether something is occluded based solely on local appearance information does not utilize this dependency. The pose prior from the previous stage... i-1 The same process as step S3 is obtained. and Patch i Unite to obtain Discover the dependencies between points under the influence of the current pose:

[0147]

[0148] Therefore, the occlusion probability Transformer module for the i-th stage is formalized as follows:

[0149]

[0150]

[0151]

[0152] After passing L OP Extract the first N tokens after layer to get the output. The occlusion probability for each keypoint is obtained using the detection head. Similar to step S4, if the number of iterations is greater than 1, then op... i-1 It exists, and is used to activate the current prediction result, formally as follows:

[0153]

[0154] In step S5, following step S4, it is necessary to predict the driver's head pose based on shared features and knowledge from other tasks.

[0155] In step S5, topological, local, and global appearance information are fused in a layer-by-layer fusion manner.

[0156] First, H generated in stage S3 i Extract the location priors of N key points After passing through a linear layer, the initial input to the pose module Transformer is obtained. After L HP The Transformer layer obtains the topological features corresponding to each keypoint. Then, based on the occlusion information of the key points, the extracted local features are fused with the topological features (after concatenation, a 1×1 convolution is applied for dimensionality reduction), and then similarly processed by L... HP After the Transformer layer, the local features (including topology and appearance) corresponding to each point are obtained. Using a three-layer CNN, a pooling layer, and a linear layer, F∈R C×H×W Convert it into a one-dimensional global appearance feature with dimension D, then expand the first dimension and copy it N times to obtain the global appearance feature. This assigns the same global appearance information to each point. and After fusion and then through L HP The process of interacting with the Transformer layer can be formalized as follows:

[0157]

[0158]

[0159] in, These are the three Transformer modules in the pose module, and the process is the same as in steps S3 and S4. If the number of iterations is greater than 1, then the pose... i-1 If it exists, process it in the same way as in step S3. And its use in guiding the current prediction results can be formalized as follows:

[0160]

[0161] Finally, in step S6, steps S3, S4, and S5 are used as one iteration, and I iterations are performed to achieve the coarse-to-fine process. The final output is the accurate prediction result generated by the last iteration of the three tasks. A schematic diagram of the estimated facial key points, occlusion probability, and head pose in this embodiment of the invention is shown below. Figure 4 As shown.

[0162] In summary, this invention efficiently solves three problems: driver facial landmark detection, occlusion probability estimation, and head pose estimation. In terms of model size, compared to performing each of these tasks individually, this invention shares the underlying network for multiple tasks, resulting in a smaller model size that is more suitable for deployment on in-vehicle edge devices. Regarding model speed, this invention can simultaneously perform the three driver state estimation tasks, which is faster than performing them individually. In terms of performance, while achieving the three tasks, this invention not only implicitly mines the inter-feature dependencies of the three tasks at the underlying level but also explicitly embeds these dependencies into the network structure at the decoder stage. By using knowledge from other tasks as prior knowledge for the current task, the model can predict the current task more human-like, achieving higher accuracy, given prior information about other driver modalities.

[0163] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0164] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0165] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0166] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0167] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0168] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0169] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0170] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0171] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A driver state estimation method based on multi-task learning, characterized in that, include: A shared pre-trained CNN model is used as the backbone network to extract features from the input target image to obtain shared features for the target task; wherein, the target task includes key point prediction task, occlusion probability estimation task and head pose estimation task. The shared features are input into each task module to complete the corresponding task prediction, and the key point prediction results, occlusion probability estimation results and head pose estimation results are obtained. Based on the key point prediction results, occlusion probability estimation results, and head pose estimation results, a driver state estimation result is generated. Specifically, the step of inputting the shared features into each task module to complete the corresponding task prediction includes: In the key point prediction module, the key point heatmap is predicted through the self-attention layer of the Transformer. In the occlusion probability assessment module, the local feature block corresponding to each key point is extracted from the shared features using the key point location information. The local feature block and the current head pose information are input into the self-attention layer to predict the occlusion probability of each key point. In the head pose estimation module, topological, local, and global appearance information are fused in a layer-by-layer fusion manner.

2. The driver state estimation method based on multi-task learning according to claim 1, characterized in that, In the keypoint prediction module, the keypoint heatmap is predicted through the self-attention layer of the Transformer, including: The shared features are input into the Transformer network for multiple iterative processing. For any number of iterations, the following steps are performed: The shared features are processed by a convolutional layer and then unfolded to obtain the first feature. The head pose and occlusion probability transmitted at the current stage are transformed by two linear layers to obtain the second and third features. The first feature, the second feature, and the third feature are concatenated to obtain the initial input features of the keypoint Transformer; The initial input features are resized and transformed in dimension to obtain the key point heatmap of the corresponding pixels.

3. The driver state estimation method based on multi-task learning according to claim 1, characterized in that, In the occlusion probability assessment module, local feature blocks corresponding to each keypoint are extracted from the shared features using keypoint location information. These local feature blocks, along with the current head pose information, are input into the self-attention layer to predict the occlusion probability of each keypoint, including: Based on the heat map of key points, the location information of key points is obtained; Based on the key point location information, local feature blocks corresponding to each key point are extracted from the shared features; The local feature blocks are subjected to a size transformation through a size convolution layer to obtain local appearance information; The prior attitude information obtained in the previous stage is processed to obtain coarse attitude information; The coarse pose information is combined with the local feature block to determine the dependency information between points under the influence of the current pose. Based on the dependency information, calculate the occlusion probability corresponding to each key point.

4. The driver state estimation method based on multi-task learning according to claim 1, characterized in that, In the head pose estimation module, topological, local, and global appearance information are fused in a layer-by-layer fusion manner, including: Based on the key point heatmap, obtain the prior location information of the corresponding key points; The prior location information is processed by a linear layer to obtain the initial input of the Transformer, and then the topological features corresponding to each key point are generated. The extracted local features are fused with topological features based on the occlusion information of key points to obtain the local features corresponding to each point; The local features are converted into one-dimensional global appearance features, and then the first dimension is expanded and copied N times to obtain the global appearance features. After fusing the local features and the global appearance features, they are then processed interactively by a Transformer.

5. A driver state estimation device based on multi-task learning, characterized in that, include: The first module is used to extract features from the input target image using a shared pre-trained CNN model as the backbone network to obtain shared features of the target task; wherein, the target task includes key point prediction task, occlusion probability estimation task and head pose estimation task. The second module is used to input the shared features into each task module to complete the corresponding task prediction, and obtain the key point prediction result, occlusion probability estimation result and head pose estimation result. The third module is used to generate driver state estimation results based on the key point prediction results, occlusion probability estimation results, and head posture estimation results. Specifically, the second module is used for: In the key point prediction module, the key point heatmap is predicted through the self-attention layer of the Transformer. In the occlusion probability assessment module, the local feature block corresponding to each key point is extracted from the shared features using the key point location information. The local feature block and the current head pose information are input into the self-attention layer to predict the occlusion probability of each key point. In the head pose estimation module, topological, local, and global appearance information are fused in a layer-by-layer fusion manner.

6. The driver state estimation device based on multi-task learning according to claim 5, characterized in that, The key point prediction module is specifically used for: The shared features are input into the Transformer network for multiple iterative processing. For any number of iterations, the following steps are performed: The shared features are processed by a convolutional layer and then unfolded to obtain the first feature. The head pose and occlusion probability transmitted at the current stage are transformed by two linear layers to obtain the second and third features. The first feature, the second feature, and the third feature are concatenated to obtain the initial input features of the keypoint Transformer; The initial input features are resized and transformed in dimension to obtain the key point heatmap of the corresponding pixels.

7. A driver state estimation device based on multi-task learning according to claim 5, characterized in that, The occlusion probability assessment module is specifically used for: Based on the heat map of key points, the location information of key points is obtained; Based on the key point location information, local feature blocks corresponding to each key point are extracted from the shared features; The local feature blocks are subjected to a size transformation through a size convolution layer to obtain local appearance information; The prior attitude information obtained in the previous stage is processed to obtain coarse attitude information; The coarse pose information is combined with the local feature block to determine the dependency information between points under the influence of the current pose. Based on the dependency information, calculate the occlusion probability corresponding to each key point.

8. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.

Images