A multi-task attribute recognition method, device, medium and equipment for driving scenes

By using a multi-task network model to identify various characteristics of the driver, the problem of insufficient real-time performance and accuracy of driver attribute identification in existing technologies is solved, and efficient driver state identification is achieved in complex environments.

CN111860253BActive Publication Date: 2024-11-08DONG GUAN ZHENGYANG ELECTRONIC MECHANICAL LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010662803.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-07-10
Publication Date
2024-11-08
Estimated Expiration
2040-07-10

AI Technical Summary

Technical Problem

Existing technologies for driver attribute recognition suffer from limitations such as single-scenario limitations, insufficient real-time performance and accuracy. Multi-model recognition leads to slow response speed, high hardware requirements, and difficulty in handling the accuracy of attribute features in situations with occlusion, poor lighting, and multiple people.

Method used

A multi-task network model is used to identify various characteristics of drivers, including mask recognition, glasses recognition, smoking recognition, and phone call recognition. By preprocessing and training the multi-task network model, the network parameters are optimized by combining the OSEM module and attention mechanism to improve recognition accuracy.

Benefits of technology

It enables efficient and accurate identification of multiple driver characteristics in complex environments, improves the real-time performance and accuracy of driver status recognition, and reduces the difficulty of annotation and computational burden.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111860253B_ABST
    Figure CN111860253B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses a multi-task attribute recognition method, device, medium and equipment for driving scenes. The method includes: obtaining image data of the driver in the driving scene; inputting the image data into a pre-trained multi-task network model for attribute recognition; the attribute recognition tasks of the multi-task network model include: mask recognition, glasses recognition, smoking recognition and phone call recognition; according to the attribute recognition output result of the multi-task network model, the multi-task attribute recognition result of the driver is determined. By implementing this technical solution, a multi-task network model can be used to simultaneously recognize multiple features of the driver, and output an output result that can overall reflect the driver's driving state, thereby improving the accuracy of driver state recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, specifically to image recognition technology, and more particularly to a method, device, medium and equipment for multi-task attribute recognition of driving scenes. Background Art

[0002] In ADAS systems, providing driver characteristics in a timely and accurate manner is a key factor in realizing the intelligentization of safe driving systems. The common method is to use different method models to identify different driver attributes.

[0003] Regarding the driver attribute recognition method, a deep learning-based method is generally used to identify a certain attribute feature of the driver. The traditional method is to combine multiple models to identify the various attribute features of the driver. However, first of all, the traditional method has a single scene and is difficult to deal with the in-cab scenes with occlusion, poor lighting, multiple people, missing important attributes, etc. Secondly, if multiple models are used to identify the driver's attributes, the system's response speed will also slow down, the model size will also increase, the hardware requirements will increase accordingly, and in the case of multiple people, it cannot be ensured that all attribute features come from the same person. Finally, for the recognition of smoking and making phone calls, most current methods use anchor-based neural network models, which not only cannot be integrated into one model with other attributes, but also increases the difficulty of calculation and labeling.

[0004] Therefore, it is necessary to design a method that encompasses all important attribute characteristics of drivers while also having good real-time performance and accuracy. Summary of the invention

[0005] The embodiments of the present application provide a multi-task attribute recognition method, device, medium and equipment for driving scenes, which can adopt a multi-task network model to simultaneously recognize multiple characteristics of the driver and output an output result that can overall reflect the driver's driving state, thereby improving the accuracy of driver state recognition.

[0006] In a first aspect, an embodiment of the present application provides a multi-task attribute recognition method for a driving scene, the method comprising:

[0007] Obtaining image data of the driver in the driving scene camera video;

[0008] The image data is input into a pre-trained multi-task network model for attribute recognition; the attribute recognition tasks of the multi-task network model include: mask recognition, glasses recognition, smoking recognition and phone call recognition;

[0009] The multi-task attribute recognition result of the driver is determined according to the attribute recognition output result of the multi-task network model.

[0010] Furthermore, the image data is input into a trained multi-task network model for attribute recognition, including:

[0011] Preprocessing the image data;

[0012] The preprocessed image data is input into a pre-trained multi-task network model for attribute recognition, and the attribute recognition results of the multi-task network model for the target tasks of whether to wear a mask, whether to wear glasses, whether to smoke, and whether to make a phone call are obtained;

[0013] Cache the attribute recognition results of the target task;

[0014] After the attribute recognition results are cached for a specific time period, it is determined whether the attribute of the target task exists based on the determination ratio of the cached attribute recognition results; if so, the target task attribute of the sample image is determined.

[0015] Further, the image data is preprocessed, including:

[0016] Obtain a sample image and determine the encoding features of a face in the sample image;

[0017] Determine the driver's face frame using a face recognition model according to the coding features;

[0018] According to the face frame of the driver, adjustments are made, the driver's face is cropped to obtain a cropped image, and finally normalized.

[0019] Furthermore, the training of the multi-task network model includes:

[0020] The preprocessed image data is input into the multi-task main network to obtain a feature map, and the OSEM module is used to extract the feature maps for different attribute recognition tasks, and the feature maps are respectively input into respective attribute recognition modules, wherein the attribute recognition modules are a mask-wearing attribute recognition module, a glasses-wearing attribute recognition module, a smoking attribute recognition module, and a phone call attribute recognition module;

[0021] The output of the entire network is the feature map of each task and the attribute recognition result of each task. The output results of the attribute recognition of each task are whether to wear a mask, whether to wear glasses, whether to smoke, and whether to make a phone call;

[0022] Compare the attribute recognition results with the annotation results, reduce their differences through the corresponding loss function, and input the feature maps of each task into the corresponding loss function to optimize the network parameters;

[0023] Adjust the weights of the loss function to optimize the training results of the multi-task network model and reduce the training time.

[0024] Furthermore, before training the multi-task network model, the method further includes:

[0025] Label the sample image data with multi-task attributes;

[0026] Determining an initial architecture of the multi-task network model;

[0027] Determine the loss function.

[0028] Furthermore, the number of output categories for the attribute recognition tasks of whether to wear a mask, whether to smoke, and whether to make a phone call are all two categories;

[0029] The output categories of the attribute recognition task of whether to wear glasses include: wearing infrared blocking glasses, wearing ordinary glasses, and not wearing glasses.

[0030] Furthermore, for the attribute recognition modules of whether to wear a mask or glasses, each module is used for attribute recognition to output the results of whether to wear a mask or glasses.

[0031] Further, for the attribute recognition modules of whether to smoke or not and whether to make a phone call or not, each recognition module includes a subject recognition module and an attention module;

[0032] The subject recognition module is used for attribute recognition to output the result of whether to smoke or make a phone call;

[0033] The attention module includes an attention suggestion subnetwork and an attention network, which are used for attribute recognition and optimization of the attribute recognition results of the main network;

[0034] Furthermore, the attention proposal subnetwork mainly includes two fully connected layers in series, which are used to output the center point coordinates of the attention area and the side length of the attention area to perform cropping and enlarging processing on the original input image.

[0035] Furthermore, the attention network will perform network analysis on the cropped and enlarged image and output new results on whether to smoke or make a phone call. It can also crop and enlarge the image again as needed, perform network analysis, and output recognition results.

[0036] Furthermore, it is determined to calculate the classification loss and Lrank loss using the fine-grained features generated by the attention network to constrain the attribute analysis results of the main network and the attention network.

[0037] Furthermore, after the multi-task network model is trained, the method further includes:

[0038] Select the network architecture and model with the best attribute analysis results based on the test accuracy;

[0039] Delete the network and its parameters that do not participate in the final attribute recognition output to reduce the network model size and forward transmission time.

[0040] In a second aspect, an embodiment of the present application provides a multi-task attribute recognition device for a driving scene, the device comprising:

[0041] An image data acquisition module, used to acquire image data of the driver in the driving scene;

[0042] A recognition module, used for inputting the image data into a pre-trained multi-task network model for attribute recognition; the attribute recognition tasks of the multi-task network model include: mask recognition, glasses recognition, smoking recognition and phone call recognition;

[0043] The multi-task attribute recognition result determination module is used to determine the multi-task attribute recognition result of the driver according to the attribute recognition output result of the multi-task network model.

[0044] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a multi-task attribute recognition method for a driving scene as described in an embodiment of the present application.

[0045] In a fourth aspect, an embodiment of the present application provides a device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multi-task attribute recognition method for the driving scene as described in the embodiment of the present application is implemented.

[0046] The technical solution provided in the embodiment of the present application obtains the image data of the driver in the driving scene; inputs the image data into a pre-trained multi-task network model for attribute recognition; the attribute recognition tasks of the multi-task network model include: mask recognition, glasses recognition, smoking recognition and phone call recognition; according to the attribute recognition output result of the multi-task network model, the multi-task attribute recognition result of the driver is determined. By adopting the technical solution provided in the present application, a multi-task network model can be used to simultaneously recognize multiple features of the driver, and output an output result that can overall reflect the driver's driving state, thereby improving the accuracy of driver state recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of a multi-task attribute recognition method for a driving scene provided in an embodiment of the present application;

[0048] Figure 2 is a flow chart of the attribute recognition process of the multi-task network model provided in an embodiment of the present application;

[0049] Figure 3is a schematic diagram of the training process of the multi-task network model provided in an embodiment of the present application;

[0050] Figure 4 is a schematic diagram of an attention suggestion subnetwork provided in an embodiment of the present application;

[0051] Figure 5 is a schematic diagram of the structure of a multi-task attribute recognition device for a driving scene provided in an embodiment of the present application;

[0052] Figure 6 It is a structural schematic diagram of a device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings.

[0054] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0055] Figure 1 It is a flowchart of the multi-task attribute recognition method for driving scenes provided in an embodiment of the present application. This embodiment can be applicable to the case of driver status recognition. The method can be executed by the multi-task attribute recognition device for driving scenes provided in an embodiment of the present application. The device can be implemented by software and / or hardware and can be integrated into devices such as smart terminals.

[0056] like Figure 1 As shown, the multi-task attribute recognition method of the driving scene includes:

[0057] S110: Acquire image data of the driver in the driving scene.

[0058] The driver's image data can be obtained through ordinary cameras, infrared cameras, etc., and can be set at the rearview mirror position of the vehicle, or other positions, as long as the driver's face image can be obtained. According to the needs of the actual application scenario, an infrared 940 camera is used to build a collection hardware and software platform to collect video data in the cab of the vehicle.

[0059] In this embodiment, it can be understood that after the image data is acquired, the image data can be cleaned, stored, and annotated. Specifically, the collected data is batch extracted into image frames, and the image data is cleaned and the target box is annotated. The use of infrared cameras and ordinary cameras for data acquisition can ensure stable imaging output in complex environments, ensure the stability of image features, and thus facilitate the learning of network models, and can also alleviate or avoid problems such as backlighting to a large extent.

[0060] In this scheme, the collected RGB image can be input into the backbone network to extract the encoding features of the image; the driving scene face recognition neural network is used to analyze the encoding features, and if it is higher than the set threshold, the face frame coordinate value closest to the previous result is given; if it is lower than the set threshold, it is directly output: no driver, and the process ends; the input image is cropped according to the proportion of the face frame coordinate value, and the cropped image is normalized and unified in size.

[0061] S120, inputting the image data into a pre-trained multi-task network model for attribute recognition; the attribute recognition tasks of the multi-task network model include: mask recognition, glasses recognition, smoking recognition and phone call recognition.

[0062] The acquired image data can be input into a pre-trained multi-task network model. The recognition tasks of the multi-task network model include: mask recognition, glasses recognition, smoking recognition and phone call recognition. Different recognition tasks can be performed simultaneously. For example, after determining the features of the image, multiple recognition tasks can be completed simultaneously. If corresponding information is recognized, such as a mask, the information that the driver is wearing a mask is displayed in the output of the mask recognition task.

[0063] It is understandable that before training the multi-task network model, it is necessary to label the driver based on whether he is wearing a mask, glasses, smoking, or making a phone call. However, this method does not require labeling of location information and is a weakly supervised anchor-free method. Compared with traditional methods, it reduces the labeling task. In this way, the labeling information and the driver image will be used as input to train the model. In the application process, no labeling information is required, only the driver image is needed to output the judgment result of whether he is wearing a mask, glasses, smoking, or making a phone call.

[0064] S130 , determining a multi-task attribute recognition result of the driver according to the attribute recognition output result of the multi-task network model.

[0065] The output result of the model can be determined according to the output results of each recognition task, and after the model outputs the corresponding result, the multi-task attribute recognition result of the driver can be determined.

[0066] The technical solution provided in the embodiment of the present application obtains the image data of the driver in the driving scene; inputs the image data into a pre-trained multi-task network model for attribute recognition; the attribute recognition tasks of the multi-task network model include: mask recognition, glasses recognition, smoking recognition and phone call recognition; according to the attribute recognition output result of the multi-task network model, the multi-task attribute recognition result of the driver is determined. By adopting the technical solution provided in the present application, a multi-task network model can be used to simultaneously recognize multiple features of the driver, and output an output result that can overall reflect the driver's driving state, thereby improving the accuracy and real-time performance of driver state recognition.

[0067] Figure 2 is a flowchart of the attribute recognition process of the multi-task network model provided in the embodiment of the present application, such as Figure 2 As shown, the attribute recognition process of the multi-task network model includes:

[0068] S210: Preprocess the image data.

[0069] The collected and cropped RGB images are input into the backbone network to extract the encoding features of the image. Specifically, the backbone network uses the SE-ResNet18 network, and the output scale feature map is a 7*7 feature map. These 7*7 feature maps are then passed through the one squeeze multi-excitation (OSME) module to extract four different types of feature maps, namely, mask, glasses, smoking, and phone call feature maps, which are used to determine the features of different attribute recognition tasks.

[0070] In the technical solution, optionally, the image data is preprocessed, including:

[0071] Obtain a sample image and determine the encoding features of a face in the sample image;

[0072] Determine the driver's face frame using a face recognition model according to the coding features;

[0073] According to the face frame of the driver, adjustments are made, the driver's face is cropped to obtain a cropped image, and finally normalized.

[0074] Among them, the driving scene face recognition neural network is used to analyze the coded features. If it is higher than the set threshold, the face frame coordinate value closest to the previous result is given; if it is lower than the set threshold, it is directly output: no driver, and the process ends. Here, since the movement range of the driver's face in the vehicle is limited, the face frame closest to the previous result can be determined as the driver's face frame.

[0075] The face frame coordinate values ​​are appropriately enlarged in proportion, the input image is cropped, and the cropped image is normalized and resized.

[0076] S220, input the preprocessed image data into a pre-trained multi-task network model for attribute recognition, and obtain the attribute recognition results of the multi-task network model for the target tasks of whether to wear a mask, whether to wear glasses, whether to smoke, and whether to make a phone call.

[0077] In this scheme, a multi-task network model can be used to identify the cropped image to determine whether the person is wearing a mask, glasses or infrared blocking glasses, smoking, or making a phone call.

[0078] S230: Cache the attribute recognition result of the target task.

[0079] In this embodiment, optionally, before inputting the cropped image into the multi-task network model for training, the method further includes:

[0080] Label the sample image data with multi-task attributes;

[0081] Determining an initial architecture of the multi-task network model;

[0082] Determine the loss function.

[0083] Correspondingly, the initial architecture of the multi-task network model uses the SE-ResNet18 network, and the output scale feature map is a 7*7 feature map. These 7*7 feature maps are then extracted into four different types of feature maps, namely, mask, glasses, smoking, and phone call feature maps, which are used to determine the features of different attribute recognition tasks.

[0084] Determine the attribute classification loss generated by the four attribute recognition tasks when performing multi-task attribute recognition; the attribute classification loss is used to compare the differences between the network output results and the labeled results during the training process, and is forwarded and back-propagated to the multi-task network model to reduce these differences. These attribute classification losses will be multiplied by different weights, which will be determined according to the difficulty of calculating the attribute features.

[0085] Among them, 4 groups of 7*7 attention feature maps for 4 categories of different attribute recognition will be input into the metric loss function together.

[0086] Specific implementation of measuring loss: stretch 4 groups of 7*7 attention feature maps into vectors, calculate the similarity of different images in a batch, and calculate the attention similarity S of the same category of our task sa To compare the attention similarity S of different categories daHigh, so the metric loss function is:

[0087]

[0088] Here n is the number of the same category in a batch multiplied by the number of different categories. pos is the number of the same category.

[0089] The metric loss function here is mainly to make the attention feature maps of whether smoking, making phone calls, wearing masks, and wearing glasses mutually exclusive features, so that different attribute features have different attention recognition areas in the original image.

[0090] The preprocessed sample image data is matched one by one with the annotated multi-task attributes and arranged in order; the preprocessed sample image is input into the network, and the final output is compared with the annotated result according to the loss function, and the difference is reduced through repeated forward and backward propagation. Finally, the difference between the network output and the annotated result is converged to a negligible level, and the various parameters in the network are the required final result. The loss weight is adjusted according to the multi-task attribute recognition result and the convergence speed to optimize the training result and speed of the multi-task network model. Specifically, the above method can be used to adjust the weight of the loss function to achieve the purpose of optimizing the multi-task network model.

[0091] S240, after the attribute recognition results of a specific time length are cached, judging whether the attribute of the target task exists based on the judgment ratio of the cached attribute recognition results; if so, determining the target task attribute of the sample image.

[0092] The attribute recognition results are cached for a certain period of time. The cache time can be set as needed. Select N consecutive frames of images. If it is greater than a certain set ratio, it is judged that the attribute exists. For example: Detect mask status: The starting state is not wearing a mask. Starting from the Nth frame, the sum of the mask status cache values ​​of the previous N frames is divided by N to get the mask wearing ratio. If its value is greater than a certain ratio, it is concluded that the mask wearing state exists. Similarly, the glasses state, smoking state, and phone call state are judged.

[0093] In this embodiment, the multi-task network model attribute recognition training process includes:

[0094] The preprocessed image data is input into the multi-task main network to obtain a feature map, and the OSEM module is used to extract the feature maps for different attribute recognition tasks, and the feature maps are respectively input into respective attribute recognition modules, wherein the attribute recognition modules are a mask-wearing attribute recognition module, a glasses-wearing attribute recognition module, a smoking attribute recognition module, and a phone call attribute recognition module;

[0095] The output of the entire network is the feature map of each task and the attribute recognition result of each task. The output results of the attribute recognition of each task are whether to wear a mask, whether to wear glasses, whether to smoke, and whether to make a phone call;

[0096] Compare the attribute recognition results with the annotation results, reduce their differences through the corresponding loss function, and input the feature maps of each task into the corresponding loss function to optimize the network parameters;

[0097] Adjust the weights of the loss function to optimize the training results of the multi-task network model and reduce the training time.

[0098] In the above technical solution, the optional number of output categories of the attribute recognition tasks of whether to wear a mask, whether to smoke, and whether to make a phone call are two categories;

[0099] The output categories of the task of identifying whether to wear glasses include: wearing infrared blocking glasses, wearing ordinary glasses, and not wearing glasses.

[0100] Among them, as we all know, there are various network structures, and you can choose the required network structure. The multi-task network structure used here is SE-Resnet18, and the initial architecture is adjusted. For example, the input size is set to 112*112, and the 7*7 convolution kernel (stride=2) of the first convolution layer is replaced with a 3*3 convolution kernel (stride=1), the first maximum pooling layer is deleted, and the second convolution layer of the first resnet block stage has a stride of 2, and in each residual block, a BN layer and dropout are added. The structure is BN layer, Prelu, convolution, BN layer, Prelu layer, convolution, dropout layer, and the dropout ratio of the dropout layer in the residual block is 0%-20%.

[0101] The output 7×7 feature maps will be passed through the one squeeze multi-excitation (OSME) module, which is mainly composed of 4 compression and excitation networks. The 4 results output by this module will be applied to the previous feature maps to obtain the feature maps of the four tasks. After the four feature maps are pooled, the feature vectors are input into the four attribute recognition modules respectively. The outputs of the four attribute recognition modules are whether to wear a mask, whether to wear glasses, whether to smoke and smoking APN, whether to make a phone call and the attention proposal sub-network (APN: attention proposal sub-network). Whether to wear a mask, whether to wear glasses, whether to smoke and whether to make a phone call are attribute recognition tasks, and the smoking APN and the phone call APN will be used as the input of the secondary attention model for cropping and enlarging the image region. The attention proposal sub-network includes two fully connected layers in series, which are used to output the coordinates of the center point of the attention region and the side length of the attention region to perform cropping and enlarging processing on the feature map.

[0102] The enlarged image will be input into the first-level attention model to re-judge the attributes: whether to smoke and the smoking APN, whether to make a phone call and the phone call APN. The APN generated by the first-level attention model will further crop and enlarge the image and input it into the second-level attention model to re-judge the attributes.

[0103] The first-level attention neural network model and the second-level attention neural network model here have the same network structure as the main network model, and will output attribute recognition results as described above. Therefore, there will be three attribute recognition results for smoking and making phone calls respectively. Finally, the model with the highest accuracy is selected as the final model (already selected during training) according to the test, and the only attribute recognition result is output.

[0104] Here, because it is difficult to judge whether smoking or making phone calls, the attention mechanism is further used to optimize the results, but it is not limited to these two tasks. You can choose which tasks need to use the attention mechanism according to your needs.

[0105] The number of outputs for the tasks with attributes of masks, infrared blocking glasses and ordinary glasses, smoking, and making phone calls are 2, 3, 2, and 2 respectively.

[0106] In the above technical solution, optionally, after the multi-task network model is trained, the method further includes:

[0107] Select the network architecture and model with the best attribute analysis results based on the test accuracy;

[0108] Delete the network and its parameters that do not participate in the final attribute recognition output to reduce the network model size and forward transmission time.

[0109] Figure 3 Schematic diagram of the training process of the multi-task network model provided in the embodiment of the present application. Figure 3 As shown in the figure, the driving scene face recognition neural network is used to analyze the coded features. If it is higher than the set threshold, the face frame coordinate value closest to the previous result is given; if it is lower than the set threshold, it is directly output: no driver, and the process ends. When it is higher than the set threshold, the face frame closest to the previous result is cropped. After the image is cropped and input into the backbone network, the mask attribute, glasses attribute, smoking attribute, and phone call attribute can be identified.

[0110] Figure 4 Schematic diagram of the attention suggestion subnetwork provided by the embodiment of the present application. Figure 4 As shown,

[0111] Assuming the attention region of interest is a square (i.e. the output of APN), the output x t ,y t , l t Among them, after the softmax function, the output probability p1.

[0112] Among them, x t ,y t , l t The horizontal and vertical coordinates of the center point of the coordinate domain of the center point of the region, and the side length of the square respectively. The original face is cropped and enlarged according to the output of the APN sub-network to generate a local area: face scale2, which is used as the input of the new sub-network. This is the first-level attention model.

[0113] The structural design of the new sub-network is the same as the main network. It then outputs the classification network and the attention suggestion sub-network APN, and outputs the probability p2 through the softmax function. The APN is the same as the previous APN network. It is trained from the extracted features to obtain the attention area information. This is the second-level attention model.

[0114] According to the output of the secondary attention model, the highly localized area is cropped and enlarged: face scale3, and face scale3 is generated. After the softmax function, the probability p3 is output. As a new input, it is input into the same sub-network as the main network, and finally output to the classification network.

[0115] Through these three-scale sub-networks, the initial face is transformed into multi-scale fine-grained regions.

[0116] The glasses and mask subtasks directly use the softmax function to represent the probability of the existence of this attribute for each element.

[0117] The loss weight is adjusted according to the attribute recognition results of the at least two tasks to complete the training of the multi-task network model.

[0118] On the basis of the above technical solution, optionally, different tasks can be selected to apply the attention mechanism as needed, thereby optionally determining the fine-grained feature classification loss and Lrank loss generated by the attention suggestion subnetwork.

[0119] The fine-grained classification loss is the same as other classification losses, which can be softmax or softmax crossentropy, etc. The focal loss is used here.

[0120] The Lrank loss function here is mainly to make the classification effect of the second-level attention image better than the classification effect produced by the first-level attention image, so that the attention areas can be transformed into more recognizable attention areas step by step.

[0121] Specifically, the labeled information of the driver's attributes (mask status, glasses status, smoking status, and phone call status) is used to train the multi-task driver attribute recognition task using focal loss.

[0122] Since almost all attributes have an imbalanced overall sample ratio, the focal loss function is used to reduce the weight of negative samples. The focal loss formula is as follows:

[0123]

[0124] in It is a balancing factor used to balance the uneven ratio of positive and negative samples.

[0125] For the smoking and phone call models, in addition to the classification loss focal loss caused by attribute classification, there is also the inter-scale classification loss Lrank caused by fine-grained feature classification.

[0126]

[0127] Represents the classification output of the t-1 layer attention model in the correct categories: probability

[0128] Through experiments, we set appropriate weights for the loss of each task. The total loss formula is as follows:

[0129] loss = W1 * loss fl +W2*loss fl +W3*(lossfl +L rank )+W4*(loss fl +L rank )+W5*L np ;

[0130] Among them, W1, W2, W3, W4, and W5 are the weights of the loss of each task, which are manually adjusted through experiments to make the loss of each task at the same order of magnitude.

[0131] The multi-task network model provided by this solution can achieve the purpose of simultaneously performing multi-task attribute recognition on the driver's status, and uses multi-granularity to recognize whether the driver is smoking or making a phone call, thereby improving the accuracy of model training.

[0132] Figure 5 Schematic diagram of the structure of the multi-task attribute recognition device for driving scenes provided in the embodiment of the present application. Figure 5 As shown, the multi-task attribute recognition device of the driving scene includes:

[0133] An image data acquisition module 510 is used to acquire image data of the driver in the driving scene;

[0134] The recognition module 520 is used to input the image data into a pre-trained multi-task network model for attribute recognition; the attribute recognition tasks of the multi-task network model include: mask recognition, glasses recognition, smoking recognition and phone call recognition;

[0135] The multi-task attribute recognition result determination module 530 is used to determine the multi-task attribute recognition result of the driver according to the attribute recognition output result of the multi-task network model.

[0136] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method.

[0137] The embodiment of the present application also provides a storage medium containing computer executable instructions, wherein the computer executable instructions are used to perform a multi-task attribute recognition method for a driving scene when executed by a computer processor, the method comprising:

[0138] Acquire image data of the driver in the driving scene;

[0139] The image data is input into a pre-trained multi-task network model for attribute recognition; the attribute recognition tasks of the multi-task network model include: mask recognition, glasses recognition, smoking recognition and phone call recognition;

[0140] The multi-task attribute recognition result of the driver is determined according to the attribute recognition output result of the multi-task network model.

[0141] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the computer system in which the program is executed, or may be located in a different second computer system, which is connected to the computer system via a network (such as the Internet). The second computer system can provide program instructions to the computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations (for example, in different computer systems connected by a network). The storage medium may store program instructions (for example, embodied as a computer program) that can be executed by one or more processors.

[0142] Of course, the storage medium containing computer executable instructions provided in an embodiment of the present application is not limited to the multi-task attribute recognition operation of the driving scene as described above, and can also execute related operations in the multi-task attribute recognition method of the driving scene provided in any embodiment of the present application.

[0143] An embodiment of the present application provides a device in which the multi-task attribute recognition device for driving scenes provided by the embodiment of the present application can be integrated. Figure 6 Schematic diagram of a device provided in an embodiment of the present application. Figure 6 As shown, this embodiment provides a device 600, which includes: one or more processors 620; a storage device 610, which is used to store one or more programs. When the one or more programs are executed by the one or more processors 620, the one or more processors 620 implement the multi-task attribute recognition method of the driving scene provided by the embodiment of the present application. The method includes:

[0144] Acquire image data of the driver in the driving scene;

[0145] The image data is input into a pre-trained multi-task network model for attribute recognition; the attribute recognition tasks of the multi-task network model include: mask recognition, glasses recognition, smoking recognition and phone call recognition;

[0146] The multi-task attribute recognition result of the driver is determined according to the attribute recognition output result of the multi-task network model.

[0147] Of course, those skilled in the art can understand that the processor 620 also implements the technical solution of the multi-task attribute recognition method of the driving scene provided in any embodiment of the present application.

[0148] Figure 6 The device 600 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0149] like Figure 6 As shown, the device 600 includes a processor 620, a storage device 610, an input device 630, and an output device 640; the number of processors 620 in the device can be one or more. Figure 6 A processor 620 is taken as an example; the processor 620, the storage device 610, the input device 630 and the output device 640 in the device can be connected via a bus or other means. Figure 6 The connection via bus 650 is taken as an example.

[0150] The storage device 610, as a computer-readable storage medium, can be used to store software programs, computer executable programs and module units, such as program instructions corresponding to the multi-task attribute recognition method of the driving scene in the embodiment of the present application.

[0151] The storage device 610 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the storage device 610 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the storage device 610 may further include a memory remotely arranged relative to the processor 620, and these remote memories may be connected via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0152] The input device 630 may be used to receive input numbers, character information or voice information, and generate key signal input related to user settings and function control of the device. The output device 640 may include a display screen, a speaker and other devices.

[0153] The device provided in the embodiment of the present application can adopt a multi-task network model to simultaneously identify multiple characteristics of the driver and output an output result that can overall reflect the driver's driving status, thereby improving the accuracy of driver status recognition.

[0154] The multi-task attribute recognition device, storage medium and equipment for driving scenes provided in the above embodiments can execute the multi-task attribute recognition method for driving scenes provided in any embodiment of the present application, and have the corresponding functional modules and beneficial effects of executing the method. For technical details not described in detail in the above embodiments, please refer to the multi-task attribute recognition method for driving scenes provided in any embodiment of the present application.

[0155] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A multi-task attribute recognition method for driving scenes, It is characterized in that The method includes: Acquire image data of the driver in the driving scene; The image data is input into a pre-trained multi-task network model for attribute recognition; the attribute recognition tasks of the multi-task network model include: mask recognition, glasses recognition, smoking recognition and phone call recognition, wherein different recognition tasks are performed simultaneously; Determining a multi-task attribute recognition result of the driver according to the attribute recognition output result of the multi-task network model; The step of inputting the image data into a pre-trained multi-task network model for attribute recognition includes: Acquire a sample image and determine the encoding features of a face in the sample image; Determine the driver's face frame using a face recognition model according to the coding features; According to the face frame of the driver, adjustment is performed, and then the face of the driver is cropped to obtain a cropped image, and finally normalized; the normalized image data is input into a pre-trained multi-task network model for attribute recognition, and the attribute recognition results of the multi-task network model for the target tasks of whether to wear a mask, whether to wear glasses, whether to smoke, and whether to make a phone call are obtained; the attribute recognition results of the target tasks are cached; after the attribute recognition results are cached for a specific time, the cached attribute recognition results are judged according to the judgment ratio whether the attributes of the target task exist; if so, the target task attributes of the sample image are determined; Determining the driver's face frame by using a face recognition model according to the coding feature includes: The driving scene face recognition neural network is used to analyze the coding features. If the coding features are higher than the set threshold, the face frame coordinate value closest to the previous result is given; if the coding features are lower than the set threshold, the following is directly output: no driver, and the process ends. The training of the multi-task network model includes: The normalized image data is input into the multi-task main network to obtain a feature map, and the OSEM module is used to extract the feature maps for different attribute recognition tasks, and the feature maps are respectively input into respective attribute recognition modules, which are respectively a mask-wearing attribute recognition module, a glasses-wearing attribute recognition module, a smoking attribute recognition module, and a phone call attribute recognition module; The output of the entire network is the feature map of each task and the attribute recognition result of each task. The output results of the attribute recognition of each task are whether to wear a mask, whether to wear glasses, whether to smoke, and whether to make a phone call; Compare the attribute recognition results with the annotation results, reduce their differences through the corresponding loss function, and input the feature maps of each task into the corresponding loss function to optimize the network parameters; Adjust the weights of the loss function to optimize the training results of the multi-task network model and reduce the training time; For the attribute recognition modules of whether to smoke or not and whether to make a phone call or not, each recognition module includes a subject recognition module and an attention module; The subject recognition module is used for attribute recognition to output the result of whether to smoke or make a phone call; The attention module includes an attention suggestion subnetwork and an attention network, which are used for attribute recognition and optimization of the attribute recognition results of the main network; Determine to calculate the classification loss and Lrank loss using the fine-grained features generated by the attention network to constrain the attribute analysis results of the main network and the attention network.

2. The method according to claim 1, It is characterized in that Before training the multi-task network model, the method further includes: Label the sample image data with multi-task attributes; Determining an initial architecture of the multi-task network model; Determine the loss function.

3. The method according to claim 1, It is characterized in that The number of output categories for the attribute recognition tasks of whether to wear a mask, whether to smoke, and whether to make a phone call are all two categories; The output categories of the attribute recognition task of whether to wear glasses include: wearing infrared blocking glasses, wearing ordinary glasses, and not wearing glasses.

4. The method according to claim 1, It is characterized in that For the attribute recognition modules of whether to wear a mask or glasses, each module is used for attribute recognition to output the results of whether to wear a mask or glasses.

5. The method according to claim 1, It is characterized in that The attention proposal subnetwork includes two fully connected layers in series, which are used to output the center point coordinates of the attention region and the side length of the attention region to perform cropping and enlarging processing on the original input image.

6. The method according to claim 1, It is characterized in that The attention network will perform network analysis on the cropped and enlarged image and output a new result of whether to smoke or make a phone call, or it will crop and enlarge the image again as needed, perform network analysis, and output the recognition result.

7. The method according to claim 1, It is characterized in that After the multi-task network model is trained, the method further includes: Select the network architecture and model with the best attribute analysis results based on the test accuracy; Delete the network and its parameters that do not participate in the final attribute recognition output to reduce the network model size and forward transmission time.

8. A multi-task attribute recognition device for driving scenes, It is characterized in that The device includes: An image data acquisition module, used to acquire the driver's image data from the driving scene video; A recognition module is used to input the image data into a pre-trained multi-task network model for attribute recognition; the attribute recognition tasks of the multi-task network model include: mask recognition, glasses recognition, smoking recognition and phone call recognition; wherein different recognition tasks are performed simultaneously; A multi-task attribute recognition result determination module, used to determine the multi-task attribute recognition result of the driver according to the attribute recognition output result of the multi-task network model; The identification module is specifically used for: Acquire a sample image and determine the encoding features of a face in the sample image; Determine the driver's face frame using a face recognition model according to the coding features; According to the face frame of the driver, adjustment is performed, and then the face of the driver is cropped to obtain a cropped image, and finally normalized; the normalized image data is input into a pre-trained multi-task network model for attribute recognition, and the attribute recognition results of the multi-task network model for the target tasks of whether to wear a mask, whether to wear glasses, whether to smoke, and whether to make a phone call are obtained; the attribute recognition results of the target tasks are cached; after the attribute recognition results are cached for a specific time, the cached attribute recognition results are judged according to the judgment ratio whether the attributes of the target task exist; if so, the target task attributes of the sample image are determined; The identification module is further used for: The driving scene face recognition neural network is used to analyze the coding features. If the coding features are higher than the set threshold, the face frame coordinate value closest to the previous result is given; if the coding features are lower than the set threshold, the following is directly output: no driver, and the process ends. The training of the multi-task network model includes: The normalized image data is input into the multi-task main network to obtain a feature map, and the OSEM module is used to extract the feature maps for different attribute recognition tasks, and the feature maps are respectively input into respective attribute recognition modules, which are respectively a mask-wearing attribute recognition module, a glasses-wearing attribute recognition module, a smoking attribute recognition module, and a phone call attribute recognition module; The output of the entire network is the feature map of each task and the attribute recognition result of each task. The output results of the attribute recognition of each task are whether to wear a mask, whether to wear glasses, whether to smoke, and whether to make a phone call; Compare the attribute recognition results with the annotation results, reduce their differences through the corresponding loss function, and input the feature maps of each task into the corresponding loss function to optimize the network parameters; Adjust the weights of the loss function to optimize the training results of the multi-task network model and reduce the training time; For the attribute recognition modules of whether to smoke or not and whether to make a phone call or not, each recognition module includes a subject recognition module and an attention module; The subject recognition module is used for attribute recognition to output the result of whether to smoke or make a phone call; The attention module includes an attention suggestion subnetwork and an attention network, which are used for attribute recognition and optimization of the attribute recognition results of the main network; Determine to calculate the classification loss and Lrank loss using the fine-grained features generated by the attention network to constrain the attribute analysis results of the main network and the attention network.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the multi-task attribute recognition method for a driving scene as described in any one of claims 1 to 7 is implemented.

10. A device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the multi-task attribute recognition method for the driving scene according to any one of claims 1 to 7 is implemented.