A method for recognizing human appendages based on multi-task learning

Abstract

This invention proposes a multi-task learning-based method for recognizing human appendages. It employs a training and detection approach that combines target detection and attribute classification. During the training phase, the multi-task learning method enables the network to extract features at multiple levels, including appendages and human attributes. In the detection phase, a combination of various fine-tuning methods is used. Experiments demonstrate significant improvements in both algorithm accuracy and iteration speed for recognizing human appendages.

Description

Technical Field

[0001] This invention belongs to the field of deep learning multi-task representation technology, and relates to a method for recognizing human appendages based on multi-task learning. Background Technology

[0002] Body accessories refer to items held or worn on the body, such as backpacks, masks, hats, and mobile phones. Identification of these body accessories is required in situations involving public safety, workplace safety, and disease prevention and control.

[0003] With the advent of artificial intelligence technology, the identification of such accessories has begun to become more intelligent. General methods for accessory identification fall into two categories: target detection and attribute recognition.

[0004] Object detection methods refer to inputting a single frame image and outputting a bounding map of appendages on the human body through detection algorithms. These algorithms can locate the positions of human appendages, but they have the following drawbacks:

[0005] 1. The algorithm logic has flaws. Although it can locate human appendages, it cannot determine whether they meet the correct characteristics. For example, in situations where wearing a mask is required, the target detection algorithm can detect the mask, but it cannot determine whether the mask is being worn correctly.

[0006] 2. Object detection algorithms have a high overhead. Generally, object detection algorithms are several times slower than attribute classification tasks, and they have higher requirements for computing devices.

[0007] Another approach is attribute recognition algorithms, which use classification models trained on a large number of images to identify human figures in images. These models have lower hardware overhead and can accurately match the recognition logic for accessories, but their drawback is poor generalization ability. They don't perform well in new scenarios.

[0008] Therefore, this invention proposes a human appendage recognition method based on multi-task learning. It uses a training and detection method that combines target detection and attribute classification to more accurately acquire full-image and local features. Tests show that this invention outperforms simple target detection algorithms (YOLOv5, RetinaNet) and attribute classification algorithms (ResNet series). Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention provides a method for recognizing human appendages based on multi-task learning. This method combines the feature extraction capabilities of target detection with the discriminative logic of classification algorithms, solving the problem that classification algorithms require large amounts of labeled datasets. It facilitates customized solutions for users while improving algorithm accuracy.

[0010] The technical solution adopted in this invention is as follows:

[0011] A method for recognizing human appendages based on multi-task learning includes the following:

[0012] S1. Surveillance cameras were set up at the application location to collect video data. Human bounding boxes were collected using an open-source human detection algorithm. Classification labels and bounding boxes and attributes of objects in the images were manually labeled to construct the dataset.

[0013] S2. Using the above dataset, train the algorithm backbone in the human appendage representation training process based on multi-task parameter sharing; thus obtaining the human appendage recognition network.

[0014] The training process for human appendage representation based on multi-task parameter sharing includes a skeleton network, an object detection sub-network, and an attribute classification sub-network (hereinafter referred to as a classifier).

[0015] The backbone network is ResNet50, which accepts input image I and feature maps C3, C4, and C5 of the 3rd, 4th, and 5th convolutional layers as the output of the backbone network.

[0016] The target detection subnetwork uses the RetinaNet network and includes an FPN module, a detection block graph subnetwork, and a detection classification subnetwork. The FPN module accepts C3, C4, and C5 as inputs. The detection block graph subnetwork and the detection classification subnetwork take the output of the FPN as input and output the block graph B of the i-th accessory, respectively. i and types i ;

[0017] The classifier takes C5 as input and may contain multiple classifiers. The j-th classifier outputs the image attribute a after passing through a pooling layer and a fully connected layer. j .

[0018] S3. Input the image I to be identified into the trained human appendage recognition network to obtain the appendage recognition result.

[0019] S4. If new accessory recognition tasks arise in the future, the fine-tuning scheme of this invention can be used for rapid iteration. This invention proposes different fine-tuning schemes for different needs:

[0020] In typical applications, if there are enough samples (more than 1000 samples), different classifiers can be assigned to different attribute tasks. During backpropagation, the parameter weights of the backbone network are not updated; only the parameter weights of the corresponding classifiers are updated. This allows for a very short training cycle to complete one iteration of the algorithm.

[0021] For applications requiring higher accuracy, the S2 characterization method can be used to train the multi-task algorithm structure in the network. In the training configuration, the label category of the detection and classification sub-network is set to binary classification of the new appendage and the background. The label category of the new classifier is set to present / absent for the new appendage.

[0022] Preferably, the classifier is trained for classification by minimizing the cross-entropy loss. The object detection subnetwork is trained for detection by minimizing the focal loss function.

[0023] Preferably, in the skeleton network, all input given images I are pre-uniformed to the same size.

[0024] Preferably, all human appendage recognition datasets undergo data augmentation processing.

[0025] Preferably, the skeleton network module is initialized using network weights pre-trained with ImageNet. It is then pre-trained once on the open-source person re-identification dataset (Market1501). Attached Figure Description

[0026] Figure 1 A flowchart for training human appendage representation based on multi-task parameter sharing;

[0027] Figure 2 Flowchart for fine-tuning training of common attributes;

[0028] Figure 3 This is a flowchart of high-precision fine-tuning training based on multi-task learning. Detailed Implementation

[0029] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments. The technical features of each embodiment of the present invention can be combined accordingly, provided that there is no mutual conflict.

[0030] The present invention provides a method for recognizing human appendages based on multi-task learning, comprising:

[0031] S1. Video data is collected via surveillance cameras in situations where this invention needs to be applied. Human bounding boxes are collected using an open-source human detection algorithm, and classification labels, bounding boxes of objects in the images, and their attributes are manually labeled. A dataset is then constructed.

[0032] S2. Using the above dataset, train the algorithm backbone in the training process of human appendage representation based on multi-task parameter sharing.

[0033] The multi-task-based parameter sharing training process for human appendage representation is as follows: Figure 1As shown, it includes a skeleton network, an object detection subnetwork, and an attribute classification subnetwork (hereinafter referred to as the classifier).

[0034] The backbone network is ResNet50, which accepts input image I and feature maps C3, C4, and C5 of the 3rd, 4th, and 5th convolutional layers as the output of the backbone network.

[0035] The target detection subnetwork uses the RetinaNet network and includes an FPN module, a detection block graph subnetwork, and a detection classification subnetwork. The FPN module accepts C3, C4, and C5 as inputs. The detection block graph subnetwork and the detection classification subnetwork take the output of the FPN as input and output the block graph B of the i-th accessory, respectively. i and types i ;

[0036] The classifier takes C5 as input and may contain multiple classifiers. The j-th classifier outputs the image attribute a after passing through a pooling layer and a fully connected layer. j .

[0037] S3. Input the image I to be identified into the trained human appendage recognition network to obtain the appendage recognition result.

[0038] S4. If new accessory recognition tasks arise in the future, the fine-tuning scheme of this invention can be used for rapid iteration. This invention proposes different fine-tuning schemes for different needs.

[0039] In typical applications, if there are enough samples (more than 1000 samples) and higher speed is required, the object detection subnetwork can be abandoned during fine-tuning, and only the original classification network structure can be used. For example... Figure 2 As shown, by simply assigning different classifiers to different attribute tasks and not updating the parameter weights of the backbone network during backpropagation, but only updating the corresponding classifier parameter weights, an algorithm iteration can be completed in a very short training cycle.

[0040] For applications requiring higher accuracy, the multi-task algorithm structure in the S2 representation training network can continue to be used, such as... Figure 3 As shown, in the training configuration, the label category of the appendage classification subnetwork is set to binary classification of the new appendage and the background. The label category of the new classifier is set to present / absent of the new appendage.

[0041] The above method will be applied to a specific embodiment below to demonstrate the specific implementation process and technical effects of the present invention.

[0042] Example

[0043] During the production process, due to the needs of epidemic prevention and control and safe production, a company needs to automatically detect multiple accessories such as employees' masks, safety helmets, and gas alarms.

[0044] 1. Dataset Construction

[0045] In this embodiment, a data collection system was established, collecting a total of 100 hours of video data through surveillance cameras installed inside the company and at construction sites. One frame was extracted every 15 seconds, and over 7500 human bounding boxes were collected using an open-source human detection algorithm. 3000 positive samples of people wearing masks and over 2500 positive samples of people wearing safety helmets were obtained through manual annotation. The attached parts of the positive sample data were also annotated with bounding boxes. The remaining data were used as negative samples. 15% of the total data was randomly selected as the test set.

[0046] 2. Representation Training of Algorithm Skeleton

[0047] In this embodiment, ResNet50 is used as the algorithm backbone. An open-source pedestrian attribute pre-trained model is selected as the initial model weights. During the representation training process, since the vast majority of data consists of standing human figures, the input image size is set to 256*128. All the positive sample images obtained above are first used as the dataset for the detection task, with three categories: background, mask, and helmet. Then, all positive and negative sample images are used as the dataset for two attribute classification tasks: with / without mask and with / without helmet, for a total of two attributes and four labels.

[0048] During training, there is one detection task and two classification tasks, so there is one FPN module, one classification subnetwork, one block diagram subnetwork, and two attribute classifiers.

[0049] 3. Fine-tuning of downstream attribute tasks

[0050] Analyzing the two types of accessories mentioned above, masks and safety helmets have a very sufficient number of positive samples;

[0051] Due to the need for epidemic prevention and control, masks are required in all public places within the company. This involves a large number of people and a wide range of applications, so the speed requirements for the algorithm are high.

[0052] Safety helmets are only required to be worn in the production workshop, where high precision is required.

[0053] In this case, different fine-tuning methods were applied to these two types of accessories to meet different needs and with different amounts of data.

[0054] 3.1 Example of a common accessory: face mask

[0055] For tasks like mask making with sufficient sample data, and where there are high requirements for algorithm speed, iteration speed, and hardware cost control, using a conventional auxiliary fine-tuning scheme is the most reasonable choice.

[0056] After the representation training is completed, a pre-trained algorithm model is obtained. The classifier continues to use the structure of the attribute task classifier from the representation training.

[0057] The model is 256*128, and the dataset settings are consistent with the mask subtask in the representation training.

[0058] Keep the skeleton model unchanged and only fine-tune the classifier weights. During fine-tuning, two configurations can be tried for classifier initialization:

[0059] 1. Retain the corresponding classifier weights from the representation training.

[0060] 2. Randomly initialize the classifier weights.

[0061] Experiments have shown that the accuracy difference between the two configurations is not significant. Configuration 1 has higher accuracy in the early stages of training, while configuration 2 has higher accuracy at the end of training.

[0062]

[0063] 3.2 Example of high-precision accessory implementation: safety helmet

[0064] For tasks like safety helmets where there is sufficient sample data and high precision is required for the algorithm, the above-mentioned high-precision accessory implementation scheme is adopted.

[0065] During fine-tuning, the dataset for the detection task becomes a binary classification detection task involving background and safety helmets, with the classification task being the same as the safety helmet task in the representation training.

[0066] During the output process, the output of the classification task is used as the final output.

[0067] This case also compares several methods:

[0068] 1. Adopt the conventional solution in section 3.1.

[0069] 2. High-precision scheme 1: Directly use the output of the classification task as the final output.

[0070] 3. High-precision solution 2 adds a logical AND operation to the results of the subtasks. A helmet is only determined to be worn when both the detection and classification tasks output helmet labels simultaneously.

[0071] While high-precision scheme 2 is slightly less accurate than high-precision scheme 1, it has a higher recall value for negative samples, i.e., those not wearing helmets. For safety production requirements, high-precision scheme 2 aligns with the actual logic of this algorithm.

[0072]

Claims

1. A method for recognizing human appendages based on multi-task learning, characterized in that, Including the following: S1. Set up surveillance cameras at the application location to collect video data: Use open-source human detection algorithms to collect human bounding boxes, and use manual annotation to set classification labels and bounding boxes and attributes of attachments in the images, thereby constructing a dataset; S2. Using the above dataset, train the human appendage recognition algorithm in the human appendage representation training process based on multi-task parameter sharing; The aforementioned human appendage recognition algorithm includes a skeleton network, a target detection sub-network, and an attribute classification sub-network, i.e., a classifier; The backbone network is ResNet50, which accepts the input image and uses the feature maps C3, C4, and C5 of the 3rd, 4th, and 5th convolutional layers as the output of the backbone network. The target detection subnetwork uses the RetinaNet network and includes an FPN module, a detection block graph subnetwork, and a detection classification subnetwork. The FPN module accepts C3, C4, and C5 as inputs, while the detection block graph subnetwork and the detection classification subnetwork take the output of the FPN as input and output the block graph B of the i-th accessory, respectively. i and types i ; The classifier takes C5 as input and may contain multiple classifiers. The j-th classifier outputs the image attribute a after passing through a pooling layer and a fully connected layer. j ; S3. Input the image I to be identified into the trained human appendage recognition network to obtain the appendage recognition result; S4. If new accessory recognition tasks arise in the future, a fine-tuning scheme will be adopted for rapid iteration.

2. The method for recognizing human appendages based on multi-task learning according to claim 1, characterized in that, The fine-tuning scheme includes the following: In general applications, if there are more than 1000 training samples, it is only necessary to assign different classifiers to different attribute tasks, and during backpropagation, the parameter weights of the backbone network are not updated, but only the parameter weights of the corresponding classifiers are updated to quickly complete one iteration of the algorithm.

3. The method for recognizing human appendages based on multi-task learning according to claim 1, characterized in that, The fine-tuning scheme includes: if a new appendage is added, and the accuracy requirement for the new appendage is higher, the multi-task algorithm structure in the representation training network in S2 is continued to be used. In the training configuration, an appendage classification sub-network is set up, and its label category is binary classification of the new appendage and the background. The label category of the new classifier is set to present / absent the new appendage.

4. The method for recognizing human appendages based on multi-task learning according to claim 1, characterized in that, The classifier is trained by minimizing the cross-entropy loss.

5. The method for recognizing human appendages based on multi-task learning according to claim 1, characterized in that, The target detection subnetwork is trained by minimizing the focal loss function.

6. The method for recognizing human appendages based on multi-task learning according to claim 1, characterized in that, In the skeleton network, all input given images I are pre-uniformed to the same size.

7. The method for recognizing human appendages based on multi-task learning according to claim 1, characterized in that, All of the aforementioned human appendage recognition datasets have undergone data augmentation processing.

8. The method for recognizing human appendages based on multi-task learning according to claim 1, characterized in that, The skeleton network is initialized with network weights pre-trained in ImageNet and pre-trained on the open-source person re-identification dataset Market1501.

Images