Steward clothing identification method and device based on intelligent multi-label classification

Abstract

The invention relates to a steward clothing identification method and device based on intelligent multi-label classification, and belongs to the technical field of computer vision and artificial intelligence. According to the method, the relevance between clothes and body parts is considered, when the trainman image is used for recognizing the clothes, features corresponding to coordinates of key skeleton points in the trainman image and trainman image features are fused to obtain fused features, the fused features are processed by using the multi-label classification model, and the recognition accuracy of the trainman image is improved. Therefore, identification of the costume of the steward is realized. According to the invention, during identification, the costume identification precision of the steward is improved through compensation of the coordinates of the key skeleton points. In addition, follow-up recognition is triggered in a key skeleton point coordinate matching mode, and unnecessary invalid detection is reduced.

Description

Technical Field

[0001] The present invention relates to a method and device for recognizing flight attendant clothing based on intelligent multi-label classification, and belongs to the technical field of computer vision and artificial intelligence. Background Art

[0002] With the continuous expansion of my country's railway transportation scale and the continuous improvement of its intelligent level, train attendants, as an important position serving passengers and ensuring driving safety, their professional image and appearance directly affect the overall service quality and public image of railway enterprises. Therefore, it is very important to check the appearance of train attendants before they go on duty.

[0003] In the existing technology, the recognition of flight attendant clothing (clothing and accessories) usually relies on target detection models (such as YOLO, Faster R-CNN, etc.). By inputting the acquired image of the flight attendant clothing to be identified into the target detection model, the clothing area can be directly located to realize the recognition of the clothing (clothing and accessories). Although the above method can realize the automatic recognition of flight attendant clothing, it has the following problems: 1) For some accessories, the detection person must be in a certain posture to be detected; 2) intensive detection is required for the entire image, and the computing performance utilization is low; 3) the detected clothing must have a fixed pattern for the target detection model to recognize (for example, short-sleeved shirts of various colors, styles and patterns are not suitable for recognition as the same category in target detection). Therefore, the current use of target detection models to directly recognize flight attendant clothing is limited by human posture and clothing category, resulting in low recognition accuracy. Summary of the Invention

[0004] The purpose of the present invention is to provide a method and device for recognizing flight attendant clothing based on intelligent multi-label classification, so as to solve the problem of low recognition accuracy caused by directly recognizing flight attendant clothing using a target detection model.

[0005] To solve the above technical problems, the present invention provides a flight attendant clothing recognition method based on intelligent multi-label classification, which includes:

[0006] 1) Obtain the image of the flight attendant to be identified;

[0007] 2) using the coordinates of key human skeleton points in the image of the crew member to be identified, determining corresponding skeleton point features, and fusing the skeleton point features with image features to obtain fused features, wherein the image features are obtained by extracting features from the image of the crew member to be identified using a feature extraction module;

[0008] 3) Use the intelligent multi-label classification model to process the fused features to identify the flight attendant's clothing.

[0009] Furthermore, the image of the flight attendant to be identified obtained in step 1) is a successfully matched image, wherein successful matching means that the coordinates of the key human skeleton points in the image meet the coordinate requirements of the key human skeleton points in the set template.

[0010] Furthermore, whether the coordinates of the key human skeleton points in the image meet the requirements of the coordinates of the key human skeleton points in the set template is obtained by calculating the similarity between the coordinates of the key human skeleton points in the image and the coordinates of the key human skeleton points in the set template.

[0011] Furthermore, the coordinates of key skeleton points of the human body in the image are extracted by a skeleton point extraction model, and the skeleton point extraction model adopts a YOLO-pose model or a Faster R-CNN model.

[0012] Furthermore, the multi-label classification model adopts the ViT model.

[0013] Furthermore, the feature extraction module is a hidden layer in the ViT model.

[0014] Furthermore, the skeleton point features are obtained by mapping the coordinates of key skeleton points of the human body using a fully connected layer.

[0015] Furthermore, the method also includes reducing the frequency of detecting the coordinates of key human skeleton points in the image when multiple consecutive frames are not successfully matched.

[0016] Furthermore, the labels of the multi-label classification model include two categories, one category representing clothing and the other category representing accessories. The loss function used in training the multi-label classification model includes loss functions representing the two categories of labels.

[0017] The present invention also provides a flight attendant clothing recognition device based on intelligent multi-label classification, comprising a processor, wherein the processor is configured to execute computer program instructions to implement the above-mentioned flight attendant clothing recognition method based on intelligent multi-label classification.

[0018] The present invention has the following beneficial effects: As an improved invention, the present invention takes into account the correlation between clothing and body parts. When using flight attendant images for clothing recognition, the present invention fuses features corresponding to the coordinates of key skeletal points in the flight attendant images with the flight attendant image features to obtain fused features. The fused features are then processed using a multi-label classification model to achieve recognition of flight attendant clothing. During recognition, the present invention improves the accuracy of flight attendant clothing recognition by compensating for the coordinates of key skeletal points. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of the flight attendant clothing recognition method based on intelligent multi-label classification of the present invention;

[0020] Figure 2 Schematic diagram of the fusion of skeleton point features and image features in the present invention;

[0021] Figure 3 This is a schematic diagram of the Query2Label multi-label classification header structure used in the present invention. DETAILED DESCRIPTION

[0022] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0023] The present invention takes into account the correlation between clothing and body parts, integrates the coordinates of key skeleton points in the flight attendant image with the flight attendant image features, and thus improves the recognition accuracy.

[0024] Implementation of flight attendant clothing recognition method based on intelligent multi-label classification

[0025] The present invention first uses the coordinates of the key human skeleton points in the crew image to be identified to determine the corresponding skeleton point features, and fuses the skeleton point features with the image features to obtain fusion features, wherein the image features are obtained by extracting the features of the crew image to be identified using the feature extraction module; then the fusion features are processed using a multi-label classification model to identify the crew clothing. The implementation process is as follows: Figure 1 As shown, the following is a detailed description.

[0026] 1. Use skeleton point matching to determine whether to perform clothing recognition on the flight attendant image.

[0027] First, the image of the flight attendant is captured using an image acquisition device. Then, the key skeleton points of the flight attendant image are extracted. The extracted key skeleton points are then matched with the coordinates of the key skeleton points in the predefined template. If the match is successful, it means that the currently captured flight attendant image meets the posture requirements in the predefined template, and subsequent clothing recognition can be performed.

[0028] Specifically, the present invention uses the YOLO-pose model to extract key bone points from the flight attendant image, and inputs the flight attendant image into the YOLO-pose model. The YOLO-pose model detects the human body boundary box and extracts multiple key bone point coordinates. In this embodiment, 17 key bone point coordinates are used. The predefined template refers to a predefined standard posture template, such as an upright posture, arms hanging down, front facing, etc. The key bone points in the predefined template refer to the key bone point coordinates under each set posture. As other embodiments, in addition to using the YOLO-pose model, other models can also be used to extract key bone point coordinates, such as Faster R-CNN, other YOLO series models, etc.

[0029] When matching, the present invention performs matching by calculating the similarity between the coordinates of the key skeleton points extracted from the flight attendant image and the coordinates of the key skeleton points in the predefined template, wherein the similarity calculation formula is:

[0030]

[0031] Among them, p i is the coordinate of the i-th key bone point detected, is the coordinate of the i-th key bone point in the predefined template, and σ is the tolerance parameter.

[0032] If the calculated similarity exceeds threshold T, a successful match is determined, and the flight attendant's posture is deemed to conform to the predefined standard posture template. This triggers subsequent clothing recognition of the flight attendant's image. By matching key skeletal points within the predefined template, flight attendant images that do not conform to the standard posture can be filtered out, preventing accessories from being obscured and undetectable due to non-standard posture, thereby improving the accuracy of subsequent clothing recognition. The present invention triggers subsequent recognition by matching key skeletal point coordinates, reducing unnecessary and invalid detection.

[0033] 2. Perform clothing recognition on the successfully matched flight attendant images.

[0034] The present invention takes into account the correlation between clothing and body parts. When using the screened flight attendant images to perform clothing recognition, the present invention integrates the coordinate features of key human skeleton points and the flight attendant image features. That is, the coordinates of key human skeleton points in the flight attendant image to be identified are obtained to determine the corresponding skeleton point features, and the skeleton point features are fused with the image features to obtain fused features. The fused features are processed using a multi-label classification model to identify the flight attendant clothing.

[0035] Specifically, if Figure 2 As shown, the present invention first obtains the coordinates of the key skeleton points corresponding to the filtered flight attendant image. In this embodiment, the coordinates of the key skeleton points corresponding to the flight attendant image obtained in step 1 can be directly used, and then the skeleton point coordinates are normalized to [0,1][0,1] and spliced into a vector (x, y coordinates of 17 points), and then mapped to image features extracted from the flight attendant image and feature vectors of the same dimension through a fully connected layer, so that the features corresponding to the coordinates of the key skeleton points can be obtained, namely, the key skeleton point features. Since the intelligent multi-label classification model in this specific embodiment can adopt a deep learning model that can realize multi-label classification, such as the ViT model, Swin-Transformer, VGG series of convolutional neural networks, ResNet series of convolutional neural networks, etc. Therefore, the present invention inputs the screened flight attendant image into the intelligent multi-label classification model, uses the features output by the hidden layer in the intelligent multi-label classification model as the image features extracted from the flight attendant image, and fuses the image features with the key skeleton point features to obtain fused features. The fused features here are obtained by adding the key point skeleton features and the image features (i.e., ViT's [CLS] token).

[0036] As a preferred embodiment, the intelligent multi-label classification model is a ViT model. The ViT model first divides the input image into image blocks of fixed size. These image blocks are flattened and mapped into vectors of fixed length through linear transformation. These vector sequences are then concatenated to a learnable classification tag ([CLS]token). The learnable position encoding of the corresponding position is then added and input into the Transformer Encoder. Finally, the features corresponding to the [CLS]token are extracted and input into the classifier for learning. The features output by the hidden layer in the ViT model are used as the image features extracted from the flight attendant image. When using other models, the data with 768 dimensions directly output can be regarded as the last token.

[0037] ViT divides the image into sequence blocks and processes them through the self-attention mechanism. The structure includes image segmentation, position encoding, Transformer encoder, but does not include MLP head. The original ViT model uses Query2Label to replace the MLP head in order to allow the model to output multiple classification results. Query2Label is not bound to the ViT model and can be replaced by other backbone networks. The purpose of the ViT model is to extract image features, and it is hoped that the features contain information about the type of clothing. Image segmentation and position encoding are to process the image to adapt to the input requirements of the transformer structure, and the transformer encoder is used to extract image features. For the present invention, the multi-label here refers to two labels, representing clothing and accessories respectively. Each label (such as "short sleeves" and "chest badge") is regarded as a learnable query vector (Query), and the dependency relationship between labels is modeled through the interaction between the Transformer Encoder and the fusion feature. For asymmetric feature interaction, image features are used as Key / Value, and label queries are used as queries. The correlation between labels and image areas is captured through cross-attention. The Transformer decoder contains a self-attention mechanism and a cross-attention mechanism. The self-attention mechanism of the Transformer decoder explicitly learns the co-occurrence or mutual exclusion relationship between labels (such as the negative correlation between "tie" and "short sleeve"); the cross-attention mechanism enables each label to dynamically focus on the relevant area in the image (such as the "chest badge" query focuses on the chest, and the "short sleeve" query focuses on the arm). At the same time, visualizing the attention weight can explain the basis of the model's decision (such as Figure 3 shown).

[0038] Data annotation format:

[0039]

[0040] The Transformer decoder embeds the label into the fused features and feeds it into Query2Label to obtain the classification result, which is then used as the recognition result. Query2Label is the type of the entire detection head and represents the name of a structure. The Transformer decoder is part of the Query2Label structure.

[0041] When training the above multi-label classification model, the two labels of clothing and accessories are included, and a multi-task weighted loss function is used. The loss function is:

[0042] L=α·Lcloth+β·Laccessory

[0043] α and β are weights that can be set based on the importance of the task. Lcloth is the loss function for clothing, and Laccessory is the loss function for accessories. BCE With LogitsLoss (BCE stands for binary cross entropy loss) is implemented by adding a Sigmoid layer before BCE. It is a function in the PyTorch library.

[0044] During the training process, Gaussian noise can be added to the coordinates of the skeleton points to improve the robustness of the model to posture errors.

[0045] The ViT model uses knowledge distillation technology and is compressed into MobileViT, which can reduce the number of parameters by 70% while maintaining 95% accuracy.

[0046] The present invention can be applied to user cooperation detection scenarios such as security monitoring and intelligent attendance. For example, it can be applied to a train attendance machine. When clocking in and out, the train attendance machine collects attendant images, and checks the clothing of train attendants, drivers, and other attendants based on the attendant images. The specific process is as follows: extract the coordinates of the key bone points in the attendant image, and judge whether the current attendant image meets the corresponding standard posture in the preset template based on the extracted key bone point coordinates. If not, it means that the attendant posture in the attendant image is not standard. If the recognition is performed directly, the accuracy is not high and the recognition result is unreliable. Therefore, the attendant image is no longer subject to subsequent recognition, avoiding the problem of low recognition accuracy caused by the attendant posture being non-standard. The subsequent recognition mechanism is triggered only when the attendant posture standard is met. After the recognition mechanism is triggered, the features corresponding to the key bone point coordinates in the attendant image are fused with the attendant image features to obtain fused features, so as to realize the association between clothing and attendant posture. The fused features are then classified using a multi-label classification model to determine the attendant's accessory information. If multiple frames (e.g., three frames) are not matched successfully, it means there is a problem with the current photo. In this case, the detection frequency of YOLOpose is reduced to save energy. When there are multiple pairs of flight attendant images in a frame, the individual with the highest matching degree is processed first.

[0047] Implementation method of flight attendant clothing recognition device based on intelligent multi-label classification

[0048] The multi-label identification device for crew clothing of the present invention includes a processor, wherein the processor is used to execute computer program instructions to implement the above-mentioned intelligent multi-label classification crew clothing identification method. The specific implementation process of this method has been described in detail in the method implementation method and will not be repeated here.

[0049] The present invention reduces invalid detections by 80% through a template trigger mechanism. Through simulation tests on a public dataset (DeepFashion2), the multi-label F1 score reaches 92.3%, which is 11.2% higher than the pure image classification model. In occlusion scenarios (such as when the arm is blocked), the accuracy drop is reduced by 40% through skeletal point coordinate compensation.

Claims

1. A flight attendant clothing recognition method based on intelligent multi-label classification, characterized by: The identification method includes: 1) Obtain the image of the flight attendant to be identified; 2) using the coordinates of key human skeleton points in the image of the crew member to be identified, determining corresponding skeleton point features, and fusing the skeleton point features with image features to obtain fused features, wherein the image features are obtained by extracting features from the image of the crew member to be identified using a feature extraction module; 3) Use the intelligent multi-label classification model to process the fused features to identify the flight attendant's clothing.

2. The method for identifying flight attendant clothing based on intelligent multi-label classification according to claim 1 is characterized in that: The image of the flight attendant to be identified obtained in step 1) is a successfully matched image, wherein successful matching means that the coordinates of the key human skeleton points in the image meet the requirements of the key human skeleton points in the set template.

3. The method for identifying flight attendant clothing based on intelligent multi-label classification according to claim 2 is characterized in that: The coordinates of the key human skeleton points in the image satisfy the requirements of the key human skeleton points in the set template by calculating the similarity between the coordinates of the key human skeleton points in the image and the coordinates of the key human skeleton points in the set template.

4. The method for identifying flight attendant clothing based on intelligent multi-label classification according to claim 3 is characterized in that: The coordinates of the key skeleton points of the human body in the image are extracted by a skeleton point extraction model, and the skeleton point extraction model adopts a YOLO-pose model or a Faster R-CNN model.

5. The method for identifying flight attendant clothing based on intelligent multi-label classification according to any one of claims 1 to 4, characterized in that: The multi-label classification model adopts the ViT model.

6. The method for identifying flight attendant clothing based on intelligent multi-label classification according to claim 5 is characterized in that: The feature extraction module is the hidden layer in the ViT model.

7. The method for identifying flight attendant clothing based on intelligent multi-label classification according to claim 2 is characterized in that: The skeleton point features are obtained by mapping the coordinates of key skeleton points of the human body using a fully connected layer.

8. The method for identifying flight attendant clothing based on intelligent multi-label classification according to claim 2 is characterized in that: The method also includes reducing the frequency of detecting the coordinates of key human skeleton points in the image when multiple consecutive frames are not successfully matched.

9. The method for identifying flight attendant clothing based on intelligent multi-label classification according to claim 5 is characterized in that: The labels of the multi-label classification model include two categories, one category representing clothing and the other category representing accessories. The loss function used in the training of the multi-label classification model includes loss functions representing the two categories of labels.

10. A flight attendant clothing recognition device based on intelligent multi-label classification, comprising a processor, characterized in that: The processor is configured to execute computer program instructions to implement the flight attendant clothing recognition method based on intelligent multi-label classification according to any one of claims 1 to 9.

Images