A method for identifying people's clothing

Abstract

The present application discloses a method for identifying clothing of people, which relates to the field of image data processing, including: collecting images of people in different scenes, different clothing and multiple postures, and constructing a training set; performing HLS color space conversion on the RGB images of the training set to obtain an image data set; constructing a key point detection model based on the HRNet architecture, where the key points include the head, limbs and torso; constructing a multi-task convolutional neural network that integrates clothing detection tasks and posture analysis tasks; using the constructed key point detection model, extracting the two-dimensional coordinate information of the key points as a feature vector representing the human posture; inputting the extracted feature vector into the constructed multi-task convolutional neural network to perform standardized detection of the clothing of people. In view of the problem of low accuracy of clothing detection of people existing in the prior art, the present application improves the detection accuracy through key point detection and multi-task learning.

Description

Technical Field

[0001] The present application relates to the field of image data processing, and in particular to a method for identifying a person's clothing. Background Art

[0002] Person clothing detection is a key technology with broad application prospects in today's society. With the continuous development of intelligent monitoring systems, face recognition technology, and virtual fitting rooms, the demand for accurate and efficient detection of personnel clothing is growing. However, traditional personnel clothing detection methods often have the problem of low accuracy when processing images with a variety of clothing, different postures, and complex backgrounds. This is mainly because the feature extraction and classification techniques used by traditional methods cannot effectively capture the key features of personnel clothing in the image, such as posture, color, etc., resulting in the accuracy and robustness of the detection results not meeting actual needs.

[0003] The problems of traditional methods for detecting clothing mainly focus on two aspects: feature extraction and classification. First, in terms of feature extraction, traditional methods often use manually designed feature extractors, such as Haar features, HOG features, etc. These feature extractors often fail to fully utilize key information in the image, such as posture, color, etc., resulting in insufficient extracted features, which affects the accuracy of clothing detection. Second, in terms of classification, traditional methods often use simple classifiers, such as SVM, KNN, etc. These classifiers often fail to effectively model and classify different clothing samples when processing images in complex scenes, resulting in low accuracy of classification results.

[0004] In the related art, for example, Chinese patent document CN110826610A provides a method for intelligently detecting whether a person is dressed in a standard manner. First, an image to be identified containing a human body posture is obtained, and then a target dressing area in the image to be identified is determined, and an image of the target dressing area is obtained. Finally, the image of the target dressing area is input into a trained classification network model to obtain a model output result on whether the dress is standard. This method can realize intelligent detection of whether a person is dressed in a standard manner, and can conveniently and quickly check whether the person's dress complies with the regulations. While quickly realizing whether the appearance and instrument are standardized, it can avoid the hidden danger of omissions when the person self-checks or checks each other, and ensure that the person appears with a standard appearance and instrument. However, if the algorithm for determining the target dressing area in this scheme is not accurate enough, it may cause the extracted target area to be incomplete or contain irrelevant parts, thereby affecting the subsequent judgment of the standardization of the dress. Therefore, the detection accuracy of this scheme needs to be further improved. Summary of the invention

[0005] In view of the problem of low accuracy in detecting people's clothing in the prior art, the present application provides a method for identifying people's clothing, which improves the detection accuracy through key point detection and multi-task learning.

[0006] The purpose of this application is achieved through the following technical solutions.

[0007] The embodiment of this specification provides a method for identifying people's clothing, including: collecting images of people in different scenes, different clothing and multiple postures to construct a training set; performing HLS color space conversion on the RGB images of the training set to obtain an image data set; constructing a key point detection model based on the HRNet architecture, where the key points include the head, limbs and torso; constructing a multi-task convolutional neural network that integrates clothing detection tasks and posture analysis tasks; using the constructed key point detection model, extracting two-dimensional coordinate information of the key points as a feature vector representing the human posture; and inputting the extracted feature vector into the constructed multi-task convolutional neural network to perform standardized detection of people's clothing.

[0008] Among them, multiple postures refer to different body postures displayed in the collected personnel images, such as standing, sitting, raising hands, bending and other action postures. In this application, the style diversity of the training set is expanded so that the model can see more abundant posture changes during the training process, and the adaptability of the model to different postures is improved. Sample data of various postures are provided for subsequent posture analysis tasks, and a posture analysis model that can handle various posture changes is trained. Different postures will cause changes in the position and occlusion of human limbs. Increasing posture diversity can improve the robustness of the model to such changes, thereby improving the accuracy of clothing detection. Images with different postures provide rich human key point annotation data for the key point detection model, which can more fully train the key point detection model to detect each key point. The purpose of collecting images with multiple postures is to increase the diversity of the training set, improve the adaptability of the model to posture changes, and provide richer and more comprehensive training data for key point detection, posture analysis, and final clothing detection tasks.

[0009] Among them, RGB image: is a color model based on the three primary colors of red (R), green (G) and blue (B). The color information contained in each color channel of RGB can be represented by 256 levels of grayscale (0-255). Combining three channels can represent various colors. HLS color space: is a model that describes color based on the three elements of hue, brightness and saturation. Among them, hue represents the basic type of color, brightness represents the brightness of the color, and saturation represents the purity of the color. In this application, the color information contained in the RGB image mainly reflects the absolute intensity of the three elements of color. The HLS color space is based on visual perception and can better represent the visual characteristics of color. Converting RGB images to HLS space can increase the amount of information and sample style diversity of image data sets, which is conducive to training more discriminative models. HLS space separates the three elements of color, which helps the model learn the characteristics of different colors and textures and improves the ability to recognize clothing details. It increases the sensitivity of the model to changes in hue and brightness, and improves the robustness of the model in processing images under different lighting conditions. Converting to HLS space is also beneficial for subsequent data augmentation using GAN to improve the realism of color conversion.

[0010] Among them, HRNet architecture: a high-resolution representation network structure for human posture estimation. Its characteristic is to maintain high-resolution representation while fusing feature information of different resolutions. Key point: a set of joint points used to describe human posture, such as coordinate points of the head, limbs, torso and other parts. Human posture can be analyzed by detecting key points. Head: the head and neck area of ​​the human body. Limbs: the two arms and two legs of the human body. Trunk: the main trunk of the human body, including the trunk and pelvic abdomen. In this application, the key point detection model is constructed using the HRNet architecture, which can simultaneously utilize high-resolution and low-resolution features to improve the positioning accuracy of key points. Detecting the key points of the head, limbs and torso can represent the position information of the main parts of the human body, which is the basis for analyzing human posture. Accurately detecting the key points of various parts of the human body provides a basis for the subsequent construction of human posture features, and also provides position priors for clothing detection. The HRNet model can efficiently handle various posture changes, improve the robustness of key point detection, and enhance the adaptability of the model to complex postures and occlusions. Detecting key points in different parts can also help analyze information such as the direction and orientation of the human body and improve the accuracy of posture representation.

[0011] Among them, the clothing detection task: identify the clothing category worn by the human body from the image, such as tops, pants, skirts and other clothing types. Posture analysis task: analyze the posture information of the human body in the image, usually including the positioning of key points of the body, posture classification, etc. In this application, a multi-task network integrating clothing detection and posture analysis is constructed, which can share the underlying feature extraction layer and improve the efficiency of calculation and expression. Posture analysis can provide human body parts and occlusion information to help the clothing detection module locate the clothing area. The clothing detection results in turn constrain and calibrate the structured feature learning of posture analysis. The mutual transmission of information between the two tasks helps the model learn more abstract and semantic feature representations and improve generalization capabilities. The multi-task framework trains a more robust model, alleviates overfitting, and optimizes the performance of clothing detection and posture analysis in an end-to-end manner. Integrating two complementary visual analysis tasks can obtain more powerful feature expressions than training them separately, thereby improving the accuracy of clothing recognition. The multi-task network integrates posture analysis and clothing detection, so that the two tasks can fuse complementary semantic information, improve the generalization performance of the model, and enhance the ability to understand and express complex images.

[0012] Among them, the feature vector of human posture is a set of numbers representing the posture information of the human body in the image. Each number in the feature vector represents the two-dimensional coordinate position of the key point of the human body (such as head, hand, foot, etc.). The coordinates of all key points in the same human image are represented as a vector, that is, a feature vector representing the posture is formed. The feature vector intuitively reflects the relative position information of various parts of the human body in the image. In this application, the human body image is input into the constructed HRNet key point detection model, and the coordinate position of each key point is output. The key point coordinates are sorted into a vector, that is, the feature vector corresponding to the human posture is obtained. The feature vector retains the complete human posture information and can be directly input into the subsequent multi-task network. Compared with directly inputting the image, the feature vector represents the posture more intuitively and greatly reduces the dimension to improve the computational efficiency. In the multi-task network, the shared module can learn the expression of the feature vector and serve the two tasks of posture analysis and clothing detection at the same time. Based on the expression of the posture feature vector, it helps the model learn the knowledge of the associated posture and clothing and improve the detection robustness. Finally, the posture feature is used to enhance the clothing detection, and the standardized feature expression and detection are performed.

[0013] Specifically, dress standardization detection is performed, a unified dress detection framework is constructed, and feature vectors of different postures and perspectives are mapped to a unified posture space. At the input end of the feature vector, a posture normalization branch is set to convert the feature vector of any posture to a standard frontal posture through a spatial transformation network. A normalization module based on a graph attention mechanism is designed to learn the association and transformation of feature vectors on the graph structure to obtain a normalized feature expression. In the dress detection branch of the multi-task network, a normalization detection head is set to input the normalized feature vector to perform normalized detection without posture deviation. Feature regularization based on GAN is added during the training process so that the features meet good distribution consistency in the latent space. At the same time, posture labels and dress labels are predicted as multi-task supervision to guide the feature vector to learn normalized expressions. Self-supervised learning is performed using unlabeled posture data to predict the posture of the input image and reduce the need for annotation. A feature vector decoding branch is set to reconstruct the input image, increase the structural constraints of the features, and optimize the expression. Finally, the normalized detection result is output to eliminate the influence of posture changes and perspective changes on the detection performance and improve the robustness.

[0014] Furthermore, the constructed RGB format training set images are converted to HLS color space, including: constructing a generative adversarial network including a color space conversion generation network and a true-false discrimination network; inputting the RGB format training set images into the color space conversion generation network, mapping the RGB images to HLS images, and outputting the generated HLS images; inputting the generated HLS images and the corresponding real HLS sample images into the true-false discrimination network for true-false discrimination, and outputting the true-false discrimination results of each HLS image; according to the true-false discrimination results, calculating the loss functions of the generation network and the discrimination network, and updating the network parameters through back propagation; repeating the above steps until the loss function is less than the threshold, and outputting the trained color space conversion generation network; taking the collected RGB format training set as input, using the trained color space conversion generation network to generate the corresponding HLS format image data set.

[0015] Among them, the color space conversion generation network: consists of an encoder and a decoder to realize the conversion of RGB images to the target HLS color space. The authenticity discrimination network: contains convolution and fully connected layers, which can distinguish the authenticity of the generated HLS images. Generative adversarial network: the generative network and the discriminative network confront and optimize each other, and improve the performance of the generative network during the confrontation process. In this application, a generative adversarial framework including the above-mentioned generative network and the discriminative network is constructed. The generative network completes the conversion from RGB to HLS and outputs the converted HLS image. The discriminative network judges the authenticity of the converted image and provides adversarial loss. Improve the realism of the generative network conversion during the confrontation process. The conversion result is made close to the real HLS distribution through parameter updating. The adversarial framework guides the learning of the mapping relationship of color conversion. The HLS images generated by GAN are of higher quality, which is conducive to subsequent clothing recognition. The generative adversarial mechanism enhances the model's ability to process color information. Improve the realism of the generated HLS images and the robustness of the model.

[0016] The color space conversion generation network includes an encoder and a decoder to map the RGB image to the HLS image. The encoder uses a convolutional layer to gradually extract the color and texture features of the RGB image and compress it into a higher-level feature expression. The encoder uses a network structure such as VGG or ResNet, and sets multiple convolution modules to extract multi-scale features. The spatial dimension is reduced by downsampling operations, the number of channels is expanded, and the compressed encoder features are output. Based on the features output by the encoder, the decoder restores the spatial dimension through upsampling and transposed convolution layers. The decoder sets up a multi-level subnetwork to gradually restore the high-resolution details of the color space conversion. While the transposed convolution layer performs upsampling, it performs feature mapping and gradually restores the HLS image. Skip connections are set to pass the encoder features to the corresponding layers of the decoder to improve information flow. The last decoder subnetwork outputs the final HLS image to achieve the conversion from RGB to HLS. During the training process, the error between the generated HLS image and the corresponding Ground Truth is minimized. The encoder encodes color and texture features, and the decoder restores the HLS image based on the features. The two work together to achieve conversion.

[0017] The authenticity discrimination network discriminates the authenticity of the generated HLS images. The input layer receives the generated HLS images and the real HLS sample images. The convolution layer uses different convolution kernels to extract the color and texture features of the HLS images. The pooling layer performs downsampling to reduce the feature dimension and parameter quantity. Set multiple groups of convolution pooling layers to extract feature information at different levels. The convolution pooling features are compressed into one-dimensional features through the Flatten layer. The fully connected layer performs nonlinear transformation on the Flatten features to learn the feature representation of authenticity discrimination. The last fully connected layer outputs the authenticity discrimination probability of each sample. Through the Softmax classification layer, it is converted into the probability distribution of the authenticity category. During the training process, the labels of the real HLS samples are used to optimize the network parameters. Minimize the cross entropy loss between the predicted categories and labels of the real samples and the generated samples. Finally, learn to judge whether the generated HLS images are realistic and guide the generation model to produce better HLS images.

[0018] Furthermore, the color space conversion generation network includes an encoder and a decoder; the convolution layer in the encoder extracts features from the RGB image; the decoder maps the features extracted by the encoder to an HLS image through upsampling and transposed convolution.

[0019] Among them, color features: reflect the color information of pixels in the image, such as color channel histogram, main color tone, etc. Texture features: reflect the texture structure information of the image, such as roughness, contrast, direction, etc. In this application, the convolution layer of the encoder can extract the color and texture features of the input RGB image. Use convolution kernels of different sizes to extract local features of color and texture. Color features reflect the color composition of the image. Texture features represent the texture of clothing fabrics. The two types of features are combined into a feature expression that is sensitive to both clothing color and material. These features contain the color and texture information of the original image and are the basis for conversion to the HLS space. The features output by the encoder support the decoder to reconstruct an HLS image containing color and texture details. Finally, a high-quality HLS image that retains the details of the original image can be generated. It is beneficial for the model to learn the representation of color and texture and enhance the modeling ability of clothing features.

[0020] Among them, upsampling: enlarging the image or feature map in the spatial dimension to increase the resolution. Transposed convolution layer: a convolution operation that can achieve upsampling and generate a feature map with a larger resolution. In this application, the decoder uses an upsampling operation to increase the spatial resolution of the color and texture feature maps. The transposed convolution layer not only performs upsampling, but also maps the low-resolution features output by the encoder to the high-resolution space. The spatial correspondence between color and texture features is learned through the convolution kernel. The transposed convolution layer gradually restores the details and resolution of the generated HLS image. Upsampling restores spatial information, and transposed convolution performs feature mapping. Reproduce the color and texture feature maps compressed by the encoder and reconstruct a high-resolution HLS image. Improve the quality of the generated HLS image so that it contains richer color and texture information. It is beneficial for the model to generate realistic HLS samples and enhance the modeling ability of color and texture.

[0021] Among them, HLS image: an image converted from RGB to HLS color space by generating a network. HLS sample: a sample image in the HLS color space extracted from a real picture. In this application, the HLS image output by the generating network and the real HLS sample are input into the discriminant network together. The HLS image is used by the discriminant network to judge the authenticity of the generated image. The HLS sample provides the distribution information of the real image in the HLS color space. The discriminant network judges the realism of the generated HLS image by comparing the HLS image and the HLS sample. If the generated HLS image can "fool" the discriminant network, it means that its quality is close to the real HLS image. The generating network is continuously optimized so that the HLS image it generates is close to the HLS sample in distribution. The HLS sample serves as supervision of the real distribution to guide the generating network to learn to convert the mapping relationship from RGB to HLS. Finally, a high-quality HLS image is obtained, providing ideal input data for subsequent clothing recognition.

[0022] Specifically, the authenticity discrimination network uses convolutional layers and pooling layers to extract features from HLS images. After the input layer, the convolutional layer is first used, and convolution kernels of different sizes are used to extract multi-scale features from HLS images. Smaller convolution kernels (3x3) are used to extract local features such as color and texture; larger kernels are used to extract global features. By setting different step lengths, the degree of downsampling of feature maps can be controlled. The pooling layer performs downsampling operations on feature maps to reduce the size of feature maps and enhance the generalization ability of features. The main visual feature information of HLS images is gradually extracted using maximum pooling. Multiple convolutional pooling modules are set to gradually expand the receptive field and learn feature representations at different levels. The subsequent fully connected layer learns a discriminant model for distinguishing the authenticity of HLS images based on these local and global features. The network structure design refers to the organization of image classification networks such as VGG and ResNet. The loss function uses a combination of deception loss and real sample loss to optimize the effect of feature extraction. Finally, a feature extractor with strong discrimination ability for HLS images is obtained, which guides the generation network to produce more realistic HLS images.

[0023] Specifically, the extracted features are input into the fully connected layer of the authenticity discrimination network for authenticity discrimination, and the features extracted by the convolution pooling layer are compressed into a one-dimensional vector by Flatten. The fully connected layer performs nonlinear transformation on the feature vector and learns more abstract feature representations. A multi-layer fully connected network is set up to gradually extract discriminative features to increase the network's expressive power. Each fully connected layer is followed by a ReLU activation function to introduce nonlinearity and enhance feature learning capabilities. The last fully connected layer outputs binary logits, which represent the non-normalized scores of the authenticity of the input image. The softmax function is applied to logits to obtain the probabilities of the two categories of authenticity. The cross entropy loss function is used to optimize the network parameters so that the probability of real samples is close to 1 and the probability of fake samples is close to 0. The generated HLS images and real HLS samples are input simultaneously during training. The real sample label is 1 and the generated sample label is 0 to guide the network to learn authenticity discrimination. Finally, the fully connected network can give an authenticity prediction for each input HLS image and guide the generated network to improve the conversion quality.

[0024] Specifically, the parameters are calculated and updated through back propagation based on the discrimination results. The output of the discriminant network represents the probability that the input HLS image belongs to a real or generated image. For real HLS samples, their labels are 1, and the expected discriminant network output probability is close to 1. For generated HLS images, their labels are 0, and the expected output probability is close to 0. The cross entropy loss L_real of real samples is defined to make the network prediction probability consistent with the real label. The cross entropy loss L_fake of generated samples is defined to encourage the discriminant network output probability to be close to 0. The goal of the generative network is to "cheat" the discriminant network to minimize L_fake. The goal of the discriminant network is to correctly classify samples to minimize L_real. The parameters of the generative network and the discriminative network are updated separately through back propagation. The generative network learns to generate more realistic HLS images to "cheat" the discriminative network. The discriminative network learns to better identify the defects of the generated images and guide the improvement of the generative network. The two networks optimize each other in the confrontation process to improve the quality of the generated HLS images. The optimization goal is to make the generated images indistinguishable from the real samples in terms of distribution.

[0025] Furthermore, the loss function includes the generative network loss and the discriminative network loss; the generative network loss includes the MSE reconstruction loss and the perceptual loss, and the perceptual loss is obtained by calculating the distance between the generated image and the real image at different feature layers through the pre-trained feature extraction network; the discriminative network loss includes the binary cross entropy loss and the feature center loss, and the feature center loss makes the feature vectors of similar samples extracted by the discriminative network gather together.

[0026] Furthermore, the network loss loss1 is calculated by the following formula:

[0027] Among them, MSE_loss represents the mean square error loss function, which calculates the difference in color values ​​between the generated HLS image and the real HLS image;

[0028] Generated_HLS represents the HLS image output by the generated network; GroundTruth_HLS represents the corresponding ground truth HLS image; λ represents the weight coefficient, which balances the proportion of MSE loss and perceptual loss; Perceptual_loss represents the perceptual loss function based on feature extraction, which calculates the visual feature difference between the generated HLS image and the real HLS image. Perceptual_loss extracts the feature maps of the two images through the feature network and calculates the similarity between the feature maps.

[0029] Among them, the ground truth HLS image, an image of the HLS color space collected from a real scene, has a real hue, brightness, and saturation distribution. In this scheme, the ground truth HLS image is the supervision target of the color space conversion. The pixel-level error is calculated between the HLS image output by the generative network and the corresponding ground truth HLS image. Minimizing the error between the two is used as the supervision signal of the generative network to optimize the parameters. The ground truth HLS image reflects the statistical distribution information of the color of the real scene. By aligning with it, the generated HLS image can maintain the color style and realism of the scene. The ground truth HLS image provides high-quality supervision for the generation process, making the result more realistic. Compared with artificially constructed HLS samples, the ground truth samples contain more natural color changes. This supervision can generate high-quality HLS images with unified style and natural colors.

[0030] Among them, the perceptual loss function: evaluates the difference in visual perception by comparing the feature extraction results of the two images. Feature extraction: Use a pre-trained network to extract high-level feature representations of the image to reflect the visual content. In this scheme, the perceptual loss function is used to calculate the feature differences between the generated HLS image and the real HLS image. High-dimensional feature vectors are extracted from the two images, and the Euclidean distance between them is measured as the perceptual loss. The feature extraction network can use pre-trained networks such as VGG and ResNet. Perceptual loss reflects the similarity of the visual effects of the two images, which is more in line with human eye perception than directly comparing pixel values. Minimizing the perceptual loss can make the generated HLS image visually closer to the real image. Incorporating the perceptual loss function can make the generated image more realistic in color and style. Finally, a more realistic and natural HLS image is generated for the human eye, improving model performance.

[0031] Among them, MSE_loss is the mean square error loss, and the difference between the two at the pixel level is calculated: The convolutional layer output of the pre-trained network is used as the feature map, which contains the semantic information of the image. Assume that the feature map of the generated HLS image is Fg, and the real HLS image is Fr, with a shape of (C, H, W). Traverse each channel c of the feature map and calculate the mean square error between the generated image and the real image features of the channel: Sum the mean square error of all channels to get the mean square error of the feature map of this layer: Repeat the above steps to calculate the average square error L between the multi-layer feature maps of the network l . Take the weighted average of the errors of each layer as the final perceptual loss: Lperceptual=Σw l ×L l , w l is the weight coefficient of the corresponding layer. Usually the middle layer has richer semantic information and is given a higher weight. lIt can be determined by the validation set. Minimize Lperceptual and reduce the difference between feature maps, so that the generated HLS image is semantically close to the real image. Optimize the generated network and output more realistic HLS results.

[0032] The discriminant network loss loss2 is calculated by the following formula:

[0033] loss2=BCE_loss(Discriminator(Generated_HLS)+

[0034] mu×Center_loss(Feature_vectors))

[0035] Among them, BCE_loss represents the binary cross entropy loss function of the discriminant network, which is used to determine the authenticity of the generated HLS image; Discriminator(Generated_HLS) represents the discriminant network; Generated_HLS represents the HLS image output by the generated network; mu represents the weight coefficient of Center_loss; Center_loss represents the center loss function of the feature vector, which is used to make the feature vectors of samples of the same category close to the class center; Feature_vectors represents the feature vector of the fully connected layer in the discriminant network.

[0036] The calculation scheme of the binary cross entropy loss function BCE_loss of the discriminant network is as follows: the discriminant network inputs the generated HLS image and the real HLS image, and outputs the probability of each image being real or generated. For the input image i, the real category is y i (1 means real, 0 means generated). The predicted probability that the discriminant network outputs image i as the real category is p i . Calculate the binary cross entropy loss for each image:

[0037] BCE_loss i =-(y i ×log(p i )+(1-y i )×log(1-p i )), the average binary cross entropy loss of all images is used as the final discriminant Loss: Minimizing BCE_loss aligns the discriminant network output probability with the true category. For real HLS images, p i Close to 1; for generating HLS images, make p i Close to 0. Optimize the parameters of the discriminant network and correctly classify the generated images and real images. This guides the generative network to produce more realistic HLS results. Finally, a discriminant model that can correctly distinguish the authenticity of HLS is obtained.

[0038] The calculation scheme of the discriminator network Discriminator (Generated_HLS), the input layer receives the generated HLS image or the real HLS image. The convolution layer is used to extract the features of the input image, which contains multiple convolution blocks, each of which consists of convolution, batch normalization, and activation functions. Use different convolution kernel sizes to extract multi-scale features and enhance feature expression. Set the step size to control the degree of downsampling of the feature map. The pooling layer further downsamples the feature map to reduce the number of parameters. The flatten layer is connected to compress the feature vector. The fully connected layer performs nonlinear transformation and learns discriminative features. The output layer is a single node activated by sigmoid, which indicates the probability that the input image is real. The binary cross entropy is used as the loss function. Back propagation updates the network parameters. The real sample label is 1, and the target probability is close to 1. The generated sample label is 0, and the target probability is close to 0. The training goal is to correctly classify the generated HLS image and the real sample. Finally, a Discriminator that can effectively distinguish the authenticity of HLS images is obtained.

[0039] Calculation scheme of the feature vector Feature_vectors of the fully connected layer in the discriminant network: The features extracted by the convolution layer and the pooling layer are compressed into a one-dimensional vector by Flatten. Assume that the length of the compressed feature vector is N. The first fully connected layer contains M nodes, the weight matrix is ​​W1(M*N), and the bias is b1(M). Calculate the feature vector of the first fully connected layer: Feature_vectors1=W1×Flattern_feature+b1, and apply the activation function (such as ReLU) to Feature_vectors1 for nonlinear transformation. The second fully connected layer is similar, and Feature_vectors2 is calculated based on Feature_vectors1 of the previous layer. Repeat similar calculations, and extract features with higher abstraction through multiple layers of full connection. The last layer of fully connected features is the final discriminant feature Feature_vectors. Update the weights of the fully connected layer through back propagation to learn the discriminant features. Make Feature_vectors have strong classification ability to discriminate generated images and real images. And the weights of the previous convolution layer can be adjusted in reverse to further improve the feature expression. The fully connected layer gradually extracts abstract discriminative features to complete the evaluation of the quality of the generated HLS image.

[0040] Furthermore, the calculation formula of the perceptual loss function Perceptual_loss is:

[0041]

[0042] in, represents the i-th layer feature map of the image feature extraction network; Generated_HLS represents the HLS image output by the generated network; GroundTruth_HLS represents the corresponding ground truth HLS image; w i Represents the weight coefficient of the feature map of the i-th layer; Distance represents the distance measurement function between feature maps, such as MSE, MAE, etc.;

[0043] in, Represents the i-th layer feature map of the pre-trained feature extraction network. Generated_HLS and GroundTruth_HLS are passed into the feature network respectively. Extract the feature maps of the two images at the i-th layer and Calculate the distance between two feature maps: The distance metric can be MSE, MAE, etc. Calculate the feature map distance of each layer respectively to get the loss of each layer. Define the weight coefficient w for the i-th layer i , usually the middle layer has a higher weight. The final perceptual loss is the weighted sum of the losses of each layer:

[0044] Minimize Perceptual_loss to make the distance between feature maps of each layer approach 0. This is equivalent to making Generated_HLS approach GroundTruth_HLS in terms of semantic features. Using perceptual loss, the generated HLS image is more realistic in visual effect. This helps to improve the overall quality of image generation. Preferably, the feature extraction network in the perceptual loss function uses a pre-trained VGG network to extract the L1 or L2 distance between the generated HLS image and the real HLS image on the shallow and deep feature maps.

[0045] The calculation formula of the center loss function Center_loss is: Where: x i Represents the feature vector of sample i in the discriminant network feature layer; cy i represents the center of the feature vector of category y to which sample i belongs; Σi(x i -cy i ) 2 Indicates calculating the distance between the sample feature vector and the corresponding category center vector.

[0046] Among them, for sample i, its category label is y i , the feature vector in the discriminant network feature layer is x i . Define category y i The center point of all eigenvectors is cy i . Center point cy i You can use category y iCalculate the eigenvector x of sample i i and the corresponding category center cy i The Euclidean distance of (x i -cy i ). Square the distance and sum it to get the sum of the distances between all samples and the class center. Center_loss is defined as 1 / 2 of the result of the previous step: Where Σi(x i -cy i ) 2 Calculate the sum of the distances between all sample feature vectors and the center of the corresponding category. Minimizing Center_loss aggregates the feature vectors of similar samples, enhancing the feature expression ability of the discriminant network. The optimization goal is to generate sample features as far away from the center of the real sample class as possible. This improves the discriminant network's ability to recognize generated samples. It helps the generative network generate more realistic samples.

[0047] Furthermore, according to the loss function loss1 and the loss function loss2, the loss loss of the generative adversarial network is calculated: the loss function of the generative adversarial network is: loss = ξ × loss1 + loss2, where ξ represents the hyperparameter that controls the proportion of the loss of the generative network and the loss of the discriminative network. Among them, ξ is a hyperparameter that needs to be adjusted during the training process, which is used to control the proportion of the loss loss1 of the generative network and the loss loss2 of the discriminative network. By adjusting the value of ξ, the ability of the generative network to generate high-quality samples and the ability of the discriminative network to distinguish between true and false samples can be balanced. If ξ is set too large, the generative network will be optimized too much, and the discriminative ability will be weak. If ξ is too small, the discriminative network will be optimized too much, reducing the training effect of the generative network. Therefore, the setting of ξ needs to determine the optimal value through the validation set experiment to strike a balance between the generative effect and the discriminative accuracy. ξ can also be adjusted dynamically during the training process, first increasing the training of the generative network and then increasing the proportion of the discriminative network.

[0048] Furthermore, a key point detection model based on the HRNet architecture is constructed, including: taking the obtained HLS format image dataset as input; constructing multiple parallel sub-networks of different resolutions, the sub-networks of different resolutions include a high-resolution sub-network and a low-resolution sub-network, the high-resolution sub-network is used to extract local features, and the low-resolution sub-network is used to extract global features; setting a multi-level residual connection structure in each sub-network, and extracting human features at different levels through convolution kernels of different scales; adding a key point heat map prediction head at the end of each sub-network, upsampling the extracted human features to restore the resolution, and outputting the confidence heat map of each key point; outputting the key point coordinates according to the confidence heat map.

[0049] Among them, local features: features extracted from local areas of the image, reflecting the details and texture information of the image. Global features: features extracted from the entire image range, reflecting the overall structure and shape of the image. In this application, local and global features are extracted respectively through sub-networks of different resolutions: the high-resolution sub-network inputs the original image, and extracts local detail features through a small convolution kernel. The low-resolution sub-network inputs the downsampled image, and the large convolution kernel extracts global structural features. Local features retain the fine texture of the image, and global features provide overall shape information. The features of the two are spliced ​​and input into the discriminant network, and the local and global information are combined to make authenticity judgments. Compared with a single network, features from multiple perspectives can be learned to improve the accuracy of discrimination. The discriminant network fuses features of different granularities, is more sensitive to defects in the generated image, and promotes the improvement of the generation network.

[0050] Among them, the high-resolution sub-network: inputs the original or higher-resolution image, and extracts local detail features through a small convolution kernel. The low-resolution sub-network: inputs the downsampled or lower-resolution image, and extracts global structural features through a large convolution kernel. In this application, multiple sub-networks with different resolutions are established for parallel learning: the high-resolution sub-network retains more original details and extracts local textures, edges and other features. The low-resolution sub-network can see more global information and extract features such as object shape and scene layout. The features of the high- and low-resolution sub-networks are spliced ​​and input into the discriminant network. The discriminant network integrates multi-granular features and is more sensitive to defects in generated samples. Compared with a single network, features can be learned from different perspectives to enhance the expressive power of the discriminant network. The optimized discriminant network promotes the generation network to output higher quality samples.

[0051] Among them, the multi-level residual connection structure: Adding short cut connections between layers in the network, skipping one or more layers. Human features at different levels: Low-level features: local simple features such as lines, edges, and colors. Middle-level features: more local features such as body parts and positions. High-level features: global features such as whole body shape, posture, and movement. In this application, in each sub-network, multi-level residual connections are set to extract human features at different levels. The initial network extracts low-level features, the middle layer extracts middle-level features, and the high-level extracts global high-level features. The residual connection allows the underlying features to be transmitted at high levels to achieve multi-granularity feature fusion. The multi-granularity human features output by the sub-network are input into the discriminator. The discriminator comprehensively judges features at different levels to enhance the accuracy of overall authenticity discrimination. Through residual connections, multi-dimensional human feature extraction and fusion are achieved in each sub-network to improve the final discrimination effect.

[0052] Among them, heat map prediction head: a prediction branch added at the end of the network, which upsamples the input features and outputs a heat map of each key point. Confidence heat map: a heat map that reflects the confidence of the key point position, and the heat value indicates the probability that the pixel contains the corresponding key point. In this application, a heat map prediction head is added to the end of each sub-network. The human features extracted by the sub-network are upsampled to restore the resolution. A confidence heat map is output for each key point. The higher the heat value, the more likely it is the location of the corresponding key point. The heat maps output by the sub-network are spliced ​​as auxiliary supervision signals for adversarial training. It helps to improve the discriminator's ability to learn human structure and posture. Assist the generator to produce more refined and natural human images.

[0053] Specifically, a key point heat map prediction head is added at the end of the sub-network, and the network backbone is gradually downsampled through convolution and pooling to extract human features. The last layer of feature maps is used as the input of heat map prediction. An upsampling module, such as bilinear interpolation or transposed convolution, is set to upsample the feature map to the original image size. A single-channel branch is added to each key point, which contains several layers of convolution. The convolution kernel size is set small, such as 1x1 or 3x3, to retain detail information. Pooling is not used to maintain the resolution of the heat map. The last layer of convolution uses sigmoid activation to output a single-channel heat map. Its value represents the confidence that the corresponding pixel is the key point. Repeat the above steps to predict a confidence heat map for each key point. All heat maps are spliced ​​together as auxiliary supervision information for the network. It can also be used for key point coordinate regression to extract accurate human skeleton structure information.

[0054] Specifically, the key point coordinates are output according to the confidence heat map, which reflects the position distribution probability of each key point. For key point k, its confidence heat map is represented as Hk, with a size of w*h. Find the maximum probability pk on the heat map Hk: pk = max(Hk), and determine the coordinates corresponding to the maximum probability (x k ,y k ):(x k ,y k )=argmax(Hk),(x k ,y k ) is the predicted coordinate of key point k. For each key point k, the position of the maximum probability can be determined as the coordinate prediction based on its confidence heat map Hk. If there are multiple local maximum areas, their geometric centers are calculated as the key point coordinates. The average position of the heat map can also be obtained as the predicted coordinates through average pooling. By connecting all the key point coordinates, a complete human key point skeleton can be output. End-to-end key point detection can be achieved using the coordinates of the confidence heat map.

[0055] Furthermore, a key point heat map prediction head is added at the end of each sub-network, including: setting a heat map prediction head according to the number of key points; in each prediction head, setting multiple groups of upsampling layers to generate a heat map according to the input human body features, and restoring the resolution of the heat map to the same size as the input human body features through upsampling; in the upsampling process, a convolution layer is also set to perform feature extraction and mapping; finally, a group of convolution layers is set to output the confidence heat map of each key point according to the heat map with restored resolution.

[0056] Specifically, the heat map prediction head is set according to the number of key points, and the number of key points of the human body is defined as N, typically 10 to 20 key points. Each key point needs to predict a confidence heat map. Therefore, the heat map prediction head needs to set N branches. Each branch contains: an upsampling module (such as bilinear interpolation), which upsamples the backbone feature map to the original resolution, a convolution layer module, which retains local information and outputs a heat map, a sigmoid activation function, and outputs a confidence value between 0 and 1. 2 to 3 convolution layers can be set in the convolution module, using small convolution kernels such as 1x1 or 3x3. Pooling operations are not used to maintain the spatial resolution of the heat map. N branches share the same upsampled features, and convolution parameters are not shared. Each branch learns the position features of the corresponding key points separately. The final output is a heat map of size NxHxW, where H and W are the input image resolutions. The number of heat maps corresponds to the number of key points in a 1:1 ratio. Preferably, the key point heat map is spliced ​​on the dimension of the input posture feature vector, and the posture key points are fused with the features of the corresponding image area to improve the accuracy of the heat map prediction.

[0057] Specifically, multiple groups of upsampling layers are set in the prediction head to restore the resolution of the heat map, and the network backbone is gradually downsampled, and the resolution of the output feature map is reduced. Assume that the backbone output feature map size is HxW. The input image size is hxw, and the goal is to restore the heat map of the same size. If upsampling is performed directly, the magnification may be too large and the information loss is serious. Therefore, multiple groups of upsampling layers are set to gradually restore the resolution: the first group of upsampling layers upsamples the HxW feature map to h / 2xw / 2, the second group upsamples to h / 4xw / 4, ..., and the last group upsamples to hxw size. Each group of upsampling can use bilinear interpolation or transposed convolution. A small number of convolutional layers can be connected between upsampling groups to enhance features. Through multiple groups of step-by-step upsampling, information loss can be effectively suppressed. Finally, a heat map of size hxw is output, which is the same as the input image resolution. This module can be inserted into any prediction head based on a convolutional network.

[0058] Specifically, a convolution layer is set during the upsampling process and a confidence heat map is finally output. Multiple groups of upsampling modules are set to gradually restore the heat map resolution to the input image size hxw. Between each group of upsampling, 1 or 2 convolution layers are inserted for feature extraction. These convolution layers help learn the feature representation after upsampling and enhance the generation effect of the heat map. The convolution kernel size can be set to 3x3 or 1x1, and the number of convolution layers is set according to the specific task needs. After the last group of upsampling, a group of convolution modules is set to generate the heat map. This group of convolution modules contains 2 to 3 convolution layers. No pooling operation is used to maintain the high resolution of hxw. The last layer of convolution uses a sigmoid activation function to output a single-channel confidence heat map. The sigmoid function maps the feature value to between 0 and 1, indicating the prediction confidence. Repeat the above process for each key point and finally output multiple confidence heat maps.

[0059] Furthermore, the multi-task convolutional neural network adopts a shared convolutional network structure; the shared convolutional network includes convolutional layers and pooling layers; the shared convolutional network is used for feature extraction and expression. Among them, the shared convolutional network structure refers to using the same series of convolutional and pooling layers as the main network for feature extraction, and then performing respective prediction tasks through different head structures. In the present application, a convolutional neural network is constructed as the main network backbone, including convolutional layers and pooling layers. The feature map output by the backbone is simultaneously input to two prediction heads. One head reconstructs the image and the other performs classification. The two heads have their own independent fully connected layer and other structures. But they share the same main network backbone for feature extraction. Shared feature extraction, different tasks promote each other's feature learning. Reduces the number of parameters and the amount of calculation.

[0060] Preferably, the shared convolutional network uses lightweight backbone networks such as MobileNet and ShuffleNet to reduce the number of model parameters and improve the computing speed.

[0061] Furthermore, the extracted posture feature vector is input into the posture analysis subnetwork of the multi-task convolutional neural network; and the obtained HLS image is input into the shared network and clothing detection subnetwork of the multi-task convolutional neural network.

[0062] Specifically, the two subnetworks share the convolutional network backbone for feature extraction. Clothing detection subnetwork: There are several fully connected layers after the backbone, and the last layer is fully connected. The number of neurons is the number of clothing categories N. The softmax activation function is used to obtain the predicted probability of each category, and the output dimension is N. The vector represents the predicted distribution of N types of clothing. Posture analysis subnetwork: There are several fully connected layers after the backbone, and the last layer has no activation function. The number of neurons is the number of key points M. The output dimension is M*2 vector, which represents the two-dimensional coordinates of M key points. The two subnetworks achieve their respective outputs through different fully connected structures of classifiers and regressors. The overall network is trained end-to-end, and the backbone learning features are suitable for both tasks. The outputs of the two subnetworks can be supervised at the same time to achieve multi-task detection of clothing and posture.

[0063] Compared with the prior art, the advantages of this application are:

[0064] The key point detection model of the HRNet architecture can accurately extract the key point information of the human body, including the head, limbs and torso, providing rich spatial information for subsequent clothing detection and posture analysis. Such comprehensive feature learning can effectively improve the ability to understand human posture and clothing;

[0065] HLS color space conversion is used to map RGB images to HLS images to obtain richer color information. This enhancement can improve the diversity and feature expression ability of training data, help the model better understand the color characteristics of different clothing, and improve the accuracy of clothing recognition;

[0066] Generative adversarial networks are used for color space conversion, and generative adversarial training is used to improve the performance of the generative network and the quality of the generated images. This optimization ensures that the generated HLS images are more realistic, which helps improve the accuracy of subsequent clothing detection;

[0067] A multi-task convolutional neural network is used to perform clothing detection and posture analysis tasks simultaneously, making full use of the feature information extracted by the shared convolutional network. This multi-task learning can improve the efficiency and generalization ability of the model, making clothing recognition and posture analysis more accurate and robust;

[0068] Introducing perceptual loss based on feature extraction, extracting visual features of images through feature networks, which helps the generated images to be closer to real images and improves the generation quality of the generation network. This loss mechanism can make the generated HLS images more visually realistic, which helps improve the performance of subsequent tasks;

[0069] Introducing the center loss in the authenticity discrimination network makes the feature vectors of samples of the same category closer to the class center, which helps improve the discrimination network's ability to distinguish different categories. This loss mechanism can improve the robustness and discrimination ability of the generative adversarial network, thereby improving the accuracy of clothing recognition;

[0070] The key point detection model adopts a multi-resolution sub-network structure, combined with multi-level residual connections, which can more comprehensively extract human features at different levels, making key point detection more accurate. This feature extraction mechanism can improve the accuracy and robustness of key point detection, thereby improving the overall performance of clothing recognition;

[0071] The shared convolutional network structure is adopted to extract features through convolutional layers and pooling layers, which effectively reduces the number of parameters and improves computational efficiency. This shared convolutional network can better utilize data information, improve the generalization ability and efficiency of the model, and thus improve the overall performance of clothing recognition;

[0072] By constructing a generative adversarial network, the conversion from RGB to HLS color space is realized, and the model's sensitivity to color and texture is enhanced; the HRNet framework is used to detect human key points and extract posture features; a multi-task convolutional network integrating clothing detection and posture analysis tasks is built, and posture features are used for standardized detection to improve detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0074] Figure 1 is an exemplary flow chart of a method for identifying a person's clothing according to some embodiments of this specification;

[0075] Figure 2 is an exemplary flow chart of performing HLS color space conversion on an RGB image according to some embodiments of this specification;

[0076] Figure 3 is an exemplary flow chart of detecting key points according to some embodiments of this specification;

[0077] Figure 4 This is an exemplary flow chart of a multi-task convolutional neural network according to some embodiments of the present specification. DETAILED DESCRIPTION

[0078] The method and system provided in the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0079] Figure 1This is an exemplary flow chart of a method for identifying a person's clothing according to some embodiments of this specification. Personnel images of different scenes, different clothing and multiple postures are collected, and RGB images containing different people, different clothing and multiple postures are collected to form a training set. The training set covers multiple scenes to ensure the generalization of the model. The RGB image of the training set is converted to the HLS color space, and the three-channel information of the image's hue, brightness and saturation is extracted. The HLS space suppresses illumination changes and improves the robustness of the model. The HRNet network architecture is used to build a model for detecting key points such as the head, limbs, and torso. HRNet contains a multi-branch parallel structure that can achieve pixel-level positioning accuracy. A convolutional neural network integrating two tasks of clothing classification and posture regression is constructed. It contains a shared convolutional layer to extract features, and fully connected branches corresponding to the two tasks. For HLS images, the coordinates of key points are extracted using the pre-trained HRNet model as feature vectors representing postures. The posture features and HLS images are used as input and passed into the constructed multi-task network. The network simultaneously learns clothing classification and posture regression to achieve personnel clothing recognition. The model recognition accuracy is evaluated on an independent test set. Continuously optimize the network and training strategies to improve the performance of clothing recognition.

[0080] Construct a training set. Collect RGB images of different people, different clothes and multiple postures to form a training set. The training set covers multiple scenes to ensure the generalization of the model. HLS color space conversion. Convert the RGB images of the training set to the HLS color space and extract the three-channel information of the image: hue, brightness and saturation. HLS space suppresses illumination changes and improves the robustness of the model. Construct an HRNet key point detection model. Use the HRNet network architecture to build a model for detecting key points such as the head, limbs and torso. HRNet contains a multi-branch parallel structure and can achieve pixel-level positioning accuracy. Construct a multi-task convolutional neural network. Construct a convolutional neural network that integrates two tasks: clothing classification and posture regression. Contains shared convolutional layers to extract features and fully connected branches corresponding to the two tasks. Pose feature extraction. For HLS images, use the pre-trained HRNet model to extract the coordinates of key points as feature vectors representing postures. Multi-task learning. Take the posture features and HLS images as input and pass them into the constructed multi-task network. The network simultaneously learns clothing classification and posture regression to achieve clothing recognition. Model evaluation: evaluate the recognition accuracy of the model on an independent test set. Continuously optimize the network and training strategy to improve the performance of clothing recognition.

[0081] Data input: Collect images of people in different scenes, different clothes and multiple postures to build a training set. Collect images of people in different scenes in daily life, such as street scenes, companies, homes, etc. Collect images of people wearing different types of clothing, such as short sleeves, jackets, boots and other types of clothing. Collect images of people in different postures, such as standing, sitting, raising hands, squatting and other action postures. Annotate the images, mark the position of the human body, clothing category, and the coordinates of key points of the posture. Use the collected annotated images as the training set, and divide them into training set, validation set and test set in a ratio of 8:1:1. The training set is used for network training, the validation set is used for parameter adjustment, and the test set is used to evaluate performance. The training set needs to be large enough, no less than 100,000 images, to ensure that images of different scenes, clothing and postures are evenly distributed. The images need to be normalized, the human body part is cropped and resized to a uniform input size, such as 224x224. Finally, a multi-task dataset including input images and annotations is formed for network training. By constructing an annotated training set containing diverse data, the network can be effectively trained to learn robust expression features and predictions for different situations.

[0082] Figure 2 This is an exemplary flow chart for performing HLS color space conversion on RGB images shown in some embodiments of this specification. Image preprocessing: construct a generative adversarial framework including a generative network and a discriminative network, and the generative network G adopts an encoder and decoder structure. The encoder uses a convolutional layer to gradually downsample and extract the features of the input image. The decoder uses upsampling and convolutional layers to map the features to a target fake image. The discriminative network D adopts a convolutional neural network structure. The input layer accepts a generated image or a real image as input. The convolutional layer and the pooling layer extract the features of the input image. The fully connected layer maps the features to the judgment result of the authenticity of the image. Networks D and G are connected through capsules to form an adversarial framework. First, fix D and train G to deceive D. Then fix G and train D to identify G's fake images. Alternate training until D can no longer identify the fake images generated by G. Network G can eventually generate high-quality fake images. The framework also defines discriminative loss, generative loss and overall loss function. The model parameters of D and G are updated by back propagation of the loss function. Finally, a pre-trained model for generating adversarial images is obtained.

[0083] The generative network includes a convolutional encoder and an upsampling decoder to achieve RGB to HLS conversion. The encoder contains several layers of convolutional layers and pooling layers. A 3x3 convolution kernel with padding is used to maintain the size of the feature map. The pooling layer gradually reduces the size of the feature map and outputs 32x32 encoded features. The decoder contains several layers of upsampling layers and convolutional layers. First, the feature map is enlarged by a certain multiple using nearest neighbor or bilinear upsampling. Several layers of convolutional layers are then used to perform feature fusion and gradually restore the size of the feature map. The last convolutional layer is mapped to a 3-channel HLS image. The input of the generative network is a 224x224x3 RGB image. The encoder gradually extracts color and texture features and outputs encoded features. Based on the encoded features, the decoder gradually upsamples and generates a 224x224x3 HLS image. During the training process, the network parameters are iteratively updated through back-propagation. The training goal is to generate an HLS image that is consistent with the input RGB content. Finally, a generative model that converts RGB images to HLS color space is obtained.

[0084] The discriminant network contains convolutional layers, pooling layers, and fully connected layers, which are used to distinguish the authenticity of HLS images. The input layer of the discriminant network accepts HLS images of size 224x224x3. First, there are several convolutional layers, using 3x3 convolution kernels with padding to maintain the size. Then there are several 2x2 maximum pooling layers, which gradually reduce the size of the feature map. After the pooling layer, there are 1-2 fully connected layers, and the number of nodes in the fully connected layer is 1024 or 512. The last fully connected layer with 1 node outputs a probability value through sigmoid activation. It represents the probability that the input image is a real sample. In training, the HLS image output by the generating network is used as a fake sample. The HLS image of the training set is used as a real sample. It is sent to the discriminant network to obtain the true and false probability. The binary cross entropy loss is calculated based on the probability and label. The parameters of the discriminant network are updated through back propagation. The training goal is to enable the discriminant network to distinguish between the generated HLS images and the real HLS images. Finally, a CNN model for distinguishing true and false HLS images is obtained.

[0085] Read the training image, perform preprocessing, and input it into the generative network encoder. Read the image in the training data set, which contains the human body and background area. Use methods such as object detection to locate the human body area and crop out the human body part. Ensure that the cropped area contains the complete human body without truncating the body parts. Resize the cropped human body image to 256x256 by scaling. The resized image size is consistent, which is convenient for network training. Input the resized RGB image directly into the encoder of the generative network. The generative network encoder contains several layers of convolution and pooling layers. The input layer size is set to 256x256x3, which is consistent with the input image size. The encoder gradually extracts the color and texture features of the input RGB image. Finally, the reduced encoded feature map is output, such as 32x32x256. The preprocessed training image is input to the encoder to extract the features of the input image. The features are passed to the decoder to generate the target HLS image.

[0086] The encoder extracts features and the decoder generates HLS images. The encoder contains multiple convolutional layers, using 3x3 convolution kernels, stride 1, and padding 1. Each convolution layer is followed by a BatchNorm layer and an activation function ReLU. Max Pooling is used to gradually reduce the size of the feature map. Repeat the above convolution-BN-activation-pooling module. The encoder gradually extracts the color, texture, shape and other features of the input RGB image. Finally, the output is a feature map of reduced size, such as 32x32x256. The decoder first enlarges the feature map through an upsampling module. Then there are several layers of transposed convolution layers, using 3x3 convolution kernels, stride 1, and padding 1. Each layer of transposed convolution is followed by BN and ReLU activation. The transposed convolution gradually restores the feature map to a size of 224x224. The last layer of transposed convolution maps to a 3-channel HLS image. The encoder extracts features and the decoder restores features to complete the conversion from RGB to HLS. Through training, the encoder and decoder can learn the conversion rules of RGB to HLS mapping. Finally, high-quality RGB to HLS color space conversion is achieved.

[0087] The generated HLS image and the real HLS sample are input into the discriminant network. The generated network completes the conversion from RGB to HLS and outputs a 224x224x3 HLS image. Read the original HLS sample corresponding to the training set, which is also 224x224x3 in size. The generated HLS image and the original HLS sample are grouped into mini batches in a 1:1 ratio. The mini batch is labeled according to the generated / real image tags and used to calculate the adversarial loss. The mini batch is input into the input layer of the discriminant network, and the input size is 224x224x3. The discriminant network uses multiple convolutional layers to extract features and uses Max Pooling to reduce the size. After extracting global features, several layers of fully connected layers are connected to output feature vectors. The last fully connected layer performs binary classification and outputs the predicted probability of the generated image. The binary cross entropy loss is calculated based on the predicted probability and label. Back propagation updates the parameters of the discriminant network. The training goal is to enable the discriminant network to distinguish between generated HLS images and real HLS images.

[0088] The generative network G completes the RGB to HLS conversion and outputs the generated HLS image. The encoder uses convolutional layers to gradually downsample and extract the features of the input RGB image. The decoder uses upsampling and convolutional layers to restore the feature map to an HLS image. Input an RGB image of size 224x224x3 to the encoder. The encoder uses 5 convolutional blocks, 2 convolutional layers in each block, to reduce the size. After the encoder, it outputs 32x32x256 encoded features. The decoder first uses bilinear upsampling to enlarge the feature map to 64x64. Then there are 4 decoding convolutional blocks, each containing convolution and transposed convolutional layers. The feature map is gradually restored to a size of 224x224. The last convolutional layer outputs a 224x224x3 HLS image. During training, the parameters of G are updated by calculating the generation loss. It gradually learns to convert RGB images into HLS images that match the content. Finally, a preprocessing model G that can complete high-quality RGB to HLS conversion is obtained.

[0089] Calculate the distance between the generated HLS image and the ground truth HLS image in color value and features as the generated loss LOSS1, and generate the network G to output a generated HLS image of size 224x224x3. Read the original HLS image of the corresponding training sample, which is also 224x224x3. Calculate the mean square error between the generated HLS image and the original HLS image in color values: MSE_loss = average ((Generated_HLS-Ground Truth_HLS)^2). Use image classification datasets such as ImageNet pre-trained VGG16 as the feature extraction network V. VGG16 contains multiple layers of convolution and pooling layers, which can gradually extract image features. Use the VGG16 network pre-trained weights, and freeze the parameters without updating. The generated network outputs a generated HLS image G_HLS of size 224x224x3. Input the real HLS sample image Real_HLS into VGG16. Extract the feature maps of the 3rd and 5th convolution blocks, which are F3 and F5 respectively. Similarly, G_HLS is input into VGG16, and features F3_G and F5_G are output. The MSE loss of the feature map is calculated as: Perceptual_loss = MSE(F3, F3_G) + MSE(F5, F5_G), where F3_G is the output feature map of the third convolutional block, representing low-level features. F5_G is the output feature map of the fifth convolutional block, representing high-level features. Similarly, the corresponding real HLS image Real_HLS is input into V, and feature maps F3 and F5 are output. The mean square error MSE of F3_G and F3 at the pixel level is calculated to represent the difference in low-level features between the two: MSE_F3 = average ((F3-F3_G)^2). Similarly, the MSE of F5 and F5_G is calculated to represent the difference in high-level features: MSE_F5 = average ((F5-F5_G)^2). The perceptual loss is defined as the combination of the two: Perceptual_loss = MSE_F3 + MSE_F5. The generative network is trained by minimizing the perceptual loss so that it generates feature layer expressions that are more similar to the real HLS. Thereby evaluating and optimizing the feature distance of the generated HLS image. The generative network is trained by minimizing the perceptual loss so that it generates images that are more similar to the real HLS. The final generated HLS image can maintain visual features similar to the original image. The generative network G outputs a 224x224x3 generated HLS image G_HLS. Calculate the mean square error MSE_loss between G_HLS and GroundTruth_HLS in color values. Define the feature extraction network V, input G_HLS and GroundTruth_HLS, and output the feature expression. Calculate the perceptual error Perceptual_loss of the feature expressions of the two. The generative loss is defined as the sum of two parts:

[0090] loss1=MSE_loss+λ×Perceptual_loss, where λ is the weight coefficient of perceptual loss, which adjusts the proportion of the two. The parameters of the generative network G are updated through the back propagation of loss1. The training goal is to minimize loss1 so that the generated HLS image is close to the real sample in color and features. G learns to map RGB to a result that is closer to the original HLS sample in terms of content and features. After multiple training iterations, G can generate high-quality HLS images that conform to the real distribution. Finally, a generative model G that can achieve high-quality RGB image to HLS image conversion is obtained.

[0091] The generator network G outputs a 224x224x3 generated HLS image G_HLS. Read the real HLS image Real_HLS corresponding to the training set. Input G_HLS and Real_HLS into the discriminant network D in a 1:1 ratio. D contains a convolutional layer to extract features, and a fully connected layer to output predicted probabilities. For each image, D outputs a predicted probability p in the interval (0, 1). p represents the probability that the image is predicted to be a real sample. The predicted probability for the G_HLS image should be close to 0 and close to 1 for Real_HLS. Define the discriminant network D, which includes convolutional layers and fully connected layers. Input batch size generated HLS images and real HLS images. For each image i, D outputs the predicted probability p i . For the generated image, label y i =0; for real images, y i = 1. Calculate the binary cross entropy loss for each image: BCE_loss i =-(y i ×log(p i )+(1-y i )×log(1-p i )), average the batch size samples to get the discriminant loss: The gradient of loss2 relative to the network parameters is obtained through back propagation. The parameters of network D are updated using an optimization algorithm such as Adam. Repeat the training to minimize the loss2. D gradually learns to distinguish between generated HLS images and real HLS images. Finally, a discriminative model D that can distinguish the authenticity of HLS images is obtained. D and the generative network G form an adversarial framework for training the generative model.

[0092] Define the generation loss loss1, and calculate the distance between the generated HLS image and the real HLS image. Define the discriminant loss loss2, which represents the cross entropy loss of the discriminant network to judge the authenticity of the image. Define the generation loss loss1, and calculate the distance between the generated HLS image and the real HLS image. Define the discriminant loss loss2, which represents the cross entropy loss of the discriminant network to judge the authenticity of the image. The total loss function is the sum of two parts: loss = ξ × loss1 + loss2, where ξ is an adjustable hyperparameter used to balance the proportion of the generation loss loss1. Derivative the loss with respect to the parameters of the generation network G and the discriminant network D. Calculate the gradient of loss relative to the parameters of G and D. Through the back-propagation algorithm, calculate the gradient from the loss layer, through the network layer, and all the way to the parameters of G and D. That is, the reverse gradient propagation of loss to the network parameter θ is realized. According to the gradient, use optimization algorithms such as Adam to update the parameters of the G and D models. Repeated updates will gradually reduce the loss function loss, and the performance of G and D will improve. Finally, G generates high-quality HLS, and D can more accurately distinguish authenticity. Initialize the model parameters θG and θD of the generator G and the discriminator D. In each training iteration, first fix the parameter θG of G and only optimize θD. Calculate the discriminant loss loss2, use the Adam optimizer to calculate the gradient, and update θD. Then fix θD and only optimize θG. Calculate the generation loss loss1, use Adam to calculate the gradient, and update θG. In this way, only one of G or D is optimized in each iteration. Repeat multiple iterations of training. G gradually generates more realistic HLS, and D's discrimination ability is improved. After sufficient adversarial training, G can generate high-quality HLS images. D can no longer tell the difference between the HLS image generated by G and the real HLS image. G deceives D, and D’s judgment is invalid, reaching the equilibrium point of adversarial training. Finally, a generative adversarial model that can be used for high-quality RGB to HLS conversion is obtained.

[0093] Figure 3 This is an exemplary flowchart for detecting key points according to some embodiments of this specification. Key point detection: define an HRNet model, which includes 4 parallel sub-networks, and the inputs are all connected to the HLS image. The resolution of sub-network 1 is 1 / 4 of the input image. The resolution of sub-network 2 is 1 / 2 of the input image. The resolution of sub-network 3 is the original size of the input image. The resolution of sub-network 4 is also the original size of the input image. Each sub-network contains multiple residual modules to extract human features step by step. In each residual module, convolution kernels of different sizes are used, such as 3x3, 5x5, and 7x7. The 3x3 convolution kernel extracts local detail features. The 5x5 and 7x7 convolution kernels extract more global human shape features. The combination of multi-scale convolution kernels can extract human features at different levels. Finally, each sub-network outputs the corresponding human feature representation. Perform key point detection and posture analysis.

[0094] HRNet contains 4 subnetworks, each of which outputs a human feature map. HRNet is defined to contain 4 parallel subnetworks, namely high-resolution, medium-resolution, low-resolution, and ultra-low-resolution subnetworks. Each subnetwork extracts the corresponding human features through a series of convolutions and residual blocks. At the tail position of each subnetwork, a key point heat map prediction module is added. The prediction module input is the human feature map output by the subnetwork. First, a 1x1 convolution layer is used to map the number of channels to the number of key point categories, such as 17 categories. Next are three groups of upsampling layers, which gradually restore the feature map to the input image size. Convolution layers are added between each group of upsampling layers to extract features. The last convolution layer has a Softmax activation function, which outputs the predicted heat map of each key point. In this way, a key point prediction module is added to the end of each subnetwork. Finally, the prediction results of the four subnetworks are integrated to obtain the coordinate information of the detected key points. The input of the prediction module is the feature map output by the subnetwork. First, there is a group of transposed convolution layers for 2x upsampling feature maps. Then there is a group of ordinary convolution layers, and the number of filters can be halved. Then the second set of transposed convolution layers continues to upsample by 2. Repeat the normal convolution layers. The last set of transposed convolution layers upsamples to the same size as the input image. In this way, after 3 sets of upsampling, the heat map is restored to the input resolution. A convolution layer is added in the middle of each upsampling to extract features. Finally, each prediction head can output a key point heat map of the corresponding size.

[0095] HRNet contains multiple subnetworks, each of which outputs a human feature map of the corresponding scale. A key point heat map prediction module is added at the end of each subnetwork. The prediction module inputs the feature map output by the subnetwork, which is of a smaller size. First, there is a 3x3 convolution layer, and the number of output channels is set to the number of key points, such as 17. Next, there are 3 groups of upsampling layers, which use transposed convolution to achieve 2x upsampling. The first group of upsampling enlarges the feature map to 1 / 4 of the original image. The second group of upsampling enlarges the size to 1 / 2 of the original image. The third group of upsampling restores it to the original image input size. Between each group of upsampling, a 3x3 convolution layer is used for feature extraction. The last convolution layer has a Softmax activation and outputs a predicted heat map. Through cascaded upsampling, the heat map is restored to the input image size. The heat map prediction module contains 3 groups of upsampling layers. Each group of upsampling layers uses transposed convolution to enlarge the size of the feature map by 2 times. Between each group of upsampling layers, a convolution layer is set for feature extraction. After the first group of upsampling, a 3x3 convolution layer is set, and the number of filters can be halved. This convolution layer extracts the features after upsampling and reduces the number of parameters. After the second group of upsampling, a 3x3 convolution layer is also set to further extract features. After the last group of upsampling, two groups of 3x3 convolution layers are set in series. The number of filters in the first group of convolutions can be the same as the previous layer. The number of filters in the second group of convolution layers is set to the number of key point categories, such as 17. The second group of convolution layers does not use an activation function and directly outputs a predicted heat map of the key points. Each pixel in the heat map represents the prediction confidence of the corresponding key point.

[0096] HRNet contains 4 subnetworks, each of which outputs a prediction heat map of K key points, where K is the number of key points. For the prediction heat map of the kth key point, traverse the response values ​​of all pixels. Find the pixel p with the largest response value and record its coordinates as (x, y). The maximum response of the heat map indicates that the prediction of the kth key point appears at this pixel. So (x, y) is used as the predicted coordinates of the kth key point. For each key point k=1 to K, repeat steps 2-5 to find the pixel with the maximum response. The coordinates of these K pixels are the prediction results of the subnetwork for all key points. Repeat the above operations for the 4 subnetworks to obtain 4 sets of predicted coordinates. Finally, the 4 sets of coordinates are fused to obtain the final prediction result of the model. HRNet contains 4 subnetworks, each of which outputs a set of key point coordinate prediction results (x1, y1), (x2, y2), (x3, y3), (x4, y4), for a total of 4 sets of predicted coordinates:

[0097] (x 11 ,y 11 ),(x 12 ,y 12 ),(x 13 ,y 13 ),(x 14 ,y14 ) comes from subnetwork 1, (x 41 ,y 41 ),(x 42 ,y 42 ),(x 43 ,y 43 ),(x 44 ,y 44 ) comes from subnetwork 4, and so on, from subnetworks 3 and 4. For each key point k, the simple average of the predicted coordinates of the four subnetworks is calculated: Average coordinate (x k ,y k ) as the final predicted coordinate of key point k. You can also give each subnetwork a weight w and calculate the weighted average coordinate in proportion. The weight can be determined according to the positioning accuracy of each subnetwork. Repeat the calculation of the weighted average coordinate of each key point to obtain the fused prediction result. The key point positioning accuracy of each subnetwork is evaluated through the validation set. Subnetworks with higher positioning accuracy are given higher weight coefficients. For example, if the average positioning error of subnetwork 1 is 3 pixels and that of subnetwork 2 is 5 pixels. Then set the weight coefficient of subnetwork 1 to 0.6 and that of subnetwork 2 to 0.4. For each key point k, calculate the weighted average coordinate: x k =0.6×x 1k +0.4×x 2k ,y k =0.6×y 1k +0.4×y 2k , repeat this step to calculate the weighted average coordinates of all K key points. Finally, the key point detection coordinates after sub-network weight fusion are output. In this way, according to the positioning ability of the sub-network, its weight coefficient is set. Output the human key point detection result integrating multi-branch information.

[0098] A large number of HLS images including human bodies are collected as data sets. Key points are annotated for each image, and the coordinates of key points of the head, limbs, torso and other parts are marked. The annotated coordinates are stored in the json file corresponding to the image. The HLS image is input into the predefined HRNet model. HRNet contains multiple parallel subnetworks to extract multi-scale features of the input image. Each subnetwork outputs a predicted heat map of each key point. Find the point with the maximum response value on the heat map and extract its coordinates as the predicted coordinates. Calculate the loss of the predicted coordinates and the annotated true coordinates to optimize the model parameters. Repeat the operation on the training data set to train the HRNet model. Finally, an HRNet model that accurately detects the coordinates of each key point on the HLS image is obtained. For each training image, the key point coordinates predicted by the model are (x1,y1),......,(x k ,y k). The annotation file provides the real key point coordinates (u1,v1),......,(u k ,v k ). Calculate the Euclidean distance between the predicted coordinates and the true coordinates of each key point: Calculate the loss for all K key points and sum them up to get the total loss. Calculate the gradient of loss relative to network parameters through the back propagation algorithm. Update the network parameters according to the gradient to reduce the loss. Repeat the above steps on the training data set to optimize the HRNet model. Finally, on the test set images, the model can accurately predict the coordinates of the key parts of the human body. The training of the HLS human posture analysis model based on HRNet is completed.

[0099] Figure 4 This is an exemplary flow chart of a multi-task convolutional neural network according to some embodiments of the present specification, feature fusion: Input HLS human image x. Define a pre-trained HRNet model and load parameters. Pass x into the HRNet model for forward calculation. HRNet contains 4 multi-branch sub-networks, and finally outputs the coordinate prediction y of all key points. y is the posture feature, which represents the posture information of the human body in the image. Then define the image feature extraction model, such as ResNet and other CNNs. Define a pre-trained image feature extraction model, such as ResNet50. Load the pre-trained model parameters into the defined model. Pass the input HLS image x into the ResNet50 model for forward calculation. ResNet50 extracts image features through a series of convolutions and residual connection blocks. Output the last fully connected layer features as the image feature representation z. The vector size of z depends on the fully connected layer settings, such as 2048 dimensions. z represents the overall content of the input image x, including image-level semantic information. The posture feature representation y of the current image has been obtained before. The image feature z and the posture feature y are spliced ​​in the channel dimension. The concatenation of the z and y feature vectors [y, z] is obtained. [y, z] integrates two types of complementary feature information. It is input to the subsequent network as a comprehensive representation of image clothing and posture. Image x first passes through HRNet and outputs posture feature y. Then it passes through the CNN model and outputs image feature z. [y, z] combines the features of two different modalities. Define a multi-task network, including a shared convolutional layer and two task-specific branches. Take [y, z] as input and pass it into the multi-task network. In the shared convolutional layer, [y, z] undergoes convolution and pooling to learn high-level feature expressions. The expressed features are used for clothing and posture prediction in two branches. The loss functions of the two tasks are optimized simultaneously through backpropagation. The multi-task network learns to parse the feature representation of posture and clothing from [y, z]. Finally, a model that can perform dual tasks based on image and posture features is obtained.

[0100] Define a multi-task convolutional neural network, including a shared convolutional layer and two task branches. The network input is the concatenation of the image feature z and the posture feature y [y, z]. [y, z] is passed to the shared convolutional layer for feature learning. The shared convolutional layer contains multiple groups of convolutional layers and maximum pooling layers. The convolutional layer uses different convolution kernels to extract features from various perspectives. The pooling layer aggregates features to obtain a higher-level feature representation. Through multiple layers of convolution and pooling, hierarchical feature extraction and fusion are performed. Finally, a high-level comprehensive feature expression of the input image and posture is obtained. The feature expression is output from the shared convolutional layer and enters the two task branches respectively. The clothing branch and the posture branch receive the shared features and perform their respective prediction tasks. In this way, the multi-task network learns the joint expression of image and posture features.

[0101] Define a multi-task network, including a dress detection branch. The dress detection branch receives features from the shared convolutional layer. The branch contains multiple fully connected layers to learn features. The first fully connected layer can reduce the number of channels to 1024 while enhancing the feature expression ability. The second fully connected layer further reduces the dimension to 512. Then there is a fully connected layer with the number of output nodes equal to the number of clothing categories, such as 10 categories of clothing. Finally, a Softmax activation function is connected to obtain the predicted probability of each category. The category corresponding to the maximum probability is the detection result of the image. The prediction of the image clothing category is completed through the hierarchical learning of the fully connected layers of the branch. At the same time, the dress detection loss is optimized to achieve the clothing parsing task. Define the posture analysis branch in the multi-task network. The branch inputs features from the shared convolutional layer. First, pass a fully connected layer to reduce the dimension of the features to 256. Then there is a second fully connected layer with the number of output nodes equal to the number of key points multiplied by 2. For example, if there are 17 key points, the number of output nodes is 34. No activation function is used, and the key point coordinate regression results are directly output. The output result is a vector containing the x and y coordinates of all key points, which has been normalized. The coordinates are mapped to the input image size to obtain the coordinate positions of the key points in the image. Regression is performed through the fully connected layer of the branch to complete the posture analysis task. At the same time, the loss of coordinate prediction is optimized to achieve human posture analysis.

[0102] The multi-task network contains two branches. The clothing branch outputs the classification result, and the posture branch outputs the regression prediction. The classification result output by the clothing branch is connected to the cross entropy loss function. The key point coordinate prediction of the posture branch is connected to the mean square error loss. The losses of the two tasks of clothing classification and posture regression are calculated. The gradient of the two losses is calculated using the back propagation algorithm. The shared parameters in the multi-task network are updated according to the gradient. The process is repeated to continuously optimize the loss functions of the two tasks on the training data. Finally, a multi-task model is obtained, which can simultaneously realize clothing parsing and human posture detection of HLS images. On the test data, the model can correctly predict the image label and key point coordinates. That is, the dual-task analysis of image clothing and posture is completed.

Claims

1. A method for identifying a person's clothing, comprising: Collect images of people in different scenes, different clothes and various postures to build a training set in RGB format; Perform HLS color space conversion on the constructed RGB format training set images to obtain an image dataset in HLS format; Using the obtained HLS format image dataset, a key point detection model based on the HRNet architecture is constructed. The key points include the head, limbs, and torso. Construct a multi-task convolutional neural network that includes clothing detection tasks and posture analysis tasks; Input the collected RGB format personnel image into the constructed key point detection model, and extract the two-dimensional coordinate information of the key points representing the human posture as the posture feature vector; The extracted posture feature vector and the corresponding HLS image are used as input, and the constructed multi-task convolutional neural network is used to perform standard detection of personnel clothing, and output clothing categories and key point coordinates; Perform HLS color space conversion on the constructed RGB format training set images, including: Construct a generative adversarial network that includes a color space conversion generation network and a true-false discrimination network; Input the RGB format training set image into the color space conversion generation network, map the RGB image to the HLS image, and output the generated HLS image; The generated HLS image and the corresponding real HLS sample image are input into the authenticity discrimination network for authenticity discrimination, and the authenticity discrimination result of each HLS image is output; According to the authenticity discrimination results, the loss functions of the generation network and the discrimination network are calculated, and the network parameters are updated through back propagation; the above steps are repeated until the loss function is less than the threshold, and the trained color space conversion generation network is output; The collected RGB format training set is used as input, and the trained color space conversion network is used to generate the corresponding HLS format image dataset; The color space conversion generation network includes an encoder and a decoder; The convolutional layer in the encoder extracts features from the RGB image; The decoder maps the features extracted by the encoder into an HLS image through upsampling and transposed convolution; The loss function includes the generation network loss and the discrimination network loss; The generated network loss includes MSE reconstruction loss and perceptual loss. The perceptual loss is obtained by calculating the distance between the generated image and the real image at different feature layers through the pre-trained feature extraction network. The discriminant network loss includes binary cross entropy loss and feature center loss. The feature center loss makes the feature vectors of similar samples extracted by the discriminant network cluster together. The network loss loss1 is calculated by the following formula: loss1=MSE_loss(Generated_HLS, GroundTruth_HLS)+λ×Perceptual_loss(Generated_HLS, GroundTruth_HLS) Among them, MSE_loss represents the mean square error loss function, which calculates the difference in color value between the generated HLS image and the real HLS image; Generated_HLS represents the HLS image output by the generated network; GroundTruth_HLS represents the corresponding ground truth HLS image; λ represents the weight coefficient, which balances the proportion of MSE loss and perceptual loss; Perceptual_loss represents the perceptual loss function based on feature extraction, which calculates the difference in visual features between the generated HLS image and the real HLS image. Perceptual_loss extracts the feature maps of the two images through the feature network and calculates the similarity between the feature maps; The discriminant network loss loss2 is calculated by the following formula: loss2=BCE_loss(Discriminator(Generated_HLS)+ mu×Center_loss(Feature_vectors)) Among them, BCE_loss represents the binary cross entropy loss function of the discriminant network, which is used to determine the authenticity of the generated HLS image; Discriminator(Generated_HLS) represents the discriminant network; Generated_HLS represents the HLS image output by the generated network; mu represents the weight coefficient of Center_loss; Center_loss represents the center loss function of the feature vector, which is used to make the feature vectors of samples of the same category close to the class center; Feature_vectors represents the feature vector of the fully connected layer in the discriminant network; The calculation formula of the perceptual loss function Perceptual_loss is: in, represents the i-th layer feature map of the image feature extraction network; Generated_HLS represents the HLS image output by the generated network; GroundTruth_HLS represents the corresponding ground truth HLS image; w i Represents the weight coefficient of the feature map of the i-th layer; Distance represents the distance measurement function between feature maps; The calculation formula of the center loss function Center_loss is: Where: x i Represents the feature vector of sample i in the discriminant network feature layer; cy i represents the center of the feature vector of category y to which sample i belongs; ∑i(x i -cy i ) 2 Indicates the distance between the sample feature vector and the corresponding category center vector; According to the loss function loss1 and the loss function loss2, the loss loss of the generated adversarial network is calculated: The loss function of the generative adversarial network is: loss = ξ × loss1 + loss2 Among them, ξ represents the hyperparameter that controls the proportion of the generation network loss and the discriminative network loss.

2. The method for identifying a person's clothing according to claim 1, characterized in that: Build a key point detection model based on the HRNet architecture, including: Take the obtained HLS format image dataset as input; Construct multiple parallel sub-networks with different resolutions. The sub-networks with different resolutions include high-resolution sub-networks and low-resolution sub-networks. The high-resolution sub-network is used to extract local features, and the low-resolution sub-network is used to extract global features. A multi-level residual connection structure is set in each sub-network to extract human features at different levels through convolution kernels of different scales; A key point heat map prediction head is added at the end of each sub-network, the extracted human features are upsampled to restore the resolution, and the confidence heat map of each key point is output; According to the confidence heat map, output the key point coordinates.

3. The method for identifying a person's clothing according to claim 2, characterized in that: Add a key point heat map prediction head at the end of each sub-network, including: Set the heatmap prediction head according to the number of key points; In each prediction head, multiple groups of upsampling layers are set to generate heat maps based on the input human features, and the resolution of the heat map is restored to the same size as the input human features through upsampling; During the upsampling process, a convolutional layer is also set for feature extraction and mapping; Finally, a set of convolutional layers is set to output the confidence heat map of each key point based on the heat map of the restored resolution.

4. The method for identifying a person's clothing according to claim 3, characterized in that: The multi-task convolutional neural network constructed includes: Build a shared convolutional network to extract features from the obtained HLS image; At the end of the shared network, the clothing detection sub-network and the posture analysis sub-network are connected in parallel; The fully connected layer of the clothing detection subnetwork outputs clothing categories; the fully connected layer of the posture analysis subnetwork outputs key point coordinates.

5. The method for identifying a person's clothing according to claim 4, characterized in that: The extracted posture feature vector is input into the posture analysis sub-network of the multi-task convolutional neural network; The obtained HLS image is input into the shared network and clothing detection sub-network of the multi-task convolutional neural network.

Images