Unsupervised human body analysis method and device based on contrast and consistency learning
Through an unsupervised method based on comparison and consistency learning, the component comparison module and pixel consistency module are used to solve the problem of insufficient distinction between similar appearance components and insufficient utilization of global features in human body analysis, and high-precision unsupervised human body analysis is achieved.
Patent Information
- Application Number
- CN202210641926.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-07
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-06-07
AI Technical Summary
The prior art is difficult to distinguish components with similar appearance but different semantics in human body analysis, and does not fully utilize the global characteristics of components, affecting the analytical accuracy.
Using an unsupervised method based on contrast and consistency learning, image views are generated through data amplification, image features are extracted using feature extractors, and feature input component comparison modules and pixel consistency modules are generated through K-Means algorithm to train the network Perform human analysis. The component comparison module distinguishes similar components through global pooling layer and fully connected layer, and the pixel consistency module encourages pixel features consistency at the same location through self-supervisation and sparse encoding.
It realizes pixel-level fine classification of unsupervised human body analysis, distinguishes semantic components and improves the accuracy of pattern analysis, and reduces dependence on labeled data.
Smart Images

Figure CN115147893B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to an unsupervised human body analysis method and device based on contrast and consistency learning. Background Art
[0002] Human body parsing is one of the hottest research topics in computer vision and an important research direction for vision-based pattern recognition applications. Human body parsing is a pixel-level fine semantic classification task that classifies each pixel in an image into human body parts with different semantics, such as head, torso, arms, and legs. It has broad application value and development prospects in the fields of human-computer interaction, posture recognition, pedestrian re-identification, and behavior analysis. Current human body parsing research is mainly based on supervised learning. As performance improves, it requires high labor and time costs to annotate data pixel by pixel.
[0003] With the rapid development of deep learning technology, many fields that require a large amount of image analysis have introduced related algorithms, and the performance of human body parsing tasks has also been greatly improved. However, when applying human body parsing to a specific scenario, it often faces the dilemma of requiring a large amount of labeled data, which greatly increases the time and economic cost of research and development. Recently, unsupervised learning has gradually gained attention. Its training goal does not introduce any labeled information. By designing a sophisticated loss function and integrating corresponding constraints, the target task is achieved. "SCOPS: Self-Supervised Co-Part Segmentation", included in the 2019 IEEE Conference on Computer Vision and Pattern Recognition, solves the target component parsing task in an unsupervised way. It designs three self-supervised constraints, geometric consistency, equivariance constraint and semantic consistency, which enables it to achieve good results in tasks such as face parsing, bird parsing and sheep parsing, but it ignores parts with similar appearance but different semantics.
[0004] In order to achieve unsupervised human parsing tasks, for example, "Unsupervised Part Segmentation through Disentangling Appearance and Shape" included in the 2021 IEEE Conference on Computer Vision and Pattern Recognition decouples the shape and appearance information of object parts and reconstructs the image. In order to decouple more effectively, an extrusion and expansion module is proposed. At the same time, the geometric concentration constraints are improved to enhance the semantic consistency of the parts. However, this method does not fully utilize the global features of the parts, which are critical for parsing the semantics of the parts.
[0005] Unsupervised general object segmentation is also widely used. For example, the invention patent "An image segmentation data annotation method and system based on unsupervised or weakly supervised methods" uses an image domain adaptation module to transform the domain of the image data to be annotated in the pixel space, and then the feature adaptation module deceives the domain discriminator by maximizing the learned source and target representations, and learns differently through adversarial means. However, this method still requires the source data to be annotated.
[0006] However, the existing technology lacks two key issues in solving human body parsing: (1) it is difficult to distinguish parts with similar appearance but different semantics; (2) the global features of the parts are not fully utilized, which affects the parsing accuracy. Summary of the invention
[0007] In view of the deficiencies in the prior art, the present invention provides an unsupervised human body analysis method and device based on comparison and consistency learning.
[0008] To achieve the above object, the present invention can adopt the following technical solutions:
[0009] An unsupervised human body parsing method based on contrast and consistency learning comprises the following steps:
[0010] Step 1: Performing data augmentation on image data containing face analysis, wherein each of the image data generates two view data;
[0011] Step 2: Input the two view data into a feature extractor to extract respective image features;
[0012] Step 3: Input the image features into the component comparison module and the pixel consistency module in sequence to distinguish similar components and encourage the consistency of pixel features at the same position. At the same time, generate pseudo labels through the K-Means algorithm;
[0013] Step 4: Use the pseudo-label as a supervisory signal to train the network. During the test, the network predicts the parsing result of the image.
[0014] The unsupervised human body parsing method based on contrast and consistency learning as described above, further, in the step 1, the shallow appearance of the image data is changed by random photometric transformation, so as to generate two view data of the image data, wherein the generation process is expressed as:
[0015] x 1 =P 1 (x)
[0016] x 2 =P 2 (x)
[0017] As described above, in the unsupervised human body parsing method based on contrast and consistency learning, further, in the step 2, the method for extracting the image features includes any form of fully convolutional neural network.
[0018] As described above, the unsupervised human body parsing method based on contrast and consistency learning, further, in the step 3, the component comparison module includes a global pooling layer and two fully connected layers,
[0019] The processing process in the component comparison module specifically includes:
[0020] For each encoded query component g, there is a set of encoded key components t0,t1,…,t n Correspondingly, for each query component g, the encoded key component has a positive sample t + and N-1 negative samples t - ;
[0021] The encoded query and key components are respectively derived from different view data input into the view data, and each query and key component encodes a human body component;
[0022] Positive sample t + Encode the same part of different view data, negative samples t - encoding other components in different view data;
[0023] Thus, the contrast loss is used to make the features of g and the positive sample t + The features are more similar to N-1 negative samples t - As different as possible, where the differentiation process is expressed as:
[0024]
[0025] Here, τ represents a temperature hyperparameter.
[0026] As described above, the unsupervised human body parsing method based on contrast and consistency learning, further, in the step 3, the pixel consistency module includes a self-supervision module and a FFNs module,
[0027] The processing process in the self-supervision module specifically includes:
[0028] Generate space corresponding features F∈R HW×C The process is expressed as:
[0029] F=multi(m),
[0030] Where Q∈R HW×C , K∈R HW×C and V∈R HW×C are the three inputs of the self-supervision module, H, W, and C represent the height, width, and number of channels of the feature extraction network output m, respectively;
[0031] The processing process in the FFNs module specifically includes:
[0032]
[0033] Among them, F∈R HW×C Denotes the output features of the self-supervisory module, D∈R HW×k represents the encoding matrix, λ represents the regularization parameter. Through the L1 loss parameter, a sparse solution α (α ≥ 0);
[0034] In order to avoid D being arbitrarily large, when α is infinitely close to 0, it is implemented by constraining the size of D. Specifically,
[0035]
[0036] This sparse coding is performed on the FFNs module, taking the output F of the self-supervisory module as input, and its sparse coding loss is as follows:
[0037]
[0038] Then, Dα and F are directly connected as the output of the network, and the output of the pixel consistency module is expressed as:
[0039]
[0040]
[0041] As described above, the unsupervised human body parsing method based on contrast and consistency learning, further, in the step 3, a pseudo label is generated by a K-Means algorithm, and the specific process is expressed as follows:
[0042]
[0043]
[0044] z 1 and z 2 Stored as pseudo labels.
[0045] The unsupervised human body parsing method based on contrast and consistency learning as described above, further, in the step 4,
[0046] With z 1 and z 2 As the training of the pseudo-label supervision network, the training process is expressed as:
[0047]
[0048] An unsupervised human body analysis device, comprising:
[0049] A storage module, which is used to store a human body parsing data set; and
[0050] A processing module coupled to the storage module is used to execute the method described to process the human body parsing data set.
[0051] Compared with the prior art, the present invention has the following beneficial effects:
[0052] (1) Unsupervised methods based on contrast and consistency learning can achieve pixel-level analysis of human images without relying on labeled data;
[0053] (2) The component comparison module distinguishes similar components from components through contrastive learning, which brings semantically consistent components closer together and semantically different components as far apart as possible;
[0054] (3) The pixel consistency module encourages the pixel features of two views of an image to be consistent at the same position;
[0055] (4) Sparse coding selects spatial position correlation to improve the accuracy of pattern analysis. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0057] Figure 1 It is a schematic diagram of the overall process of an embodiment of the present invention;
[0058] Figure 2A schematic diagram of the overall structure of an unsupervised human body parsing network based on contrast and consistency learning is designed for an embodiment of the present invention. DETAILED DESCRIPTION
[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0060] Example:
[0061] It should be noted that the terms "including" and "having" and any variations of the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0062] It is understandable that the present invention proposes to use unsupervised learning to parse the human body, achieving the goal of not relying on labeled data and accurately predicting the category of parts. Using unsupervised methods to achieve the most fine-grained expression of human semantics makes this task very challenging. At the same time, in natural scenes, due to the diversity of scale, occlusion, posture and morphology of human body parts, the apparent characteristics of human body parts vary greatly.
[0063] Therefore, we propose a component comparison module to obtain global component information and distinguish similar components. At the same time, we propose a pixel consistency module to encourage two views of an image obtained after amplification to have consistent pixel features at the same position, thereby achieving accurate unsupervised human body parsing.
[0064] See also Figure 1 , an unsupervised human body parsing method based on contrast and consistency learning can include the following steps:
[0065] Step S1, performing data augmentation on the face analysis image, generating two views for each image;
[0066] Step S2, inputting the two views into the feature extractor respectively to extract image features;
[0067] Step S3, inputting the image features into the component comparison module and the pixel consistency module respectively to distinguish similar components and encourage the consistency of pixel features at the same position, and at the same time, generating pseudo labels through the K-Means algorithm;
[0068] Step S4, using the pseudo-label as a supervisory signal to train the network. During the test, the network predicts the parsing result of the image.
[0069] As an optional implementation, in some embodiments, in step S1, the shallow appearance of the image is changed by random photometric transformation (such as brightness change, grayscale change, hue change) to obtain two views (x 1 and x 2 ), but the semantic content of the two views remains basically unchanged. It can enhance the robustness of the network and reduce the interference of state diversity such as posture and morphology. The formula is as follows:
[0070]
[0071] As an optional implementation, in some embodiments, in step S2, the feature extraction network can be any form of fully convolutional neural network (such as: DeepLab series, PSPNet). This method uses the improved ResNet18 and further uses FPN to expand the receptive field of the network. The two views generated in step S1 are respectively sent to the feature extractor f θ , extract image features.
[0072] As an optional implementation, in some embodiments, in step S3, the unsupervised human body parsing method based on contrast and consistency learning includes two modules: component contrast and pixel consistency. The specific steps are as follows:
[0073] Sub-step S31, the component comparison module consists of a global pooling layer and two fully connected layers. For each encoded query component g, there is a set of encoded key components t0, t1, ..., t n Correspondingly, for each query component g, the encoded key component has a positive sample t + and N-1 negative samples t - The encoded queries and keys come from different views of the input image, and each query and key encodes a human body part. Positive samples t + Encode the same parts in different views, negative samples t - Encode other parts in different views. This method uses contrast loss to make the features of g and positive samples t + The features are more similar to N-1 negative samples t - As different as possible, the formula is as follows:
[0074]
[0075] Here, τ represents a temperature hyperparameter.
[0076] Substep S32 and substep S31 will lose some useful information and cannot meet the requirements of dense pixel classification. Therefore, the pixel consistency module refers to the transformer encoding part and first uses a multi-head self-supervision module, which embeds the features of each element and considers the association with other elements to obtain the spatial correspondence of each pixel. Specifically, we use the output of the feature extraction network through the LN layer to generate three inputs of the self-supervision module, namely Q∈R HW×C , K∈R HW×C and V∈R HW×C , where H, W and C represent the height, width and number of channels of the feature extraction network output m respectively. The self-supervised module generates the spatial corresponding feature F∈R HW×C , the whole process can be described as follows:
[0077] F=multi(m), (3)
[0078] Spatial correlation is obtained by end-to-end training on the human semantic parsing dataset. Feature selection can improve the effectiveness of features and is the key to pattern analysis. Therefore, this method performs sparse operations on the FFNs module of the transformer encoding part. We first briefly review the sparse code algorithm. Sparse code is an algorithm for learning useful sparse representations of any given data. The following mathematical representation of the objective function can help solve this problem:
[0079]
[0080] Among them, F∈R HW×C Denotes the output features of the self-supervisory module, D∈R HW×k represents the encoding matrix, and λ represents the regularization parameter. Through the L1 loss parameter, a sparse solution α (α ≥ 0) is obtained. In order to avoid D being arbitrarily large, when α is infinitely close to 0, it is implemented by constraining the size of D, as shown below
[0081]
[0082] This method performs this sparse coding on FFNs, with the output F of the self-supervisory module of the transformer encoding part as input, and its sparse coding loss is as follows:
[0083]
[0084] Then, Dα and F are directly connected as the output of the network. The pixel consistency module can be described as follows:
[0085]
[0086] In sub-step S33, in order to further obtain pseudo labels of the network, we use K-Means to cluster the features. The process is as follows:
[0087]
[0088] z 1 and z 2 Stored as pseudo labels.
[0089] In some embodiments, in step S4, z 1 and z 2 As the training of pseudo-label supervision network, the process is as follows:
[0090]
[0091] Through the above method, we rely on unlabeled data to accurately predict the category of pixels. The pseudo code of the whole process is as follows:
[0092] Table 1 Schematic diagram of unsupervised human body parsing based on contrast and consistency learning
[0093]
[0094] It is understandable that in order to ensure the stability of training, a smaller learning rate can be used to train for several epochs, and then a larger learning rate can be used to accelerate the convergence of the network and gradually iterate the training.
[0095] At the same time, the present invention also provides an electronic device for executing an unsupervised human body parsing method based on contrast and consistency learning, which includes: an image enhancement module, a component comparison module, a pixel consistency module and a pseudo-label storage module, wherein the image enhancement module amplifies each image into two views through data amplification to enhance network robustness. The component comparison module distinguishes similar components from components through contrast learning, which brings semantically consistent components closer and semantically different components as far as possible. The pixel consistency module encourages two views to have feature consistency at the same position of pixels. The pseudo-label storage module is used to store pseudo-labels generated by clustering through the K-Means algorithm, and then train the parsing network.
[0096] At the same time, the present invention also provides a device for executing an unsupervised human body parsing method based on contrast and consistency learning, which includes: a storage module and a processing module coupled to the storage module, wherein the storage module is used to store a human body parsing data set; the processing module is used to execute the code converted by the unsupervised human body parsing method and then process the human body parsing data set.
[0097] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0098] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.
Claims
1. An unsupervised human body parsing method based on contrast and consistency learning, characterized in that: The following steps are involved: Step 1: Performing data augmentation on image data containing face analysis, wherein each of the image data generates two view data; Step 2: Input the two view data into a feature extractor to extract respective image features; Step 3: Input the image features into the component comparison module and the pixel consistency module in sequence to distinguish similar components and encourage the consistency of pixel features at the same position. At the same time, generate pseudo labels through the K-Means algorithm; Step 4: Use the pseudo-label as a supervisory signal to train the network. During the test, the network predicts the image analysis result. Wherein, in the step 1, the shallow appearance of the image data is changed by random photometric transformation, thereby generating two view data of the image data, wherein the generation process is expressed as: x 1 =P 1 (x) x 2 =P 2 (x); In step 2, the method for extracting the image features includes any form of fully convolutional neural network; In step 3, the component comparison module includes a global pooling layer and two fully connected layers. The processing process in the component comparison module specifically includes: For each encoded query component g, there is a set of encoded key components t0,t1,…,t n Correspondingly, for each query component g, the encoded key component has a positive sample t + and N-1 negative samples t - ; The encoded query and key components are respectively derived from different view data input into the view data, and each query and key component encodes a human body component; Positive sample t + Encode the same part of different view data, negative samples t - encoding other components in different view data; Thus, the contrast loss is used to make the features of g and the positive sample t + The features are more similar to N-1 negative samples t - As different as possible, where the differentiation process is expressed as: Among them, L r represents the component contrast loss and τ represents a temperature hyperparameter.
2. The unsupervised human body parsing method based on contrast and consistency learning according to claim 1, characterized in that: In step 3, the pixel consistency module includes a self-supervision module and a FFNs module. The processing process in the self-supervision module specifically includes: Generate space corresponding features F∈R HW×C The process is expressed as: F=multi(m), Where Q∈R HW×C , K∈R HW×C and V∈R HW×C are the three inputs of the self-supervision module, H, W, and C represent the height, width, and number of channels of the feature extraction network output m, respectively; The processing process in the FFNs module specifically includes: Among them, F∈R HW×C Denotes the output features of the self-supervisory module, D∈R HW×k represents the encoding matrix, λ represents the regularization parameter, and a sparse solution α (α≥0) is obtained through the L1 loss parameter; In order to avoid D being arbitrarily large, when α is infinitely close to 0, it is implemented by constraining the size of D. Specifically, This sparse coding is performed on the FFNs module, taking the output F of the self-supervisory module as input, and its sparse coding loss is as follows: Then, Dα is directly connected to F as the output of the network, the output of the pixel consistency module and Respectively expressed as: in, represents random data augmentation, f θ represents the backbone network, S σ Represents a pixel-consistent module.
3. The unsupervised human body parsing method based on contrast and consistency learning according to claim 2, characterized in that: In step 3, a pseudo label z is generated by the K-Means algorithm 1 and pseudo-label z 2 , the specific process is expressed as: z 1 and z 2 Stored as pseudo labels.
4. The unsupervised human body parsing method based on contrast and consistency learning according to claim 3, characterized in that: In step 4, With z 1 and z 2 As the training of the pseudo-label supervision network, the training process is expressed as: L view Indicates pixel consistency loss.
5. An unsupervised human body analysis device, characterized in that: include: A storage module, which is used to store a human body analysis data set; as well as, A processing module coupled to the storage module, configured to execute the method according to any one of claims 1 to 4 to process the human body parsing data set.
