Multi-attitude personnel re-identification method based on semantic and attitude information fusion

Abstract

The invention discloses a multi-attitude personnel re-identification method based on semantic and attitude information fusion, and relates to the field of computer vision. The model is mainly composed of a semantic guidance module, a visual encoder, a posture guidance feature aggregation module and a decoder based on a posture view. Comprising the following steps that firstly, original image input is transmitted through a semantic guiding module, feature mapping containing global semantic information related to personnel is obtained, and therefore a model can extract more global semantic information features and transmit the feature information between different images; next, in a visual encoder, robust global context semantic information is captured using a Transform-based image classification module and camera viewing angle information. Then, a posture guiding feature aggregation module is used for integrating posture information into global context semantic features and body part features, and the module solves the problem of incomplete feature learning. Subsequently, in a pose view-based decoder, a set of learnable pose views and a pose view matching module are used to implicitly enhance the difficult-to-distinguish body part features, which utilize the learnable pose views to capture features associated with the body part, reduce background noise, and solve the problem of feature mismatch. According to the process, finer features can be obtained on the basis of feature fusion, and the accuracy and efficiency of multi-posture personnel re-recognition are improved.

Description

Technical Field

[0001] The present invention involves multiple fields such as computer vision, data processing, similarity calculation, and more specifically, the field of person re-identification. Background Art

[0002] Person re-identification is a subfield of image retrieval tasks, which aims to match the image to be searched with the images in the gallery. However, most methods focus on a single posture, especially the standing posture outdoors. In real life, since people may not always stand in different scenes and may take different postures, the need for person re-identification in different postures is also common. In public places, the most common posture change of people is the transition between standing and sitting. The performance of traditional person re-identification methods that only focus on standing postures will be greatly reduced. For example, in scenes such as libraries and factories where sign-in is required, people stand when signing in, but sit when working or studying. Multi-pose person re-identification can be used to improve work efficiency and check the usage of library seats. Summary of the invention

[0003] In view of this, the present invention provides a multi-posture person re-identification method based on the fusion of semantic and posture information to solve the problem of multi-posture person re-identification, so as to facilitate the better application of person re-identification technology in real life. This is of great significance for personnel tracking, personnel retrieval, personnel positioning and other aspects.

[0004] In order to achieve the above object, the present invention adopts the following technical solution: a multi-posture person re-identification method based on semantic and posture information fusion, the specific steps include the following:

[0005] Data collection and preprocessing: The model first collects a dataset containing people in multiple poses. The data is preprocessed, including unifying the size and labeling for subsequent analysis;

[0006] Semantic Guidance Module: Batch processed images first enter the semantic guidance module, where 1×1 convolution reduces the dimension and network parameters. Softmax normalization is applied in the context modeling stage to highlight the semantic features related to people;

[0007] Visual encoder: In the visual encoder, the image is divided into multiple patches and the side information is added to convert it into a feature map to vectorize the image;

[0008] Posture-guided feature aggregation module: uses the human posture estimation method HRNet to extract posture information from images, integrates the posture information into global contextual semantic features and body part features, and solves the problem of limited body information in multi-pose human images;

[0009] Pose-view based decoder: uses a set of learnable pose views to facilitate learning to distinguish body parts;

[0010] Encoder loss function: The encoder is optimized by applying ID loss and triplet loss to global features and local features;

[0011] Decoder loss function: The decoder is optimized by applying ID loss and triplet loss to global features and local features;

[0012] Training and Testing: During the training phase, the pose estimation module uses the pre-trained model, while other modules are simultaneously trained using the overall objective loss. During the testing phase, the length of the vector is normalized by padding with zeros.

[0013] Furthermore, after the semantic guidance module uses context modeling and feature conversion, the feature map finally outputted is: .

[0014] Furthermore, the visual encoder utilizes a learnable class embedding is appended to the input, and its output is used as the global feature . Spatial information is integrated by adding learnable position embeddings, and the impact of side information changes on features is mitigated by adding weighted side information. Therefore, the input sequence of each layer of Transformer is represented as .

[0015] Furthermore, the posture-guided feature aggregation module regards the comparison between local features and posture-guided features as a set of similarity measurement problems, and finally obtains the posture-guided feature aggregation: .

[0016] Furthermore, the pose-view based decoder uses a set of learnable pose views to facilitate learning to distinguish body parts and perform pose-view matching: .

[0017] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a multi-posture person re-identification method based on the fusion of semantic and posture information, and the beneficial effects of the present invention are:

[0018] This paper proposes a new research direction for the problem of multi-pose person re-identification: multi-pose person re-identification. Traditional person re-identification methods usually focus on identity matching from a fixed pose, but in practical applications, the pose of an individual varies greatly, which may lead to a decrease in re-identification accuracy. Therefore, accurately identifying the same person in different poses is an important research direction.

[0019] The present invention proposes a multi-posture person re-identification method based on semantic and posture information fusion. By introducing semantic guidance and posture guidance features, the method can better capture and distinguish identity features under different postures, effectively improving the re-identification performance.

[0020] The core advantage of the present invention is that it can fuse semantic information with posture information, making the model more robust when dealing with posture changes. Through semantic guidance, the model can better understand the high-level features of the person; while posture guidance helps the model focus on features related to posture, making the recognition of people in complex scenes more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 This is a process diagram of a multi-pose person re-identification method based on the fusion of semantic and pose information;

[0023] Figure 2 This is a diagram illustrating an example of a semantic guidance module;

[0024] Figure 3 It is the overall flow chart of the model of the present invention; DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0026] The present invention discloses a multi-posture person re-identification method based on the fusion of semantic and posture information, and the specific steps include the following:

[0027] Step 1: Data collection and preprocessing. We constructed a multi-pose dataset Multipose for multi-pose person re-identification. The Multipose dataset consists of indoor images of people in sitting and standing postures, following the naming convention of the dataset Market-1501, and the size is uniformly adjusted to 256×128.

[0028] Step 2: Semantic guidance module.

[0029] ① Context modeling. The purpose of context modeling is to group features at all positions together by weighted averaging, where each feature is assigned a specific weight. This process aggregates features from all locations to obtain global contextual features, providing a comprehensive representation of the entire input data.

[0030] ②Transformation. Use Representation feature transformation process, which aims to capture the dependencies between different channels. The purpose of this transformation is to model and understand the relationship between channels, enhancing the representation by considering the interactions between them.

[0031] The batch processed images first enter the semantic guidance module, where 1×1 convolution reduces the dimensionality and network parameters. Softmax normalization is applied in the context modeling stage to highlight the semantic features related to people. The refined images are then passed to the Transform block for size adjustment in preparation for further analysis. The final output feature map is: .

[0032] in is the global attention pooling weight, is the bottleneck transformation.

[0033] Step 3: Visual Encoder. After the semantic guidance module, the image is converted into feature maps and enters the visual encoder.

[0034] ① Divide the patch. We built an encoder based on Vision Transformer (ViT). Given a feature map ,in represents the height, width, and number of channels. We first Divide Fixed size patches , using the sliding window method. The stride is , the size of each image patch is recorded as , determine the number of patches , as shown below: .

[0035] in Rounding function. Equal to the patch size When , the segmented patches do not overlap. , the generated patches are overlapping, which can alleviate the loss of spatial neighborhood information in the feature map. The smaller the value, the more patches the feature map is divided into. Generally, increasing the number of patches can improve performance, but also requires more computational resources.

[0036] ② Position encoding. In the re-identification task, the image resolution is usually different from the resolution used in image classification, which makes it impractical to directly use the position encoding pre-trained on ImageNet. In order to handle different input resolutions, a bilinear 2D interpolation method is used. This technique is similar to the method used in ViT, allowing the position encoding to be adaptive and learnable to any specified resolution.

[0037] ③ Side information. After obtaining robust feature representation, these features are still susceptible to camera or viewpoint changes. In other words, due to scene bias, the trained model often has difficulty distinguishing the same person from different angles. Therefore, we use the Side Information Embedding (SIE) method, which incorporates non-visual information such as camera ID ( ) and viewpoint labels ( )) is incorporated into the embedding representation to learn invariant features. The camera and viewpoint embeddings are represented as and , which are jointly encoded as The input sequence is then fed into the Transformer layer as follows: .

[0038] A learnable class embedding token is appended to the input, and its output is used as the global feature . Spatial information is integrated by adding learnable position embedding, and the impact of side information changes on features is mitigated by adding weighted side information. Therefore, the input sequence of each layer of Transformer is expressed as:

[0039] in represents the input sequence embedding; is the position code, Encodes side information. is a hyperparameter used to balance the side information embedding weight. Applied to a flat patch, converts it to dimensional embedding. Then, Transformer layer pairs Process and learn feature representation. The encoder finally outputs , which is divided into encoder global semantic features and local features Two parts, The first line of In order to enhance the ability to distinguish the relevant features of human body parts, the human body part features Divide into continuous groups, each with a dimension of , each group and semantic global features Together, they are processed through a shared Transformer layer to extract Local grouping part features, expressed as .

[0040] Step 4: Pose-guided Feature Aggregation Module. Since the body information of multi-pose human images is limited, the information of non-body parts may not be clear, resulting in poor performance. Therefore, we use a human pose estimator to extract pose information from the image.

[0041] ① Pose estimation. We use HRNet as a method for human pose estimation. For a given person image, the estimator extracts The heatmap is then created using the landmarks Each heatmap is scaled down to The maximum response point in each heat map indicates the location of the key point. We establish a threshold To distinguish high confidence from low confidence. Each heat map is from the set is assigned a label , The heat map annotation format is as follows: .

[0042] in, express Confidence score of the landmark. Each pixel value in the heat map represents the confidence that the location is a key point. The maximum value in each heat map represents the confidence score of the key point , the position of the maximum value represents the coordinates of the key point.

[0043] ② Posture-oriented feature aggregation. In order to integrate posture information, we set , aligning it exactly with the number of keypoints. Then, the fully connected layer Processing is performed to obtain the local features of the group Improved heatmap with same dimensionality . The improved heatmap and Multiply element by element to generate posture-guided features .Although encodes specific information about different human body parts, but our goal is to determine which parts of the body contribute most to each specific body part. To achieve this, we use a matching and assignment mechanism that treats the comparison between local features and pose-guided features as a set of similarity measures. Finally, we obtain the pose-guided feature aggregation For each , we identify The two features are then combined into . The form is as follows: , .

[0044] in , express Zhongyu Most similar features.

[0045] Step 5: Pose view based decoder. We use a set of learnable pose views to facilitate learning to distinguish body parts. These views are represented as , Added as a query to each cross-attention layer. The keys and values ​​are derived from the pose heatmap and encoder output An average pooling layer is applied to the heatmap. , and then with Multiply and finally get the output Formally, the formulas for query, key, and value are as follows: .

[0046] in . Linear Projection and Applied to pose views and features . This can then be obtained by implementing a multi-head attention mechanism and two fully connected layers View Collection .

[0047] ① Pose view matching module. In the cross attention mechanism, Pose views learn discriminative features, but it is unclear what specific information is learned. To identify features related to the human skeleton, we introduce a pose view matching module. Each feature in corresponds to a key point. By evaluating the pose part view and The similarity between Related to a specific key point, then the matched and Combine to form the final feature set The formula is as follows, , .

[0048] Since the landmark confidence score can reveal which part of the feature contains information about the human body, the heatmap label It can help us to set the view feature set There are two types of heatmap labels. The features in the view set of compose a high-confidence key point view feature set , the remaining features constitute the low confidence key point view feature set ,in express The number of features whose heatmap label is 1.

[0049] Step 6: Encoder loss function. The encoder is optimized by applying ID loss and triplet loss to global features and local features. ID loss is the cross entropy loss: .

[0050] For triples , the triplet loss is: .

[0051] The loss function of the encoder can be expressed as: .

[0052] in Represents the true distribution of labels, usually expressed as a one-hot encoding. In a one-hot encoding, the position corresponding to the true class of the sample is 1, while the positions corresponding to all other classes are 0. The model predicts that the sample belongs to The probability of the class. is the total number of categories.

[0053] Step 7: Decoder loss function. In order to emphasize the features of non-occluded parts of the body, we use Pose-guided Push Loss:

[0054] in represents the training batch size. If and If the features in are similar, then will be high, prompting the learnable pose view to be adaptively adjusted. In order to facilitate the learning of the decoder view feature representation, An average pooling layer is applied on top to derive the pose-guided decoder global features Subsequently, ID loss and triplet loss are used to guide the pose-guided decoder global features. and high confidence features The learning is shown below. .

[0055] Step 8: Training and testing. During the training phase, the pose estimation module uses the pre-trained model, while other modules, including the encoder and decoder, are trained using the overall target loss at the same time, as shown below: .

[0056] in and represents the weighting factor of the encoder and decoder losses. In the test phase, , , and Connection Formation , and ignore However, due to The lengths are different, so this can be challenging to implement. To address this, we normalize by padding with zeros. arrive Length: .、

[0057] The multi-posture person re-identification method based on semantic and posture information fusion proposed in the present invention solves the multi-posture person re-identification problem, and the method is simple and easy to implement.

[0058] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A multi-pose person re-identification method based on semantic and pose information fusion, characterized in that: The specific steps include the following: Data collection and preprocessing: The model first collects a dataset containing people in multiple poses. The data is preprocessed, including unifying the size and labeling for subsequent analysis; Semantic Guidance Module: Batch processed images first enter the semantic guidance module, where 1×1 convolution reduces the dimension and network parameters; Softmax normalization is applied in the context modeling stage to highlight the semantic features related to people; Visual encoder: In the visual encoder, the image is divided into multiple patches and the side information is added to convert it into a feature map to vectorize the image; Posture-guided feature aggregation module: uses the human posture estimation method HRNet to extract posture information from images, integrates the posture information into global contextual semantic features and body part features, and solves the problem of limited body information in multi-pose human images; Pose-view based decoder: uses a set of learnable pose views to facilitate learning to distinguish body parts; Encoder loss function: The encoder is optimized by applying ID loss and triplet loss to global features and local features; Decoder loss function: The decoder is optimized by applying ID loss and triplet loss to global features and local features; Training and Testing: During the training phase, the pose estimation module uses the pre-trained model, while other modules are simultaneously trained using the overall objective loss. During the testing phase, the length of the vector is normalized by padding with zeros.

2. The multi-posture person re-identification method based on semantic and posture information fusion according to claim 1 is characterized in that: After context modeling and feature conversion, the final output feature map is: .

3. The multi-posture person re-identification method based on semantic and posture information fusion according to claim 1 is characterized in that: A learnable class embedding is appended to the input, and its output is used as the global feature , spatial information is integrated by adding learnable position embedding, and the impact of side information changes on features is mitigated by adding weighted side information. Therefore, the input order of each layer of Transformer is expressed as: .

4. The multi-posture person re-identification method based on semantic and posture information fusion according to claim 1 is characterized in that: The comparison between local features and pose-guided features is regarded as a set of similarity measurement problems, and finally the pose-guided feature aggregation is obtained: .

5. The multi-posture person re-identification method based on semantic and posture information fusion according to claim 1 is characterized in that: Use a set of learnable pose views to facilitate learning to distinguish body parts and perform pose view matching: .

Images