A video dressing-changing person re-identification method with robust auxiliary modality

Abstract

The present invention discloses a method for re-identifying pedestrians in video with a robust auxiliary modality, aiming to improve the recognition accuracy of pedestrians after changing clothes in surveillance videos. The method comprises: selecting a video dressing dataset to construct a training set; constructing a contour posture fusion auxiliary enhancement model, extracting clothing-independent features through preprocessing, encoding, and pedestrian clothing-independent feature mining stages; training the model to obtain a predicted ID probability vector; inputting a test video sequence into the model for pedestrian re-identification; the method realizes more robust pedestrian recognition by constructing an auxiliary modality that resists clothing information interference and the influence of dressing.

Description

Technical Field

[0001] The present invention relates to a person re-identification (ReID) technology, and in particular to a video dressing-changing person re-identification method for constructing a robust auxiliary modality. Background Art

[0002] As a sub-problem in the field of computer vision, pedestrian re-identification (ReID) aims to accurately retrieve the same pedestrian from different surveillance videos, which has important application value in the fields of smart security, smart cities, and judicial investigation. In recent years, the recognition accuracy of ReID models has been significantly improved by learning pedestrian appearance features. However, in real life, pedestrian clothing information often changes over time. For example, criminals may change their clothes in a short period of time to escape pursuit, which causes the recognition accuracy of traditional ReID models based on pedestrian appearance features to be affected to a certain extent.

[0003] It is well known that people can recognize their familiar friends in real life even if their friends' clothes have changed. This is because the human brain can capture features that are not related to clothing, such as the friend's body shape, gait, etc. In order to reduce the impact of clothing on the ReID model, a video-clothing-changing ReID (VCC-ReID) method has been proposed. Existing video-clothing-changing ReID methods mainly focus on how to extract features that are not related to clothing.Most of the current methods, such as the method proposed by Liu et al. (from Liu, Feng, et al. "Learning clothing and pose invariant 3D shape representation for long-term person re-identification." Proceedings of the IEEE / CVF International Conference on Computer Vision. 2023. (Learning clothing and pose invariant 3D shape representation for long-term person re-identification, Proceedings of the IEEE / CVF International Conference on Computer Vision)), the method proposed by Chang et al. (from Chang, Zhigang, et al. "Seq-Masks: Bridging the gap between appearance and gait modeling for video-based person re-identification." 2021International Conference on Visual Communications and Image Processing(VCIP). IEEE, 2021. (Sequence-Mask: Bridging the gap between appearance and gait modeling for video-based person re-identification, International Conference on Visual Communications and Image Processing)), and the method proposed by Zhenyu et al. (from Zhenyu Cui, Jiahuan Zhou, Yuxin Peng, Shiliang Zhang, and Yaowei Wang. Dcr-reid: Deepcomponent reconstruction forcloth-changing person re-identification. IEEETCSVT, 2023. (Circuits and Systems in Video Technology for Cloth-Changing Person Re-identification Based on Deep Component Reconstruction) and the method proposed by Xiong et al. (derived from Cloth-Changing Person Re-identification with InvariantFeature Parsing for UAVs Applications) generate auxiliary modalities from the original images of pedestrians to help improve the model's ability to extract features unrelated to clothing, thereby making the features of pedestrians in different clothing similar.However, the auxiliary modalities generated by these methods can resist the interference of clothing information (such as color and texture), but ignore the impact of changing clothes. For example, the method proposed by Chang et al. generates a contour map modality without clothing information, and the auxiliary model focuses on the pedestrian gait characteristics. But it is not difficult to imagine that when pedestrian clothing undergoes a major change (changing clothes), there will be obvious differences in the contour map, which will affect the model recognition ability to a certain extent. Therefore, studying a more robust auxiliary modality that can resist the interference of clothing information itself and is not affected by changing clothes will help improve the model recognition accuracy.

[0004] Obviously, the joint features of pedestrians are natural attributes of the human body, which are usually relatively stable and not easy to change. Even if the pedestrian's clothing changes, the pose heat map containing the pedestrian's joint information can also resist the change of pedestrian clothing well. Nevertheless, since the pose heat map is only composed of the pedestrian's joint information, the features it contains that are not related to clothing are limited. The pedestrian's contour map contains rich pedestrian discriminant features, but it is still affected by clothing changes to a certain extent. In other words, compared with the pose heat map, the contour map has richer pedestrian discriminant information, which increases the discriminability between pedestrians and helps to distinguish pedestrians, but it cannot narrow the distance between the feature values ​​of the same pedestrian when the clothing changes greatly, and the pose heat map can make up for this defect. This means that the information of the contour map and the pose heat map is complementary, so the two can be combined to construct an auxiliary modality that is more robust to the clothing information itself and the change of clothing, thereby weakening the influence of clothing on the model. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method for re-identifying pedestrians after changing clothes in videos by constructing a robust auxiliary modality. By constructing a more robust auxiliary modality that can resist the interference of the clothing itself information and is not affected by the change of clothes, the model can effectively improve the recognition accuracy of pedestrians after changing clothes in surveillance videos.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: a method for video dressing pedestrian re-identification with a robust auxiliary modality, characterized by comprising the following steps:

[0007] Step 1: Select a video dressing dataset; then construct a training set based on the video dressing dataset. The training set contains multiple training video sequences, and each training video sequence contains T frames of video frames with the same person;

[0008] Step 2: Build a contour posture fusion auxiliary enhancement model, including preprocessing stage, encoding stage, and pedestrian clothing-independent feature mining stage;

[0009] In the preprocessing stage, each video frame in the training video sequence is scaled into a standard original image of uniform size through an image scaling module, each video frame in the training video sequence is segmented through a semantic segmentation model to obtain a corresponding contour map, and each video frame in the training video sequence is passed through a posture estimation model to obtain a corresponding posture heat map;

[0010] In the encoding stage, each standard original image is input into the first encoder to obtain the corresponding original image features; each contour image is input into the second encoder to obtain the corresponding contour image features; each posture heat map is input into the third encoder to obtain the corresponding posture heat map features;

[0011] In the stage of mining pedestrian clothing-independent features, T original image features, T contour features and T posture heat map features are input into the spatial branch used to extract pedestrian clothing-independent features at the spatial level, and the spatial representation features corresponding to the training video sequence are obtained; T contour features and T posture heat map features are input into the temporal branch used to extract pedestrian clothing-independent features at the temporal action level of pedestrians, and the pedestrian temporal action features corresponding to the training video sequence are obtained; the spatial representation features corresponding to the training video sequence and the pedestrian temporal action features are concatenated to obtain the temporal and spatial fusion features corresponding to the training video sequence; the temporal and spatial fusion features corresponding to the training video sequence are input into the fully connected layer to obtain the predicted ID probability vector corresponding to the training video sequence;

[0012] Step 3: Based on the training set, the contour posture fusion auxiliary enhancement model is trained, and after the training, the trained contour posture fusion auxiliary enhancement model is obtained;

[0013] Step 4: Randomly select a pedestrian video sequence; then, based on the pedestrian video sequence, obtain a test video sequence containing T frames of pedestrian video frames; then input the test video sequence into the trained contour and posture fusion auxiliary enhancement model to obtain the temporal and spatial fusion features corresponding to the test video sequence, and then achieve the purpose of pedestrian re-identification in the test video sequence based on the temporal and spatial fusion features.

[0014] In step 1, the construction process of the training set is as follows: for any video sequence in the video dressing dataset, the number of video frames contained in the video sequence is set to N, and the video sequence is divided into T groups according to the time axis, each group contains (NN%T) / T video frames, where % is the modulo operator; then a video frame is randomly extracted from each group; and the T video frames extracted from the T groups constitute a training video sequence; each video sequence in the video dressing dataset is processed in the same way to obtain the corresponding training video sequence, and then the training set is obtained.

[0015] In step 2, the sizes of the standard original image, contour image, and posture heat map are all C×H×W, where C is the number of channels, H is the height, and W is the width.

[0016] In step 2, the first encoder, the second encoder, and the third encoder are three ResNet50 networks with independent weights. The sizes of the original image features, the contour image features, and the posture heat map features are all C1×H1×W1, where C1 is the number of channels, H1 is the height, and W1 is the width.

[0017] In the step 2, the spatial branch consists of an adaptive spatial feature fusion module and a spatial feature guidance module; each contour map feature and a corresponding posture heat map feature are input into the adaptive spatial feature fusion module together, and the contour map feature and the posture heat map feature are complementary fused to obtain the corresponding contour posture fine-grained fusion feature, and the size of the contour posture fine-grained fusion feature is C1×M, M=H1×W1; then each original image feature and a corresponding contour posture fine-grained fusion feature are input into the spatial feature guidance module together to obtain the corresponding contour guidance feature, and the size of the contour guidance feature is C1×M; then the T contour guidance features corresponding to the training video sequence are respectively subjected to the spatial maximum pooling operation, and then the T results after the spatial maximum pooling operation are subjected to the temporal average pooling operation in the channel dimension to obtain the spatial representation feature corresponding to the training video sequence, and the size of the spatial representation feature is C1.

[0018] The adaptive spatial feature fusion module includes three layers of normalization, six MLPs, two cross attentions, and a feedforward network, and the weights of the two cross attentions are independent; the implementation process of the adaptive spatial feature fusion module is: the features obtained by merging the width and height of the contour map features are input into the first layer normalization, and the features output by the first layer normalization are respectively input into three of the MLPs, and these three MLPs map the features output by the first layer normalization into three features with unchanged shapes, which are respectively used as query information, key information, and value information corresponding to the contour map features; the features obtained by merging the width and height of the posture heat map features are input into the second layer normalization, and the features output by the second layer normalization are respectively input into the other three MLPs, and these three MLPs map the features output by the second layer normalization into three features with unchanged shapes, which are respectively used as The query information, key information, and value information corresponding to the posture heat map feature; the query information corresponding to the contour map feature and the key information and value information corresponding to the posture heat map feature are input into the first cross attention to obtain the first attention feature; the query information corresponding to the posture heat map feature and the key information and value information corresponding to the contour map feature are input into the second cross attention to obtain the second attention feature; the first attention feature is added to the feature of the normalized output of the first layer, and the second attention feature is added to the feature of the normalized output of the second layer, and then the feature obtained by splicing the two addition results in the channel dimension is input into the third layer normalization, and the feature of the normalized output of the third layer is input into the feedforward network, and the feature output of the feedforward network is added to the feature of the normalized output of the third layer to obtain the corresponding contour posture fine-grained fusion feature.

[0019] The spatial feature guidance module includes three layers of normalization, three MLPs, a cross attention, and a feedforward network; the implementation process of the spatial feature guidance module is: the features obtained by merging the width and height of the original image features are input into the first layer of normalization, the features output by the first layer of normalization are input into the first MLP, and the first MLP maps the features output by the first layer of normalization into shape-invariant mapping features as query information corresponding to the original image features; the contour posture fine-grained fusion features are input into the second layer of normalization, the features output by the second layer of normalization are input into the second and third MLPs respectively, and the second MLP maps the features output by the second layer of normalization into shape-invariant mapping features. The third MLP maps the features of the second layer normalization output into shape-invariant mapping features as the key information corresponding to the contour posture fine-grained fusion features. The query information corresponding to the original image features and the key information and value information corresponding to the contour posture fine-grained fusion features are input into the cross attention to obtain the attention features. The attention features are added to the features of the first layer normalization output, and the result of the addition is input into the third layer normalization. The features of the third layer normalization output are input into the feedforward network, and the features output by the feedforward network are added to the features of the third layer normalization output to obtain the corresponding contour guided features.

[0020] In the step 2, the temporal branch is composed of an adaptive motion feature enhancement module; after the T contour map features are subjected to a temporal average pooling operation in the channel dimension, they are subjected to a merging operation in the spatial dimension to obtain a first temporal feature, and the size of the first temporal feature is C1×M; after the T posture heat map features are subjected to a temporal average pooling operation in the channel dimension, they are subjected to a merging operation in the spatial dimension to obtain a second temporal feature, and the size of the second temporal feature is C1×M; then the first temporal feature and the second temporal feature are input into the adaptive motion feature enhancement module together to obtain a temporal fusion enhancement feature corresponding to the training video sequence, and the size of the temporal fusion enhancement feature is C1×M1, where M1=2×M; then the temporal fusion enhancement feature is subjected to a spatial maximum pooling operation to obtain a pedestrian temporal motion feature corresponding to the training video sequence, and the size of the pedestrian temporal motion feature is C1.

[0021] The adaptive action feature enhancement module includes four layers of normalization, six MLPs, two cross attentions, and two feedforward networks, and the weights of the two cross attentions are independent; the implementation process of the adaptive action feature enhancement module is: inputting the first temporal feature into the first layer of normalization, inputting the features output by the first layer of normalization into three MLPs respectively, and the three MLPs map the features output by the first layer of normalization into three features with unchanged shapes, which are respectively used as the query information, key information, and value information corresponding to the first temporal feature; inputting the second temporal feature into the second layer of normalization, and inputting the features output by the second layer of normalization into another three MLPs respectively, and the three MLPs map the features output by the second layer of normalization into three features with unchanged shapes, which are respectively used as the query information, key information, and value information corresponding to the second temporal feature; inputting the query information corresponding to the first temporal feature and the key information and value information corresponding to the second temporal feature into the first cross attention, and obtaining The first attention feature is obtained; the query information corresponding to the second temporal feature and the key information and value information corresponding to the first temporal feature are input into the second cross attention to obtain the second attention feature; the first attention feature is added to the feature output by the first layer normalization, the result of the addition is input into the third layer normalization, the feature output by the third layer normalization is input into the first feedforward network, the feature output by the first feedforward network is added to the feature output by the third layer normalization to obtain the first temporal enhancement feature; the second attention feature is added to the feature output by the second layer normalization, the result of the addition is input into the fourth layer normalization, the feature output by the fourth layer normalization is input into the second feedforward network, the feature output by the second feedforward network is added to the feature output by the fourth layer normalization to obtain the second temporal enhancement feature; the first temporal enhancement feature and the second temporal enhancement feature are concatenated in the channel dimension to obtain the temporal fusion enhancement feature corresponding to the training video sequence.

[0022] The training process of step 3 is:

[0023] Step 3.1: Randomly divide all training video sequences in the training set into multiple batches, so that each batch contains batchsize training video sequences;

[0024] Step 3.2: Take one of the batches in the training set, and use all the training video sequences in this batch as the input of the contour and pose fusion-assisted enhancement model. Input them into the contour and pose fusion-assisted enhancement model for training, and obtain the predicted ID probability vector corresponding to each training video sequence in this batch.

[0025] Step 3.3: For each training video sequence in this batch, calculate the ID loss and triplet loss corresponding to the training video sequence, and then calculate the total loss corresponding to the training video sequence, which is the sum of the ID loss and the triplet loss; then use the total loss corresponding to all training video sequences in this batch to update the network parameters through back propagation;

[0026] Step 3.4: Repeat the process of step 3.2 and step 3.3 until all batches of the training set have trained the contour pose fusion auxiliary enhancement model once;

[0027] Step 3.5: Repeat the process from step 3.1 to step 3.4, and end the training process after executing the preset rounds to obtain the trained contour posture fusion auxiliary enhancement model.

[0028] Compared with the prior art, the advantages of the present invention are:

[0029] 1) The present invention proposes a multi-dimensional contour and posture fusion auxiliary enhancement model, which utilizes the feature complementarity and similar semantics of contour map and posture heat map, and extracts clothing-independent features in video frames from the multi-dimensional space-time perspective by mining pedestrian clothing-independent features. At the spatial level, considering that there are interference features such as background, clothing color, and clothing texture in the standard original image, and the fusion mode of contour map and posture heat map does not contain these interferences, and the fusion mode lacks fine-grained clothing-independent features of pedestrians in the standard original image, such as faces, jewelry, etc., the fusion mode is used as an auxiliary mode to assist the contour and posture fusion auxiliary enhancement model to extract clothing-independent features in the original image features and eliminate interference information such as background and clothing appearance. Based on the above considerations, the contour map and posture heat map are fused at the video frame level using an adaptive spatial feature fusion module in the spatial branch to obtain the spatial complementary features of each video frame, and then the spatial feature guidance module uses the spatial complementary features to guide the contour and posture fusion auxiliary enhancement model to focus on the pedestrian clothing-independent features in each video frame. At the temporal level, what is extracted is the temporal motion features of pedestrians. This feature exists and is consistent in the standard original image, contour image, and posture heat map. The standard original image contains more interference noise, so the temporal motion features of pedestrians are captured and fused directly from the contour image features and the posture heat map features. Therefore, in the temporal branch, the adaptive motion feature enhancement module is used to fuse the temporal motion features in the contour image and the posture heat map to enhance the motion feature's resistance to clothing changes. In summary, the contour-gesture fusion-assisted enhancement model can effectively improve the recognition accuracy of pedestrians who have changed clothes in surveillance videos.

[0030] 2) A large number of experiments have proved that the contour and pose fusion auxiliary enhancement model proposed in the present invention can better mine more comprehensive and robust clothing-independent features from video frames, and these features are more resistant to changes in pedestrian clothing.

[0031] 3) Each training video sequence and test video sequence contains only T video frames. Since the state of pedestrians does not change much in a short period of time in a pedestrian video sequence, a video frame can be randomly selected from multiple consecutive video frames, and T video frames can be selected from the entire pedestrian video sequence. This method not only reduces the computational complexity of the model, but also reduces the memory occupancy rate of the computing device. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a composition framework diagram of the contour posture fusion auxiliary enhancement model constructed in the method of the present invention;

[0033] Figure 2 It is a composition framework diagram of an adaptive spatial feature fusion module (ASFM) in a contour posture fusion auxiliary enhancement model constructed in the method of the present invention;

[0034] Figure 3 It is a composition framework diagram of the spatial feature guidance module (SGM) in the contour posture fusion auxiliary enhancement model constructed in the method of the present invention;

[0035] Figure 4 It is a composition framework diagram of an adaptive action feature enhancement module (AMEM) in the contour posture fusion auxiliary enhancement model constructed in the method of the present invention;

[0036] Figure 5 It is a schematic diagram of the corresponding relationship between the contour map and the posture heat map;

[0037] Figure 6 Schematic diagram of posture consistency between contour map and posture heat map in the same video frame;

[0038] Figure 7 A schematic diagram of mAP and Rank 1 in the CL-FULL case in the ablation test on the CCPG-D dataset to analyze the impact of the number of frames on the model;

[0039] Figure 8 The t-SNE visualization of the corresponding features of videos of 9 randomly selected pedestrians under various conditions in the CCPG-D dataset is shown. Each pedestrian is labeled with a specific color. Part (a) shows the visualization results of the Baseline model, and part (b) shows the visualization results of the model of the present invention. DETAILED DESCRIPTION

[0040] The present invention is further described in detail below with reference to the accompanying drawings.

[0041] The present invention proposes a method for constructing a robust auxiliary modality for video dressing-changing pedestrian re-identification, which comprises the following steps:

[0042] Step 1: Select a video dressing dataset, which contains multiple video sequences. Each video sequence contains several video frames with the same person, and each video frame is a three-channel color image. Then, based on the video dressing dataset, construct a training set. The training set contains multiple training video sequences, and each training video sequence contains T frames of video frames with the same person.

[0043] It is further defined that the process of constructing the training set is as follows: for any video sequence in the video dressing dataset, the number of video frames contained in the video sequence is set to N, and the video sequence is divided into T groups according to the time axis, each group contains (NN%T) / T video frames, where % is the modulo operator, and T=8 is taken in this embodiment; then a video frame is randomly extracted from each group; and the T video frames extracted from the T groups constitute a training video sequence; each video sequence in the video dressing dataset is processed in the same way to obtain the corresponding training video sequence, and then the training set is obtained.

[0044] Step 2: To solve the problem that the auxiliary modality in the current VCC-ReID task is not robust enough, in order to extract more robust clothing-independent features from video sequences, a silhouette pose fusion assisted enhancement model (CSPAE, Combination of Silhouette and Pose Heatmap Fusion Assisted Enhancement) is constructed, as shown in Figure 1 As shown, the method includes a preprocessing stage, an encoding stage for extracting pedestrian-related features, and a pedestrian clothing-independent feature mining stage for extracting features that are independent of pedestrian clothing.

[0045] In the preprocessing stage, each video frame in the training video sequence (the i-th video frame is I i , i=1,2,…,T) is scaled into a standard original image of uniform size (I i The corresponding standard original image is r i , i=1,2,…,T), each video frame in the training video sequence is segmented by the semantic segmentation model (Segmentation) to obtain the corresponding contour map (I i The corresponding contour is s i, i=1,2,…,T), each frame in the training video sequence is passed through the pose estimation model (Pose Estimation) to obtain the corresponding pose heat map (I i The corresponding posture heat map is p i , i=1,2,…,T).

[0046] In the encoding stage, each standard original image is input into the first encoder E r In the example above, the corresponding original image features are obtained; each contour image is input into the second encoder E s In the example above, the corresponding contour features are obtained; each posture heat map is input into the third encoder E p , and obtain the corresponding posture heat map features.

[0047] In the stage of mining pedestrian clothing-independent features, T original image features, T contour image features, and T posture heat map features are input into the spatial branch used to extract spatial-level pedestrian clothing-independent features to obtain the spatial representation features f corresponding to the training video sequence. s ; Input T contour features and T posture heat map features into the temporal branch used to extract pedestrian clothing-independent features at the temporal action level of pedestrians, and obtain the pedestrian temporal action features f corresponding to the training video sequence t ; The spatial representation feature f corresponding to the training video sequence s and pedestrian temporal motion features f t After splicing, the temporal and spatial fusion features f corresponding to the training video sequence are obtained; the temporal and spatial fusion features f corresponding to the training video sequence are input into the fully connected layer (FC) to obtain the predicted ID probability vector corresponding to the training video sequence.

[0048] Here, the sizes of the standard original image, contour image, and posture heat map are all C×H×W, where C is the number of channels, H is the height, and W is the width. In this embodiment, C=3, H=256, and W=128.

[0049] As mentioned above, the image scaling module adopts the existing image scaling technology; the semantic segmentation model adopts the existing semantic segmentation algorithm, which is derived from He, Kaiming, et al. "Mask r-cnn." Proceedings of the IEEEinternational conference on computer vision. 2017. (Mask r-convolution, International Computer Vision Conference); the pose estimation model adopts the existing pose estimation algorithm, which is derived from Sun, Ke, et al. "Deep high-resolution representation learning for human pose estimation." Proceedings ofthe IEEE / CVF conference on computer vision and pattern recognition. 2019. (High-resolution representation learning for human pose estimation, Computer Vision and Pattern Recognition Conference).

[0050] The preferred solution is that the first encoder E r , the second encoder E s , the third encoder E p It is a ResNet50 network with three independent weights. The sizes of the original image features, contour image features, and posture heat map features are all C1×H1×W1, where C1 is the number of channels, H1 is the height, and W1 is the width. In this embodiment, C1=2048, H1=16, and W1=8.

[0051] The preferred solution is that the spatial branch consists of an adaptive spatial feature fusion module (ASFM) and a spatial feature guidance module (SGM); each contour feature and a corresponding posture heat map feature are input into the adaptive spatial feature fusion module, and the contour feature and the posture heat map feature are complementary fused to obtain the corresponding contour posture fine-grained fusion feature, s i The corresponding contour features and p i The corresponding posture heat map feature is subjected to ASFM to obtain the contour posture fine-grained fusion feature f i fuse, the size of the contour posture fine-grained fusion feature is C1×M, M=H1×W1; then each original image feature and a corresponding contour posture fine-grained fusion feature are input into the spatial feature guidance module, and the contour posture fine-grained fusion feature is used to guide the contour posture fusion auxiliary enhancement model to weaken the attention to the features related to pedestrian clothing in the original image features, and enhance the attention to the features unrelated to clothing, and obtain the corresponding contour guidance feature, r i The corresponding original image features and f i fuse The contour guidance feature obtained after SGM is f i g , the size of the contour-guided feature is C1×M; then the T contour-guided features corresponding to the training video sequence are subjected to the spatial maximum pooling operation respectively, and then the T results after the spatial maximum pooling operation are subjected to the temporal average pooling operation in the channel dimension to obtain the spatial representation feature f corresponding to the training video sequence s , spatial representation feature f s The size of is C1.

[0052] The two modalities have similar semantics in describing pedestrians, which is manifested in the spatial correspondence between the contour graphic structure and the joint point positions, such as Figure 5 As shown, based on this position correspondence, the present invention proposes an adaptive spatial feature fusion module to fuse the contour area with the corresponding joint area. Therefore, in the spatial branch, the adaptive spatial feature fusion module will use the spatial similarity semantics of the two modal information to fuse the contour map and the posture heat map to obtain complementary features in space, namely, the contour posture fine-grained fusion features; then the spatial feature guidance module uses the complementary features in space to guide the contour posture fusion auxiliary enhancement model to focus on the clothing-independent features of the original image features.

[0053] The preferred solution is, Figure 2 As shown in the figure, the adaptive spatial feature fusion module includes three layer normalization (layernorm, i.e. Figure 2 Norm shown in ), six MLPs ( Figure 2), two cross-attentions, and a feedforward network (FFN). The weights of the two cross-attentions are independent. Since the features representing the same part in the two modal information (i.e., the contour map and the posture heat map) are similar, the two modalities can be fine-grained fused with the help of the attention mechanism, that is, the features of different modalities at the same position can be effectively fused; the implementation process of the adaptive spatial feature fusion module is as follows: the feature of size C1×M obtained by merging the width and height of the contour map features is input into the first layer normalization, and the features output by the first layer normalization are respectively input into three of the MLPs. Following the Transformer, the three MLPs map the features output by the first layer normalization into three features with unchanged shapes, which are respectively used as the query information Q corresponding to the contour map features. s , key information K s , value information V s ; The feature of size C1×M obtained by merging the width and height of the posture heat map feature is input into the second layer normalization, and the features output by the second layer normalization are input into the other three MLPs respectively. Following the Transformer, these three MLPs map the features output by the second layer normalization into three features with unchanged shapes, which are respectively used as the query information Q corresponding to the posture heat map feature p , key information K p , value information V p ; The query information Q corresponding to the contour feature s And the key information K corresponding to the posture heat map feature p Sum value information V p Input into the first cross attention to obtain the first attention feature; the query information Q corresponding to the posture heat map feature p And the key information K corresponding to the contour feature s Sum value information V s Input it into the second cross attention to obtain the second attention feature; add the first attention feature to the feature of the normalized output of the first layer, add the second attention feature to the feature of the normalized output of the second layer, and then concatenate the two addition results in the channel dimension to obtain the feature input into the third layer normalization, input the feature of the normalized output of the third layer into the feedforward network, and add the feature output of the feedforward network to the feature of the normalized output of the third layer to obtain the corresponding contour posture fine-grained fusion feature.

[0054] Here, MLP (multi-layer perceptron) is the existing technology, and the default is a two-layer perceptron; cross attention is the existing technology; the feedforward network can generally use a linear layer.

[0055] The auxiliary modality in the existing ReID model is not robust enough to clothing. Considering that the contour map has rich appearance information, it can increase the discriminability between pedestrians, but due to the interference of clothing, it cannot effectively reduce the difference between the same pedestrian in different clothing. The posture heat map containing the information of human joints can reduce the interference of clothing, but due to the lack of information, it cannot effectively increase the discriminability between pedestrians, which means that the two modal information are complementary in space. At the same time, since the two modal information are descriptions of the same pedestrian, there is spatial similarity between them. In space, it is manifested as the correspondence between the position of the contour map and the posture heat map of the same part. Therefore, the present invention uses this spatial similarity semantics to fuse the two modal information to obtain a more robust modality to clothing. In order to fuse the two modal information, the present invention adopts two cross attentions to perform fine-grained fusion of spatially similar semantic features in the two modal spaces, that is, to interpretatively fuse the external shape (contour) and internal structure (joints) of the body (that is, to use similar semantics to fine-grainedly fuse the corresponding parts of the body in the two modal information).

[0056] In order to realize the adaptive spatial feature fusion module, the present invention designs an alternating cross attention structure. Specifically, the alternating cross attention structure is composed of two cross attentions, each cross attention uses one modal information to query and fuse similar semantics in another modal information, and the query modalities in the two cross attentions are different, so as to ensure the mutual interaction between the two modal information. Compared with the use of Hadamard product or addition, the alternating cross attention structure realizes more accurate feature fusion between the contour map and the posture heat map, realizes information interaction between the contour map and the posture heat map, establishes various spatial relationships between the contour map and the posture heat map, and realizes a more robust representation of clothing-independent features with the complementary advantages of the two modal information, and also provides a new way for multimodal fusion.

[0057] The original image features contain abundant clothing-irrelevant features, such as facial features, head shape features, body shape features, etc. of pedestrians, and these features only occupy a small area in the original image features, and there are more interference features such as the color and texture of pedestrians' clothing. At the same time, for the fine-grained fusion features of contour posture, it also contains the head shape features and body shape features of pedestrians, but does not contain interference features related to pedestrians' clothing. Therefore, in order to make the contour posture fusion auxiliary enhancement model pay more attention to the clothing-irrelevant features in the original image features and weaken the attention to the interference features, the present invention proposes a spatial feature guidance module based on cross-attention, which is based on the cross-attention mechanism and uses the contour posture fine-grained fusion features to guide the contour posture fusion auxiliary enhancement model to pay more attention to the clothing-irrelevant features in the original image features. The preferred scheme is, Figure 3 As shown in the figure, the spatial feature guidance module includes three layer normalizations (layernorm, i.e. Figure 3 Norm in), three MLPs ( Figure 3 ), a cross-attention, and a feed-forward network (FFN); the implementation process of the spatial feature guidance module is: the feature of size C1×M obtained by merging the width and height of the original image feature is input into the first layer normalization, and the feature output of the first layer normalization is input into the first MLP. Following the Transformer, the first MLP maps the feature output of the first layer normalization into a mapping feature with unchanged shape as the query information Q corresponding to the original image feature r ; Input the contour posture fine-grained fusion feature of size C1×M into the second layer normalization, and input the features of the second layer normalization output into the second and third MLPs respectively. Following the Transformer, the second MLP maps the features of the second layer normalization output into shape-invariant mapping features as the key information K corresponding to the contour posture fine-grained fusion feature fuse The third MLP maps the normalized output features of the second layer into shape-invariant mapping features as the value information V corresponding to the contour posture fine-grained fusion features fuse ; The query information Q corresponding to the original image features r And the key information K corresponding to the fine-grained fusion feature of the contour posture fuse Sum value information V fuse Input into the cross attention to obtain the attention feature; add the attention feature to the feature of the normalized output of the first layer, input the result of the addition into the normalization of the third layer, input the feature of the normalized output of the third layer into the feedforward network, and add the feature output of the feedforward network to the feature of the normalized output of the third layer to obtain the corresponding contour-guided feature.

[0058] The preferred solution is that the temporal branch is composed of an adaptive motion feature enhancement module (AMEM); after the T contour map features are subjected to a temporal average pooling operation in the channel dimension, they are subjected to a merging operation in the spatial dimension to obtain a first temporal feature, and the size of the first temporal feature is C1×M; after the T posture heat map features are subjected to a temporal average pooling operation in the channel dimension, they are subjected to a merging operation in the spatial dimension to obtain a second temporal feature, and the size of the second temporal feature is C1×M; then the first temporal feature and the second temporal feature are input together into the adaptive motion feature enhancement module, and the adaptive motion feature enhancement module enhances and fuses the two temporal features to obtain a temporal fusion enhancement feature corresponding to the training video sequence, and the size of the temporal fusion enhancement feature is C1×M1, where M1=2×M; then the temporal fusion enhancement feature is subjected to a spatial maximum pooling operation to obtain a pedestrian temporal motion feature f corresponding to the training video sequence t , such as gait, arm swing and other pedestrian temporal motion features, pedestrian temporal motion features f t The size of is C1.

[0059] The posture of the same pedestrian in the same video frame is consistent with that of the contour image and the posture heat map, so the temporal actions extracted from multiple video frames are also similar, that is, the two modalities have similar semantics in time, such as Figure 6 As shown, based on this similarity, the present invention proposes an adaptive action feature enhancement module to fuse similar action features between the two modalities. Therefore, in the temporal branch, the adaptive action feature enhancement module utilizes the similar semantics of the temporal actions of the two modalities to fuse and enhance the action features in the contour map and the posture heat map.

[0060] The preferred solution is, Figure 4 As shown in the figure, the adaptive motion feature enhancement module includes four layer normalizations (layernorm, i.e. Figure 4 Norm shown in ), six MLPs ( Figure 4), two cross-attentions, and two feedforward networks (FFNs), and the weights of the two cross-attentions are independent; the implementation process of the adaptive action feature enhancement module is: the first temporal feature is input into the first layer normalization, and the features of the first layer normalization output are respectively input into three MLPs, following the Transformer, and the three MLPs map the features of the first layer normalization output into three features with unchanged shapes, which are respectively used as the query information, key information, and value information corresponding to the first temporal feature; the second temporal feature is input into the second layer normalization, and the features of the second layer normalization output are respectively input into the other three MLPs, following the Transformer, and the three MLPs map the features of the second layer normalization output into three features with unchanged shapes, which are respectively used as the query information, key information, and value information corresponding to the second temporal feature; the query information corresponding to the first temporal feature and the key information and value information corresponding to the second temporal feature are input into the first cross In the attention, the first attention feature is obtained; the query information corresponding to the second temporal feature and the key information and value information corresponding to the first temporal feature are input into the second cross attention to obtain the second attention feature; the first attention feature is added to the feature of the first layer normalization output, the result of the addition is input into the third layer normalization, the feature of the third layer normalization output is input into the first feedforward network, the feature output of the first feedforward network is added to the feature output of the third layer normalization to obtain the first temporal enhancement feature; the second attention feature is added to the feature output of the second layer normalization, the result of the addition is input into the fourth layer normalization, the feature of the fourth layer normalization output is input into the second feedforward network, the feature output of the second feedforward network is added to the feature output of the fourth layer normalization to obtain the second temporal enhancement feature; the first temporal enhancement feature and the second temporal enhancement feature are concatenated (Concatenation) in the channel dimension to obtain the temporal fusion enhancement feature corresponding to the training video sequence.

[0061] As a video-based recognition task, video frames contain unique biological information, such as pedestrian gait information, arm movement information, etc. Since these are unique features of each person, it is also crucial to extract these dynamic information from the video. In order to better utilize the temporal action information contained in the contour map and the posture heat map, the present invention proposes an attention-based adaptive action feature enhancement module (AMEM) for fine-grained fusion of temporal features between two modalities. The adaptive action feature enhancement module also has two cross attentions, forming an alternating cross attention structure. Unlike the adaptive spatial feature fusion module (ASFM), the features after cross attention in AMEM first pass through their respective feedforward networks (FFN) and then perform concatenation operations on the channel dimension. This is because considering that temporal features are more abstract and complex than spatial features, two separate FFNs are used to map the temporal features of each modality respectively. AMEM uses the similar semantics of temporal actions of the two modalities and uses cross attention to align and fuse the same actions. Specifically, since both modalities describe the motion features of the same pedestrian, they are similar when describing the same motion features. For example, the motion trajectory of a pedestrian's feet is similar regardless of the modality. Therefore, with the help of this similarity, the temporal actions of the two modalities are fused to obtain more robust temporal features.

[0062] Step 3: Based on the training set, the contour posture fusion auxiliary enhancement model is trained, and after the training is completed, the trained contour posture fusion auxiliary enhancement model is obtained.

[0063] Further defined, the training process of step 3 is:

[0064] Step 3.1: Randomly divide all training video sequences in the training set into multiple batches, so that each batch contains batchsize training video sequences.

[0065] In actual processing, if the total number of training video sequences in the training set is divisible by batchsize, it is divided into Num batches, where Num = total number of training video sequences / batchsize; if the total number of training video sequences in the training set is not divisible by batchsize, the excess part is discarded to obtain int(Num) batches, where int() represents the rounding function.

[0066] Step 3.2: Take one of the batches in the training set, and use all the training video sequences in this batch as the input of the contour and pose fusion-assisted enhancement model. Input them into the contour and pose fusion-assisted enhancement model for training, and obtain the predicted ID probability vector corresponding to each training video sequence in this batch.

[0067] In the video dressing dataset, each pedestrian has a unique ID. Assuming that the video dressing dataset involves Number pedestrians, then each training video sequence corresponds to a real ID probability vector with a dimension of Number. If the index of the pedestrian in this training video sequence is b, b=1,2,…, Number, then the value of the element with index b in the real ID probability vector is 1, and the rest are 0; after training by the present invention, each training video sequence corresponds to a predicted ID probability vector with a dimension of Number, and the value of each element in the predicted ID probability vector is greater than or equal to 0 and less than or equal to 1. The index of the element with the largest value in the predicted ID probability vector is considered to be the index of the pedestrian in this training video sequence, thereby achieving the purpose of prediction.

[0068] Step 3.3: For each training video sequence in this batch, calculate the ID loss and triplet loss corresponding to the training video sequence, and then calculate the total loss corresponding to the training video sequence, which is the sum of the ID loss and the triplet loss; then use the total loss corresponding to all training video sequences in this batch to update the network parameters through back propagation.

[0069] Here, the ID loss corresponding to the training video sequence is obtained by calculating the cross entropy loss between the predicted ID probability vector and the true ID probability vector corresponding to the training video sequence; the triplet loss corresponding to the training video sequence is obtained based on the distance between the anchor point and the positive sample and the negative sample.

[0070] Step 3.4: Repeat the process of steps 3.2 and 3.3 until all batches of the training set have trained the contour pose fusion auxiliary enhancement model once.

[0071] Step 3.5: Repeat the process from step 3.1 to step 3.4, and end the training process after executing the preset rounds to obtain the trained contour posture fusion auxiliary enhancement model. Here, the preset rounds can be set to 300 rounds.

[0072] Step 4: arbitrarily select a pedestrian video sequence; then, based on the pedestrian video sequence, obtain a test video sequence containing T frames of pedestrian video frames; then input the test video sequence into the trained contour posture fusion auxiliary enhancement model to obtain the time-space fusion features corresponding to the test video sequence, and then achieve the purpose of pedestrian re-identification in the test video sequence based on the time-space fusion features. Here, the method of obtaining T frames of pedestrian video frames is the same as the method used in constructing the training set, that is, setting the number of video frames contained in the arbitrarily selected pedestrian video sequence to N, then divide this pedestrian video sequence into T groups according to the time axis, each group contains (NN%T) / T frames of video frames; then randomly extract a frame of video frames from each group; and then the T frames of video frames extracted from the T groups constitute the test video sequence.

[0073] In order to further illustrate the feasibility and effectiveness of the method of the present invention, a simulation experiment was carried out on the method of the present invention.

[0074] 1. Dataset and evaluation strategy

[0075] The CCVID dataset is a dressing dataset that only contains pedestrians' forward perspectives. The dataset contains 226 subjects, each of whom has 2 to 5 sets of clothing. Each dressing situation only contains one head-on perspective, and the shooting content is a video sequence of pedestrians in outdoor scenes. According to the training strategy given in the CCVID dataset, the video sequences of 75 subjects were selected for training in the simulation, and the video sequences of 151 subjects were used for testing. It is divided into two situations: comprehensive setting (General) and dressing setting (CC). The comprehensive setting is: there are scenes in which pedestrians change clothes and pedestrians do not change clothes in the probe and gallery; the dressing setting is that the pedestrians in the probe and gallery wear different clothes.

[0076] The CCPG-D dataset is an emerging dataset of dressing close to real scenes, proposed by Beijing Jiaotong University in 2023. The dataset contains 200 subjects, each of whom has seven dressing conditions, each of which contains 10 perspectives, and each perspective is shot by a camera set up at 2.7 meters or 3 meters. The shooting content is a video sequence of pedestrians in indoor and outdoor scenes, and the pedestrians' faces and shoes are erased. According to the training strategy given in the CCPG-D dataset, the video sequences of the first 100 subjects are selected as the training set in the simulation, and the video sequences of the last 100 subjects are selected as the test set. It is divided into three cases: CL-FULL, CL-UP, and CL-DN. CL-FULL: the clothing of pedestrians in the gallery and probe is completely different; CL-UP: the tops of pedestrians in the gallery and probe are different, but the pants are the same; CL-DN: the tops of pedestrians in the gallery and probe are the same, but the pants are different.

[0077] 2. Implementation details

[0078] The size of the standard original image, contour image, and pose heat map is 256×128. The CCPG-D dataset provides contour images but not pose heat maps; the CCVID dataset does not provide contour images and pose heat maps. For contour images, the pre-trained MaskRCNN network is used as the semantic segmentation model to extract the contour images of pedestrians from the video frames; for pose heat maps, the pre-trained HRNet network is used as the pose estimation model to extract the pose heat maps of pedestrians from the video frames. The pre-trained ResNet50 network is used for frame-level feature extraction of standard original images, contour images, and pose heat maps. When training the contour and pose fusion assisted enhancement model, the Adam optimizer is used, random erasing is used for data enhancement, and the batchsize is set to 8, that is, each batch involves 4 pedestrians, each pedestrian corresponds to 2 video sequences, and each video sequence contains 8 video frames. On the CCPG-D dataset, the learning rate was 0.0001, trained for 300 epochs (rounds), and the learning rate was divided by 10 after every 100 epochs; on the CCVID dataset, the learning rate was 0.00005, trained for 150 epochs, and the learning rate was divided by 10 after every 50 epochs.

[0079] 3. Experimental results

[0080] Table 1 shows the experimental results of Rank 1 accuracy and mAP under different conditions on the CCVID dataset. Since the CCVID dataset only contains the forward view of pedestrians, in order to further verify the generalization and robustness of the proposed method in multi-view scenarios, it is evaluated on a dataset CCPG-D containing multiple views of pedestrians. Table 2 shows the experimental results of Rank 1 accuracy and mAP under different conditions on the CCPG-D dataset.

[0081] Table 1 Experimental results of Rank 1 accuracy and mAP under different conditions on the CCVID dataset

[0082]

[0083] Table 2 Experimental results of Rank1 accuracy and mAP under different conditions on CCPG-D dataset

[0084]

[0085] In Table 1, the AP3D method is derived from Xinqian Gu, Hong Chang, Bingpeng Ma, Hongkai Zhang, and Xilin Chen. Appearance-preserving 3D convolution for video-based person re-identification. In European Confer-ence on Computer Vision, pages 228–243. Springer, 2020. (Appearance-preserving 3D convolution for video pedestrian re-identification, European Computer Vision Conference), the TCLNet method is derived from Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Temporal complementary learning for video person re-identification. In ECCV, 2020. (Temporal complementary learning for video pedestrian re-identification, European Computer Vision Conference), and the CAL method is derived from Gu, Xinqian, et al. "Clothes-changing person re-identification with rgb modality only." Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2022. (Re-identification of people with clothing changes using only RGB, Computer Vision and Pattern Recognition), SEMI method comes from Temporal 3D Shape Modeling for Video-based Cloth-Changing Person Re-Identification (Temporal 3D Shape Modeling for Video-based Cloth-Changing Person Re-Identification, Winter Conference on Computer Vision Applications), CAL+3DinvarReID method comes from Liu, Feng, et al. "Learning clothing and pose invariant 3D shape representation for long-term person re-identification." Proceedings of the IEEE / CVF International Conference on Computer Vision. 2023. (Learning Clothing- and Pose-Invariant 3D Shape Representations for Long-Term Person Re-identification, International Conference on Computer Vision), DCR-ReID method is derived from Zhenyu Cui, Jiahuan Zhou, Yuxin Peng, Shiliang Zhang, and Yaowei Wang. Dcr-reid: Deep component reconstruction for cloth-changing person re-identification. IEEE TCSVT, 2023. (Journal of Circuits and Systems for Video Technology).

[0086] As can be seen from Table 1, the method of the present invention is compared with some advanced methods on the most popular video dressing dataset CCVID. Experiments show that the method of the present invention achieves the best results in both General and CC conditions. Compared with the DCR-ReID method, the Rank 1 and mAP of the method of the present invention are 1.8% higher under the comprehensive condition, and the Rank 1 is improved by 2.5% and the mAP is improved by 2.1% under the dressing condition. Thanks to the method of obtaining more robust features by fusing the contour map with the posture heat map, the method of the present invention is better than the DCR-ReID method using only the contour map in both cases.

[0087] In Table 2, the BiCnet-TKS method is from Ruibing Hou, Hong Chang, Bingpeng Ma, RuiHuang, and Shiguang Shan. Bicnet-tks: Learning efficient spatial-temporal representation for video person re-identification. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 2014–2023, 2021. (Learning effective spatial-temporal representation for video pedestrian re-identification, Computer Vision and Pattern Recognition Conference), the PSTA method is from Yingquan Wang, Pingping Zhang, Shang Gao, Xia Geng, Hu Lu, and Dong Wang. Pyramid spatial-temporal aggregation for video-based person re-identification. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 12026–12035, 2021. (Pyramid-shaped spatial-temporal aggregation for video pedestrian re-identification, International Conference on Computer Vision), and the PiT method is from Xianghao Zang, Ge Li, and Wei Gao.Multi-direction and multi-scale pyramid in transformer for video-basedpedestrian retrieval. IEEE Transactions on Industrial Informatics, 2022.

[0088] As can be seen from Table 2, the method of the present invention can still achieve an optimal result on the CCPG-D dataset. Since the CCPG-D dataset is a new dataset, the present invention is the first CC-ReID method to be tested on this dataset. Therefore, the open source model CAL is reproduced on this dataset for comparison. The rest are the reproduction results provided by the author of the CCPG-D dataset. Table 2 shows the accuracy of these methods. It can be seen that compared with the traditional video-based ReID method PSTA, the mAP of the present invention is improved by 4.3% and Rank 1 is improved by 11.7% in the case of CL-FULL; in the case of CL-UP, the mAP is improved by 2.8% and Rank 1 is improved by 6.4%; in the case of CL-DN, the mAP is improved by 8.9% and Rank 1 is improved by 10.1%. Compared with the VCC-ReID method CAL, the mAP of the present invention is improved by 2.2% and Rank 1 is improved by 4.2% in the case of CL-FULL; in the case of CL-UP, the mAP is improved by 1.4% and Rank 1 is improved by 2.3%; in the case of CL-DN, the mAP is improved by 0.8% and Rank 1 is improved by 2.7%. It can be seen that even in complex scenarios, the method of the present invention has better accuracy.

[0089] 4. Ablation experiment

[0090] To verify the effectiveness of each module, a series of ablation experiments are conducted on the more complex CCPG-D dataset.

[0091] Validity of time and space branches: In order to verify the effectiveness of each time and space branch, ablation experiments are conducted on the time and space branches, and the results are shown in Table 3. It can be observed from Table 3 that no matter whether the modules contained in the temporal branch or the spatial branch are removed, the model accuracy decreases, which shows that the clothing-independent features contained in the pedestrian space and temporal branch are very important and helpful for pedestrian recognition.

[0092] Table 3 Ablation test results on the CCPG-D dataset to analyze the impact of each branch on the model

[0093]

[0094] Analysis of data from different modalities: To verify the impact of different modalities on the model, the effects of the model in each modality were compared, as shown in Table 4. By comparing Groups A and B, it can be seen that the pedestrian information contained in the contour map can promote the model's recognition of pedestrians. However, the accuracy of Group B is still lower than that of Group D, proving that the fusion of posture heatmap and contour map can provide more robust clothing-independent features for pedestrians. By comparing the results of Group C with Group A, it was found that adding only posture heatmaps did not improve the recognition of pedestrians much. It is believed that this is because it is difficult to distinguish pedestrians based on the joint point information provided in the posture heatmap alone, so it is not very helpful for pedestrian recognition.

[0095] Table 4 Ablation test on CCPG-D dataset to analyze the impact of each modality on the model

[0096]

[0097] The impact of frame rate on the model: In order to verify the impact of frame rate on the model, the effects of the model at different frame rates were compared, such as Figure 7 As shown. Figure 7 It can be seen that the recognition accuracy of the model increases with the increase of the number of frames, which proves that the number of frames has a certain impact on the model. Considering that the video depicts the pedestrian's movements more delicately due to the increase in the number of frames, the movement features captured by the model are more discriminative, resulting in a continuous increase in recognition accuracy. After the number of video frames exceeds 33 frames, the performance of the model tends to be stable. It is believed that this is because 33 frames have basically covered the complete movement cycle of pedestrians, and the description of pedestrians' movements is already delicate enough, so the improvement brought by continuing to increase the number of frames is already very limited.

[0098] Effectiveness of alternating cross attention: In order to verify the effectiveness of the alternating cross attention fusion method proposed in the present invention, various commonly used fusion methods are compared with it, as shown in Table 5. In Table 5, each row represents the results obtained using the corresponding multimodal fusion method in ASFM and AMEM. From the ablation experiment, it can be observed that the fusion methods using addition or Hadamard product are not as good as the alternating cross attention proposed in the present invention. Since each modality uses an independent feature extractor, i.e., an encoder, there are different situations in the positions of the features describing the same part, so the positions of similar semantic features between the two modalities are flexible. Simple fusion can only fuse features at the same position, so this flexibility cannot be handled by addition and Hadamard product methods. Alternating cross attention is based on the similarity between different features for fusion, and is not affected by the position of the features. Therefore, this flexibility problem can be overcome and accurate fusion of similar semantic features between the two modalities can be achieved.

[0099] Table 5 Ablation test on CCPG-D dataset to analyze the impact of different fusion methods on the model

[0100]

[0101] 5. Visualization

[0102] In order to study the distribution of features learned by the model, a t-SNE visualization experiment was performed on the CCPG-D test set to compare the spatial distribution of features proposed by the baseline and the model of the present invention. Figure 8 As shown in the figure. For the Baseline, it can be observed that the features within each category are discretized, and the features of different categories are mixed together. In contrast, the features learned by the model of the present invention are more compact within the category, and the gap between categories is more obvious, which shows that the method of the present invention can more effectively learn the clothing-independent features of pedestrians and distinguish pedestrians.

[0103] The above experiments are sufficient to illustrate the feasibility and effectiveness of the method of the present invention.

Claims

1. A video dressing-changing pedestrian re-identification method with a robust auxiliary modality is characterized by The following steps are involved: Step 1: Select a video dressing dataset; then construct a training set based on the video dressing dataset. The training set contains multiple training video sequences, and each training video sequence contains T frames of video frames with the same person; Step 2: Build a contour posture fusion auxiliary enhancement model, including preprocessing stage, encoding stage, and pedestrian clothing-independent feature mining stage; In the preprocessing stage, each video frame in the training video sequence is scaled into a standard original image of uniform size through an image scaling module, each video frame in the training video sequence is segmented through a semantic segmentation model to obtain a corresponding contour map, and each video frame in the training video sequence is passed through a posture estimation model to obtain a corresponding posture heat map; In the encoding stage, each standard original image is input into the first encoder to obtain the corresponding original image features; each contour image is input into the second encoder to obtain the corresponding contour image features; each posture heat map is input into the third encoder to obtain the corresponding posture heat map features; wherein the size of the original image features, contour image features, and posture heat map features are all C1×H1×W1, C1 is the number of channels, H1 is the height, and W1 is the width; In the stage of mining pedestrian clothing-independent features, T original image features, T contour features and T posture heat map features are input into the spatial branch used to extract pedestrian clothing-independent features at the spatial level, and the spatial representation features corresponding to the training video sequence are obtained; T contour features and T posture heat map features are input into the temporal branch used to extract pedestrian clothing-independent features at the temporal action level of pedestrians, and the pedestrian temporal action features corresponding to the training video sequence are obtained; the spatial representation features corresponding to the training video sequence and the pedestrian temporal action features are concatenated to obtain the temporal and spatial fusion features corresponding to the training video sequence; the temporal and spatial fusion features corresponding to the training video sequence are input into the fully connected layer to obtain the predicted ID probability vector corresponding to the training video sequence; The spatial branch consists of an adaptive spatial feature fusion module and a spatial feature guidance module; each contour map feature and a corresponding posture heat map feature are input into the adaptive spatial feature fusion module together, and the contour map feature and the posture heat map feature are complementary fused to obtain the corresponding contour posture fine-grained fusion feature, and the size of the contour posture fine-grained fusion feature is C1×M, M=H1×W1; then each original image feature and a corresponding contour posture fine-grained fusion feature are input into the spatial feature guidance module together to obtain the corresponding contour guidance feature, and the size of the contour guidance feature is C1×M; then the T contour guidance features corresponding to the training video sequence are respectively subjected to the spatial maximum pooling operation, and then the T results after the spatial maximum pooling operation are subjected to the time average pooling operation in the channel dimension to obtain the spatial representation feature corresponding to the training video sequence, and the size of the spatial representation feature is C1; The adaptive spatial feature fusion module includes three layers of normalization, six MLPs, two cross attentions, and a feedforward network, and the weights of the two cross attentions are independent; the implementation process of the adaptive spatial feature fusion module is: the features obtained by merging the width and height of the contour map features are input into the first layer normalization, and the features output by the first layer normalization are respectively input into three of the MLPs, and these three MLPs map the features output by the first layer normalization into three features with unchanged shapes, which are respectively used as query information, key information, and value information corresponding to the contour map features; the features obtained by merging the width and height of the posture heat map features are input into the second layer normalization, and the features output by the second layer normalization are respectively input into the other three MLPs, and these three MLPs map the features output by the second layer normalization into three features with unchanged shapes, which are respectively used as The query information, key information, and value information corresponding to the posture heat map feature; the query information corresponding to the contour map feature and the key information and value information corresponding to the posture heat map feature are input into the first cross attention to obtain the first attention feature; the query information corresponding to the posture heat map feature and the key information and value information corresponding to the contour map feature are input into the second cross attention to obtain the second attention feature; the first attention feature is added to the feature of the first layer normalization output, and the second attention feature is added to the feature of the second layer normalization output, and then the feature obtained by splicing the two addition results in the channel dimension is input into the third layer normalization, and the feature of the third layer normalization output is input into the feedforward network, and the feature output by the feedforward network is added to the feature of the third layer normalization output to obtain the corresponding contour posture fine-grained fusion feature; The spatial feature guidance module includes three layers of normalization, three MLPs, a cross attention, and a feedforward network; the implementation process of the spatial feature guidance module is: the features obtained by merging the width and height of the original image features are input into the first layer of normalization, the features output by the first layer of normalization are input into the first MLP, and the first MLP maps the features output by the first layer of normalization into shape-invariant mapping features as query information corresponding to the original image features; the contour posture fine-grained fusion features are input into the second layer of normalization, the features output by the second layer of normalization are input into the second and third MLPs respectively, and the second MLP maps the features output by the second layer of normalization into shape-invariant mapping features. The third MLP maps the features of the second layer normalization output into shape-invariant mapping features as the key information corresponding to the contour posture fine-grained fusion features. The query information corresponding to the original image features and the key information and value information corresponding to the contour posture fine-grained fusion features are input into the cross attention to obtain the attention features. The attention features are added to the features of the first layer normalization output, and the result of the addition is input into the third layer normalization. The features of the third layer normalization output are input into the feedforward network, and the features output by the feedforward network are added to the features output by the third layer normalization to obtain the corresponding contour guidance features. The temporal branch is composed of an adaptive action feature enhancement module; after the T contour map features are subjected to a temporal average pooling operation in the channel dimension, a first temporal feature is obtained after a merging operation in the spatial dimension, and the size of the first temporal feature is C1×M; after the T posture heat map features are subjected to a temporal average pooling operation in the channel dimension, a second temporal feature is obtained after a merging operation in the spatial dimension, and the size of the second temporal feature is C1×M; then the first temporal feature and the second temporal feature are input together into the adaptive action feature enhancement module to obtain a temporal fusion enhancement feature corresponding to the training video sequence, and the size of the temporal fusion enhancement feature is C1×M1, where M1=2×M; then the temporal fusion enhancement feature is subjected to a spatial maximum pooling operation to obtain a pedestrian temporal action feature corresponding to the training video sequence, and the size of the pedestrian temporal action feature is C1; The adaptive action feature enhancement module includes four layers of normalization, six MLPs, two cross attentions, and two feedforward networks, and the weights of the two cross attentions are independent; the implementation process of the adaptive action feature enhancement module is: inputting the first temporal feature into the first layer of normalization, inputting the features output by the first layer of normalization into three MLPs respectively, and the three MLPs map the features output by the first layer of normalization into three features with unchanged shapes, which are respectively used as the query information, key information, and value information corresponding to the first temporal feature; inputting the second temporal feature into the second layer of normalization, and inputting the features output by the second layer of normalization into another three MLPs respectively, and the three MLPs map the features output by the second layer of normalization into three features with unchanged shapes, which are respectively used as the query information, key information, and value information corresponding to the second temporal feature; inputting the query information corresponding to the first temporal feature and the key information and value information corresponding to the second temporal feature into the first cross attention, and obtaining The first attention feature is obtained; the query information corresponding to the second temporal feature and the key information and value information corresponding to the first temporal feature are input into the second cross attention to obtain the second attention feature; the first attention feature is added to the feature output by the first layer normalization, the result of the addition is input into the third layer normalization, the feature output by the third layer normalization is input into the first feedforward network, the feature output by the first feedforward network is added to the feature output by the third layer normalization to obtain the first temporal enhancement feature; the second attention feature is added to the feature output by the second layer normalization, the result of the addition is input into the fourth layer normalization, the feature output by the fourth layer normalization is input into the second feedforward network, the feature output by the second feedforward network is added to the feature output by the fourth layer normalization to obtain the second temporal enhancement feature; the first temporal enhancement feature and the second temporal enhancement feature are concatenated in the channel dimension to obtain the temporal fusion enhancement feature corresponding to the training video sequence; Step 3: Based on the training set, the contour posture fusion auxiliary enhancement model is trained, and after the training, the trained contour posture fusion auxiliary enhancement model is obtained; Step 4: Randomly select a pedestrian video sequence; then, based on the pedestrian video sequence, obtain a test video sequence containing T frames of pedestrian video frames; then input the test video sequence into the trained contour and posture fusion auxiliary enhancement model to obtain the temporal and spatial fusion features corresponding to the test video sequence, and then achieve the purpose of pedestrian re-identification in the test video sequence based on the temporal and spatial fusion features.

2. According to claim 1, a method for constructing a robust auxiliary modality for video dressing pedestrian re-identification is characterized in that In step 1, the construction process of the training set is as follows: for any video sequence in the video dressing dataset, the number of video frames contained in the video sequence is set to N, and the video sequence is divided into T groups according to the time axis, each group contains (NN%T) / T video frames, where % is the modulo operator; then a video frame is randomly extracted from each group; and the T video frames extracted from the T groups constitute a training video sequence; each video sequence in the video dressing dataset is processed in the same way to obtain the corresponding training video sequence, and then the training set is obtained.

3. The video dressing-changing pedestrian re-identification method with robust auxiliary modality constructed according to claim 1 is characterized in that In step 2, the sizes of the standard original image, contour image, and posture heat map are all C×H×W, where C is the number of channels, H is the height, and W is the width.

4. A method for constructing a robust auxiliary modality for video dressing-changing pedestrian re-identification according to claim 1 or 3, characterized in that In step 2, the first encoder, the second encoder, and the third encoder are three ResNet50 networks with independent weights.

5. The video dressing-changing pedestrian re-identification method with robust auxiliary modality constructed according to claim 1 is characterized in that The training process of step 3 is: Step 3.1: Randomly divide all training video sequences in the training set into multiple batches, so that each batch contains batchsize training video sequences; Step 3.2: Take one of the batches in the training set, and use all the training video sequences in this batch as the input of the contour and pose fusion-assisted enhancement model. Input them into the contour and pose fusion-assisted enhancement model for training, and obtain the predicted ID probability vector corresponding to each training video sequence in this batch. Step 3.3: For each training video sequence in this batch, calculate the ID loss and triplet loss corresponding to the training video sequence, and then calculate the total loss corresponding to the training video sequence, which is the sum of the ID loss and the triplet loss; then use the total loss corresponding to all training video sequences in this batch to update the network parameters through back propagation; Step 3.4: Repeat the process of step 3.2 and step 3.3 until all batches of the training set have trained the contour pose fusion auxiliary enhancement model once; Step 3.5: Repeat the process from step 3.1 to step 3.4, and end the training process after executing the preset rounds to obtain the trained contour posture fusion auxiliary enhancement model.

Images