Multi-modal fusion clothes changing pedestrian re-identification method under multi-angle intelligent monitoring scene

Abstract

The invention discloses a multi-modal fusion clothes changing pedestrian re-identification method in a multi-angle intelligent monitoring scene, and the method comprises the steps: extracting the skeleton features and global image features of an original pedestrian image, carrying out the reinforcement of the image head features through employing the skeleton features under the condition of not introducing an extra face model, thereby obtaining a mask between the skeleton head feature # imgabs0 # and the global image feature Fa, and carrying out the recognition of the image head feature # imgabs0 # and the global image feature Fa. Then, the mask is multiplied by the global image features to generate weighted features, weighted averaging of the global image features Fa is achieved, the global image features Fa can pay more attention to the head area while maintaining the image features, finally, the weighted features and the global image features Fa are fused, and the head area is obtained. The method comprises the following steps of: obtaining an enhanced global image feature # imgabs1 # guided by a skeleton head, carrying out alignment reconstruction on the enhanced global image feature # imgabs2 # and a feature fs obtained by carrying out self-attention coding on a skeleton feature Fk, and effectively extracting appearance and body structure information. And then, a pedestrian target object to be searched is quickly matched.

Description

Technical Field

[0001] The present invention relates to a pedestrian recognition method, in particular to a multi-modal fusion clothing-changing pedestrian re-recognition method in a multi-angle intelligent monitoring scenario, belonging to the technical field of pedestrian re-recognition after clothing change. Background Art

[0002] Recognizing the same person in different cameras or different scenes is a basic but important task in security monitoring projects. Pedestrian re-identification aims to detect the location of a specific target in different cameras, identify its identity, and then track it. The pedestrian re-identification system automatically analyzes the identity information of pedestrians in video streams in real time. Under the condition that most faces are not clear in actual real-life scenes, it can track and locate specific people across time and space in a timely manner. It plays an important role in security monitoring, intelligent transportation, and smart cities.

[0003] Most current studies are based on the assumption that pedestrians appear under different cameras in a short period of time, so that pedestrians' clothing does not change in different camera ranges. Based on this assumption, short-term pedestrian re-identification has been widely studied. However, short-term pedestrian re-identification does not take into account that in real life scenarios, pedestrians may change their appearance significantly due to changing clothes. Therefore, pedestrian re-identification that focuses more on the scene of changing clothes is gaining more and more attention.

[0004] In order to extract robust features that are independent of clothing. Some existing methods introduce face recognition models to obtain pedestrian facial features. However, this method requires an additional face recognition model and is aimed at scenes where the face is clearly visible, which brings great limitations to practical applications. Other research methods focus on extracting features that are independent of clothing from auxiliary modes (contours, gait, skeletons, 3D shapes), but these methods do not take into account the huge differences between RGB features and other modal features. They are only relatively simple interactions and cannot make full use of rich multimodal information. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-modal fusion clothing-changing pedestrian re-identification method in a multi-angle intelligent monitoring scenario, which can extract features of pedestrians under multiple cameras that are unrelated to the clothes they wear, and then quickly match the pedestrian features under multiple cameras to extract the pedestrian target object to be searched.

[0006] In order to achieve the above object, the present invention provides a multi-modal fusion clothing-changing pedestrian re-identification method in a multi-angle intelligent monitoring scenario, comprising the following steps:

[0007] S1: Collect image data. Use monitoring equipment to collect image data. The image data must contain pedestrian images, which are used as input image data I. At the same time, P*M images are used as batch input, where P represents the pedestrian category and M represents the number of images corresponding to each pedestrian in different environments by multiple cameras.

[0008] Assume the input image data is:

[0009] I={I0,…,I i , I P×M}

[0010] Where: {0,…,i,P×M} represents the image sequence number;

[0011] S2: Extract global image features F from pedestrian images through image feature extraction network a , extract bone features F through the bone feature extraction network k ,HRNet outputs in the form of bone heatmap;

[0012] S3: Skeleton head enhancement module, using the head area of ​​the skeleton heat map extracted by the skeleton feature extraction network in step S2 as a mask to enhance the global image feature F a Head features;

[0013] S4: Enhanced global image features after skeleton head enhancement and the skeletal features F k After the self-attention encoded feature fs, the aligned and reconstructed image feature F′ is obtained. G and bone features F′ K ;

[0014] S5: For the image feature F′ after alignment and reconstruction G and bone features F′ K Extract the image feature prototypes C respectively p and the skeletal feature prototype K p , image feature prototype C p and the skeletal feature prototype K p Then align and reconstruct the image features F′ G and bone features F′ K Make a loss, thereby shortening the feature distance between the same person, through the image feature prototype C p The learned image features are expressed as f a Indicates that after the skeleton feature prototype K p The learned skeleton features are expressed as f s express;

[0015] S6: For the image feature f obtained in step S5 abovea and bone features f s Then use the cross attention mechanism to get the integrated fusion feature F f , and the fusion feature F f As matching features for clothing change re-identification.

[0016] Step S2 of the present invention is specifically as follows: the pedestrian image data is passed into two branches, wherein the first branch extracts the global image feature F from the pedestrian image through the image appearance branch network a , using ResNet50 as the backbone, the stride of the first convolutional layer in the res4 block is set to 1 to improve the feature resolution; the second branch is to extract the skeleton feature F through the skeleton feature extraction network HRNet of the body structure feature branch k ,HRNet outputs in the form of a skeleton heat map to indicate the location of key points, including 17 key points, representing 17 key areas of the human body.

[0017] Step S3 of the present invention is specifically:

[0018] For the extracted bone feature F k Separate the head skeleton features. The output dimension of HRNet is [B, C, H, W], where B represents the batch and C represents the number of channels, which also represents the 17 key areas of the human body. The 5 key point features representing the head can be extracted as the skeleton head features. The specific implementation only needs to extract the first 5 dimensions of the C dimension representing the channel as the skeleton output. For the separated skeleton head features containing 5 head key points Use a 1x1 convolutional layer for dimensionality reduction and reshaping, and then calculate the skeleton head features through matrix multiplication and the global image feature F a The correlation between them is normalized by the Softmax layer so that the sum of the weights of each row is 1. The global maximum pooling layer is used to extract the maximum weight and reduce the computational complexity to obtain the skeleton head features. and the global image feature F a The weight coefficient between them is the similarity mask, and then the similarity mask is combined with the global image feature F a Multiply to generate weighted feature mask*F a , to achieve the global image feature F a The weighted average of the global image feature F a While maintaining the image features, it can pay more attention to the head area, and finally combine the weighted features and the global image features F a Fusion, to obtain enhanced global image features guided by the skeleton head For the bone feature F kUse a self-attention layer to encode the output, aiming to capture the relationship between different skeleton key points;

[0019] The enhanced global image features guided by the skeleton head are obtained in step S3 of the present invention. The formula is as follows:

[0020]

[0021] Where: C1, C2 represent 1x1 convolution layers with different convolution kernels;

[0022] Indicates the features of the skeletal head after separation;

[0023] F a Represents global image features;

[0024] GMP represents the global maximum pooling layer;

[0025] S represents the calculated skeletal head features and the global image feature F a Similarity mask mask;

[0026] Represents enhanced global image features.

[0027] Step S4 of the present invention is specifically as follows:

[0028] Will enhance the global image features The skeleton feature fs after self-attention encoding is first obtained through the fully connected layer to obtain the features of the same dimension, and then the embedding layer is used to map it to a deeper feature space. Then, the skeleton features of the deep layer (embedding layer output) and the image features of the shallow layer (fully connected layer output) are aligned by minimizing the maximum mean difference loss MMD. Specifically, it is only necessary to calculate the distance between the deep skeleton features and the shallow image features, and to achieve alignment by minimizing the distance through training, so as to capture the features unrelated to clothes through the image while filtering out the mismatched redundant and interference information. The embedding layer uses the FC fully connected layer to map the high-dimensional features to the low-dimensional space;

[0029] In order to restore the information loss caused by forced alignment, the present invention further performs penalized reconstruction through a reconstruction layer and a recon loss function to ensure that the aligned features retain their own features. The reconstruction layer uses a FC fully connected layer to reconstruct the original output features and minimize the maximum mean difference loss MMD and the recon loss function formula as follows:

[0030]

[0031] Where: u(·) and σ(·) represent the mean and variance of the transformed features;

[0032] L MMD Represents minimizing the maximum mean difference loss MMD;

[0033] g represents the features before the reconstruction layer;

[0034] Represents the features of the reconstructed layer;

[0035] L recon Represents the recon loss.

[0036] Step S5 of the present invention is specifically:

[0037] Based on the image feature prototype method, for each batch of P×M image data, the image features S of the same person are extracted using the method of steps S1-S4 above. img ={x1,…,x m}, the image features of the same person are averaged to construct the prototype centroid, and the image prototype centroid C of P people is obtained. P ={c1,…,c p}, the prototypes in each batch can capture more intrinsic features of the same person from the perspective of a larger data set, and have better robustness to clothing changes. Then, the constraints on the features are implemented by minimizing the distance between the image instances and the corresponding image prototypes of the same person. In the process of implementing the constraints, the inherent clothing-independent features of the same person are learned by narrowing the gap between the image instances and the image prototypes.

[0038] The formula for obtaining the image prototype (average method) and the distance formula (the distance between the instance and the corresponding prototype) are:

[0039]

[0040] Where: x i Represents the image features extracted from each image, that is, the image features F' obtained after step S4 G ;

[0041] M represents the M images of each person, which means the M features of each person obtained later;

[0042] represents the features of the i-th image instance of the p-th person;

[0043] p means different people;

[0044] c p Represents the image prototype of the pth person;

[0045] Based on the method of skeleton feature prototype, for each batch of P×M image data, the image features S of the same person are extracted using the method of steps S1-S4 above. k ={y1,…,y m}, the same person’s skeletal features are averaged to construct the prototype centroid, and the image prototype centroid K of P people is obtained. P ={k1, ..., k p}, then the constraints on features are achieved by minimizing the distance between the bone instance and the corresponding bone prototype of the same person. In the process of implementing the constraints, by narrowing the gap between the bone instance and the bone prototype, we can learn the inherent clothing-independent characteristics of the same person;

[0046] The formula for obtaining the skeleton prototype (average method) and the distance formula (the distance between the instance and the corresponding prototype) are:

[0047]

[0048] Where: y i Represents the skeleton features extracted from each image, that is, the skeleton features F' obtained after step S4 K ;

[0049] M represents the M images of each person, which means the M features of each person obtained later;

[0050] Represents the features of the i-th skeleton instance of the p-th person;

[0051] p means different people;

[0052] k p Represents the skeletal prototype of the pth person.

[0053] Step S6 of the present invention is specifically:

[0054] The attention cross module of step S6 takes the image feature and the bone feature as an example. First, the multi-head cross attention mechanism is used to interact by calculating the dot product similarity between the image feature and the bone feature. The similarity matrix is ​​normalized by scaling and softmax, and the result is regarded as the attention weight to perform the weighted sum of the features. In this way, for each image feature, body bone information with similar response can be found in it; specifically, the image feature is taken as Q, the bone feature is represented by K and V, and then the image feature f integrated with the bone feature is calculated by the attention module s→a ; Perform corresponding calculations on the bone-image features on the other side to obtain f a→s ,Finally, the interacted skeleton features and image features are concatenated as the final robust and discriminative features.

[0055] The image feature f that integrates the skeleton features is calculated through the attention module s→a The formula is:

[0056]

[0057] Where: represents the normalization layer;

[0058] The skeleton feature f is calculated by the attention module to integrate the image features a→s The formula is:

[0059]

[0060] Compared with the prior art, the present invention proposes a method of re-identification of the dress-changing pedestrians based on the fusion of multi-modal features of bone points and RGB in intelligent monitoring multi-angle scenarios. By extracting the bone characteristics and global image features of the original pedestrian image, the bone head features are strengthened through the extracted bone features without introducing an additional face model, thereby obtaining the bone head features. and the global image feature F a The weight coefficient between them is the similarity mask, and then the similarity mask is combined with the global image feature F a Multiply to generate weighted feature mask*F a , to achieve the global image feature F a The weighted average of the global image feature F a While maintaining the image features, it can pay more attention to the head area, and finally combine the weighted features and the global image features F a Fusion, to obtain enhanced global image features guided by the skeleton head Enhanced global image features after bone head enhancement and the skeletal features F k The feature fs after self-attention encoding is aligned and reconstructed, and the appearance and body structure information is effectively extracted. By aligning the features of each instance with its prototype, the distance between the same pedestrians is shortened, and the differences between different people are magnified. The appearance features are refined with the help of body structure features that are independent of clothes, and the body structure features supplement the aligned appearance features that are robust to clothing changes. Through the above method, the present invention can extract pedestrians and their human features that are independent of clothes in multiple cameras and multiple scenes under the monitoring of multiple camera scenes, and then by matching the re-identification features of pedestrians changing clothes extracted from multiple cameras, the pedestrian target object to be searched can be quickly matched under the conditions of multiple cameras, environmental changes and clothing changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1A flow chart of the method of the invention;

[0062] Figure 2 This is a structural diagram of the present invention for enhancing image features based on skeletal head features;

[0063] Figure 3 It is a structural diagram of the alignment and reconstruction of the skeleton features and the RGB image features of the present invention;

[0064] Figure 4 A schematic diagram of a prototype module for extracting image features according to the present invention;

[0065] Figure 5 It is a framework diagram of the cross attention module of the present invention;

[0066] Figure 6 This is a diagram of the re-identification framework for pedestrians who have changed clothes in actual monitoring application scenarios. DETAILED DESCRIPTION

[0067] The present invention will be further described below in conjunction with the accompanying drawings.

[0068] like Figure 1-Figure 5 As shown, a multi-modal fusion clothing-changing pedestrian re-identification method in a multi-angle intelligent monitoring scenario includes the following steps:

[0069] S1: Collect image data. The image data is collected through intelligent monitoring devices at multiple different angles. The image data must contain pedestrian images, which are used as input image data I. In order to better aggregate the characteristic attributes of the same person and separate the characteristics between different people, P*M images are used as batch input, where P represents the pedestrian category and M represents the number of images corresponding to each pedestrian.

[0070] Assume the input image data I is:

[0071] I={I0,…,I i , I P×M}

[0072] Where: {0,…,i,P×M} represents the image sequence number;

[0073] S2: Extract global image features F from pedestrian images through image feature extraction network a , extract bone features F through the bone feature extraction network k ,HRNet outputs in the form of bone heatmap;

[0074] Step S2 is as follows: the pedestrian image data is passed into two branches, wherein the first branch extracts the global image features F from the pedestrian image through the image appearance branch network. a, ResNet50 (deep convolutional neural network) is used as the backbone, and the stride of the first convolution layer in the res4 block is set to 1 to improve feature resolution; the second branch is to extract bone features F through the bone feature extraction network HRNet of the body structure feature branch k ,HRNet outputs in the form of a skeleton heat map to indicate the location of key points, including 17 key points, representing 17 key areas of the human body.

[0075] S3: Skeleton head enhancement module, using the head area of ​​the skeleton heat map extracted by the skeleton feature extraction network in step S2 as a mask to enhance the global image feature F a Head features;

[0076] Step S3 is specifically as follows:

[0077] For the extracted bone feature F k Separate the head skeleton features. The output dimension of HRNet is [B, C, H, W], where B represents the batch and C represents the number of channels, which also represents the 17 key areas of the human body. The 5 key point features representing the head can be extracted as the skeleton head features. The specific implementation only needs to extract the first 5 dimensions of the C dimension representing the channel as the skeleton output. For the separated skeleton head features containing 5 head key points Use a 1x1 convolutional layer for dimensionality reduction and reshaping, and then calculate the skeleton head features through matrix multiplication and the global image feature F a The correlation between them is normalized by the Softmax layer so that the sum of the weights of each row is 1. The global maximum pooling layer is used to extract the maximum weight and reduce the computational complexity to obtain the skeleton head features. and the global image feature F a The weight coefficient between them is the similarity mask, and then the similarity mask is combined with the global image feature F a Multiply to generate weighted feature mask*F a , to achieve the global image feature F a The weighted average of the global image feature F a While maintaining the image features, it can pay more attention to the head area, and finally combine the weighted features and the global image features F a Fusion, to obtain enhanced global image features guided by the skeleton head For the bone feature F k Use a self-attention layer to encode the output, aiming to capture the relationship between different skeleton key points;

[0078] The enhanced global image features guided by the skeleton head are obtained The formula is as follows:

[0079]

[0080] Where: C1, C2 represent 1x1 convolution layers with different convolution kernels;

[0081] Indicates the features of the skeletal head after separation;

[0082] F a Represents global image features;

[0083] GMP represents the global maximum pooling layer;

[0084] S represents the calculated skeletal head features and the global image feature F a Similarity mask mask;

[0085] Represents enhanced global image features.

[0086] Enhanced global image features after bone head enhancement and the skeletal features F k After the self-attention encoded feature fs, the aligned and reconstructed image feature F' is obtained. G and skeletal features F' K ;

[0087] Step S4 is specifically as follows:

[0088] Will enhance the global image features The skeleton feature fs after self-attention encoding is first obtained through the fully connected layer to obtain the features of the same dimension, and then the embedding layer is used to map it to a deeper feature space. Then, the skeleton features of the deep layer (embedding layer output) and the image features of the shallow layer (fully connected layer output) are aligned by minimizing the maximum mean difference loss MMD. Specifically, it is only necessary to calculate the distance between the deep skeleton features and the shallow image features, and to achieve alignment by minimizing the distance through training, so as to capture the features unrelated to clothes through the image while filtering out the mismatched redundant and interference information. The embedding layer uses the FC fully connected layer to map the high-dimensional features to the low-dimensional space;

[0089] In order to restore the information loss caused by forced alignment, the reconstruction layer and recon loss function are further punished for reconstruction to ensure that the aligned features retain their own features. The reconstruction layer uses the FC fully connected layer to achieve reconstruction of the original output features and minimize the maximum average difference loss MMD and recon loss function formulas are:

[0090]

[0091] Where: u(·) and σ(·) represent the mean and variance of the transformed features;

[0092] L MMD Represents minimizing the maximum mean difference loss MMD;

[0093] g represents the features before the reconstruction layer;

[0094] Represents the features of the reconstructed layer;

[0095] L recon Represents the recon loss.

[0096] S5: For the image feature F′ after alignment and reconstruction G and bone features F′ K Extract the image feature prototypes C respectively p and the skeletal feature prototype K p , image feature prototype C p and the skeletal feature prototype K p Then align and reconstruct the image features F′ G and bone features F′ K Make a loss, thereby shortening the feature distance between the same person, through the image feature prototype C p The learned image features are expressed as f α Indicates that after the skeleton feature prototype K p The learned skeleton features are expressed as f s express;

[0097] Step S5 is specifically as follows:

[0098] Based on the image feature prototype method, for each batch of P×M image data, the image features S of the same person are extracted using the method of steps S1-S4 above. img ={x1,…,x m} (In order to facilitate the representation of this step, the image feature F′ obtained after step S4 is G Use x i The image features of the same person are averaged to construct the prototype centroid, and the image prototype centroid C of P people is obtained. P ={c1,…,c p}, the prototypes in each batch can capture more intrinsic features of the same person from the perspective of a larger data set, and have better robustness to clothing changes. Then, the constraints on the features are implemented by minimizing the distance between the image instances and the corresponding image prototypes of the same person. In the process of implementing the constraints, the inherent clothing-independent features of the same person are learned by narrowing the gap between the image instances and the image prototypes.

[0099] The formula for obtaining the image prototype (average method) and the distance formula (the distance between the instance and the corresponding prototype) are:

[0100]

[0101] Where: x i Represents the image features extracted from each image, that is, the image features F' obtained after step S4 G ;

[0102] M represents the M images of each person, which means the M features of each person obtained later;

[0103] represents the features of the i-th image instance of the p-th person;

[0104] p means different people;

[0105] c p Represents the image prototype of the pth person;

[0106] Based on the method of skeleton feature prototype, for each batch of P×M image data, the image features S of the same person are extracted using the method of steps S1-S4 above. k ={y1,…,y m} (In order to facilitate the representation of this step, the bone feature F' obtained after step S4 is K Use y i The same person’s skeletal features are averaged to construct the prototype centroid, and the image prototype centroid K of P people is obtained. P ={k1, ..., k p}, then the constraints on features are achieved by minimizing the distance between the bone instance and the corresponding bone prototype of the same person. In the process of implementing the constraints, by narrowing the gap between the bone instance and the bone prototype, we can learn the inherent clothing-independent characteristics of the same person;

[0107] The formula for obtaining the skeleton prototype (average method) and the distance formula (the distance between the instance and the corresponding prototype) are:

[0108]

[0109] Where: y i Represents the skeleton features extracted from each image, that is, the skeleton features F' obtained after step S4 K ;

[0110] M represents the M images of each person, which means the M features of each person obtained later;

[0111] Represents the features of the i-th skeleton instance of the p-th person;

[0112] p means different people;

[0113] k p Represents the skeletal prototype of the pth person.

[0114] S6: For the image feature f obtained in step S5 above a and bone features f s Then use the cross attention mechanism to get the integrated fusion feature F f , and the fusion feature F f As matching features for clothing change re-identification.

[0115] Step S6 is specifically as follows:

[0116] The attention cross module of step S6 takes the image feature and the bone feature as an example. First, the multi-head cross attention mechanism is used to interact by calculating the dot product similarity between the image feature and the bone feature. The similarity matrix is ​​normalized by scaling and softmax, and the result is regarded as the attention weight to perform the weighted sum of the features. In this way, for each image feature, body bone information with similar response can be found in it; specifically, the image feature is taken as Q, the bone feature is represented by K and V, and then the image feature f integrated with the bone feature is calculated by the attention module s→a ; Perform corresponding calculations on the bone-image features on the other side to obtain f a→s ,Finally, the interacted skeleton features and image features are concatenated as the final robust and discriminative features.

[0117] The image feature f that integrates the skeleton features is calculated through the attention module s→a The formula is:

[0118]

[0119] Where: represents the normalization layer;

[0120] The skeleton feature f is calculated by the attention module to integrate the image features a→s The formula is:

[0121]

[0122] The deep convolutional neural network, matrix multiplication, normalization processing, global maximum pooling layer calculation weight coefficient, self-attention layer, self-attention coding and other related operations adopted in the present invention are all conventional operations of those skilled in the art and will not be described here.

[0123] Provide a use case for the invention

[0124] The overall workflow of clothing change re-identification in intelligent surveillance multi-camera system is as follows: Figure 6 As shown in FIG. 1 , in actual application reasoning, the target person image is first input and regarded as a query image. Then, pedestrian detection is performed on the previously acquired pedestrian query image (pedestrian detection can be replaced by any target detection algorithm. The higher the target detection accuracy, the more accurate the subsequent pedestrian re-identification). After the pedestrian detection in the image is completed, the pedestrian feature is extracted from the area where the pedestrian is located using the method of the present invention, which is recorded as the query feature (i.e., the fusion feature F f );

[0125] In an intelligent monitoring system, the number of cameras is usually not a single camera, but multiple cameras, and multiple scenes and multiple angles are involved. Therefore, in practical applications, the image data can be directly specified as a gallery data set, or multiple RTSP video streams can be deframed in real time, and then the images deframed at the same time are recorded as gallery data sets. For each image in the gallery data set, a pedestrian detection algorithm is used to detect pedestrians in the image, and then the method of the present invention is used to extract the features of pedestrians changing clothes, which are recorded as gallery features (i.e., fusion features F f );

[0126] Then calculate the distance between the query feature and the gallery feature (distance calculation can be calculated by commonly used cosine distance or Euro-style distance), and record the match if the distance is less than the set threshold as successful and mark the traveler ID in the picture, otherwise it will be recorded as failed in the match.

Claims

1. A multi-modal fusion clothing-changing pedestrian re-identification method in a multi-angle intelligent monitoring scenario, characterized in that: The following steps are involved: S1: Collect image data. The image data must include pedestrian images, which are used as input image data I. At the same time, P*M images are used as batch input, where P represents the pedestrian category and M represents the number of images corresponding to each pedestrian in different environments by multiple cameras. Assume the input image data I is: I={I0,…,I i ,I P×M } Where: {0,…,i,P×M} represents the image sequence number; S2: Extract global image features F from pedestrian images through image feature extraction network a , extract bone features F through the bone feature extraction network k ,HRNet outputs in the form of bone heatmap; S3: Skeleton head enhancement module, using the head area of ​​the skeleton heat map extracted by the skeleton feature extraction network in step S2 as a mask to enhance the global image feature F a Head features; S4: Enhanced global image features after skeleton head enhancement and the skeletal features F k After the self-attention encoded feature fs, the aligned and reconstructed image feature F' is obtained. G and skeletal features F' K ; S5: For the image feature F' after alignment and reconstruction G and skeletal features F' K Extract the image feature prototypes C respectively p and the skeletal feature prototype K p , image feature prototype C p and the skeletal feature prototype K p Then align and reconstruct the image features F' G and skeletal features F' K Make a loss, thereby shortening the feature distance between the same person, through the image feature prototype C p The learned image features are expressed as f a Indicates that after the skeleton feature prototype K p The learned skeleton features are expressed as f s express; S6: For the image feature f obtained in step S5 above a and bone features f s Then use the cross attention mechanism to get the integrated fusion feature F f , and the fusion feature F f As matching features for clothing change re-identification.

2. According to the multi-angle intelligent monitoring scenario multi-modal fusion clothing-changing pedestrian re-identification method of claim 1, it is characterized by: Step S2 is as follows: the pedestrian image data is passed into two branches, wherein the first branch extracts the global image features F from the pedestrian image through the image appearance branch network. a , using ResNet50 as the backbone, the stride of the first convolutional layer in the res4 block is set to 1; the second branch is the skeleton feature extraction network HRNet extracted through the body structure feature branch k ,HRNet outputs in the form of a skeleton heat map to indicate the location of key points, including 17 key points, representing 17 key areas of the human body.

3. According to the multi-angle intelligent monitoring scenario multi-modal fusion clothing-changing pedestrian re-identification method of claim 2, it is characterized by: Step S3 is specifically as follows: For the extracted bone feature F k Separate the head skeleton features. The output dimension of HRNet is [B, C, H, W], where B represents the batch and C represents the number of channels. The 5 key point features representing the head are extracted as the skeleton head features, and the first 5 dimensions of the C dimension representing the channel are extracted as the skeleton output. For the separated skeleton head features containing 5 head key points Use a 1x1 convolutional layer for dimensionality reduction and reshaping, and then calculate the skeleton head features through matrix multiplication and the global image feature F a The correlation between them is normalized by the Softmax layer so that the sum of the weights of each row is 1. The global maximum pooling layer is used to extract the maximum weight and reduce the computational complexity to obtain the skeleton head features. and the global image feature F a The weight coefficient between them is the similarity mask, and then the similarity mask is combined with the global image feature F a Multiply to generate weighted feature mask*F a , to achieve the global image feature F a The weighted average of the global image feature F a While maintaining the image features, it can pay more attention to the head area, and finally combine the weighted features and the global image features F a Fusion, to obtain enhanced global image features guided by the skeleton head For the bone feature F k Use a self-attention layer to encode the output to capture the relationship between different skeleton key points.

4. According to the method of multi-modal fusion clothing-changing pedestrian re-identification in multi-angle intelligent monitoring scenarios in claim 3, it is characterized in that: In step S3, the enhanced global image features guided by the skeleton head are obtained The formula is as follows: Where: C1, C2 represent 1x1 convolution layers with different convolution kernels; Indicates the features of the head of the skeleton after separation; F a Represents global image features; GMP represents the global maximum pooling layer; S represents the calculated skeletal head features and the global image feature F a Similarity mask mask; Represents enhanced global image features.

5. According to the method of multi-modal fusion clothing-changing pedestrian re-identification in multi-angle intelligent monitoring scenarios in claim 3, it is characterized in that: Step S4 is specifically as follows: Will enhance the global image features The skeleton feature fs after self-attention encoding is first obtained through a fully connected layer to obtain features of the same dimension, and then the embedding layer is used to map it to a deeper feature space. The deep skeleton features and the shallow image features are then aligned by minimizing the maximum mean difference loss MMD. Specifically, it is only necessary to calculate the distance between the deep skeleton features and the shallow image features, and achieve alignment by minimizing the distance through training, so as to capture features unrelated to clothes through the image while filtering out mismatched redundant and interference information. The embedding layer uses the FC fully connected layer to map high-dimensional features to low-dimensional space.

6. According to the multi-angle intelligent monitoring scenario multi-modal fusion clothing-changing pedestrian re-identification method of claim 5, it is characterized in that: Penalized reconstruction is performed through the reconstruction layer and the recon loss function. The reconstruction layer uses the FC fully connected layer to reconstruct the original output features and minimize the maximum mean difference loss MMD and the recon loss function formula: Where: u(·) and σ(·) represent the mean and variance of the transformed features; L MMD Represents minimizing the maximum mean difference loss MMD; g represents the features before the reconstruction layer; Represents the features of the reconstructed layer; L recon Represents the recon loss.

7. According to the multi-angle intelligent monitoring scenario multi-modal fusion clothing-changing pedestrian re-identification method of claim 5, it is characterized by: Step S5 is specifically as follows: Based on the image feature prototype method, for each batch of P×M image data, the image features S of the same person are extracted using the method of steps S1-S4 above. img ={x1,…,x m }, the image features of the same person are averaged to construct the prototype centroid, and the image prototype centroid C of P people is obtained. P ={c1,…,c p }, the prototypes in each batch can capture more intrinsic features of the same person from the perspective of a larger data set, and have better robustness to clothing changes. Then, the feature constraints are implemented by minimizing the distance between the image instances and the corresponding image prototypes of the same person. In the process of implementing the constraints, the gap between the image instances and the image prototypes is narrowed to learn the inherent clothing-independent features of the same person; the formula for obtaining the image prototype and the distance formula between the instance and the corresponding prototype are: Where: x i Represents the image features extracted from each image, that is, the image features F' obtained after step S4 G ; M represents the M images of each person, which means the M features of each person obtained later; represents the features of the i-th image instance of the p-th person; p means different people; c p Represents the image prototype of the pth person; Based on the method of skeleton feature prototype, for each batch of P×M image data, the image features S of the same person are extracted using the method of steps S1-S4 above. k ={y1,…,y m }, the same person’s skeletal features are averaged to construct the prototype centroid, and the image prototype centroid K of P people is obtained. P ={k1, ..., k p Then, the feature constraints are implemented by minimizing the distance between the skeleton instance and the corresponding skeleton prototype of the same person. In the process of implementing the constraints, the gap between the skeleton instance and the skeleton prototype is narrowed to learn the inherent clothing-independent features of the same person; The formula for obtaining the skeleton prototype and the distance formula between the instance and the corresponding prototype are: Where: y i Represents the skeleton features extracted from each image, that is, the skeleton features F' obtained after step S4 K ; M represents the M images of each person, which means the M features of each person obtained later; Represents the features of the i-th skeleton instance of the p-th person; p means different people; k p Represents the skeletal prototype of the pth person.

8. According to the multi-angle intelligent monitoring scenario multi-modal fusion clothing-changing pedestrian re-identification method of claim 5, characterized in that: Step S6 is specifically as follows: The attention cross module of step S6 takes the image feature and the bone feature as an example. First, the multi-head cross attention mechanism is used to interact by calculating the dot product similarity between the image feature and the bone feature. The similarity matrix is ​​normalized by scaling and softmax, and the result is regarded as the attention weight to perform the weighted sum of the features. In this way, for each image feature, body bone information with similar response can be found in it; specifically, the image feature is taken as Q, the bone feature is represented by K and V, and then the image feature f integrated with the bone feature is calculated by the attention module s→a ; Perform corresponding calculations on the bone-image features on the other side to obtain f a→s ,Finally, the interacted skeleton features and image features are concatenated as the final robust and discriminative features.

9. The method for re-identifying pedestrians by multi-modal fusion in a multi-angle intelligent monitoring scenario according to claim 8 is characterized in that: The image feature f that integrates the skeleton features is calculated through the attention module s→a The formula is: Where: represents the normalization layer; The skeleton feature f is calculated by the attention module to integrate the image features a→s The formula is:

Images