Multi-angle intelligent monitoring scene multi-modal fusion clothes-changing pedestrian re-identification method

Abstract

The multi-angle intelligent monitoring scene multi-modal fusion clothes-changing pedestrian re-identification method extracts the skeleton features and global image features of the original pedestrian image, strengthens the head features of the image by using the skeleton features without introducing an additional face model, thereby obtaining the skeleton head features and the global image features F a Between the mask, then the mask is multiplied with the global image features to generate a weighted feature, realizing the weighted average of the global image features F a , so that the global image features F a Can pay more attention to the head region while maintaining the image features, and finally the weighted features and the global image features F a Fusion, get the enhanced global image features guided by the skeleton head, and the enhanced global image features and the skeleton features F k After the self-attention coding, the features fs are aligned and reconstructed, and the appearance and body structure information are effectively extracted, the application can extract the features irrelevant to each pedestrian and the clothes, and then match the searched pedestrian target object.

Description

TECHNICAL FIELD

[0001] The present application relates to a pedestrian recognition method, in particular to a multi-modal fusion clothing-changing pedestrian re-identification method in a multi-angle intelligent monitoring scene, belonging to the technical field of clothing-changing pedestrian re-identification. BACKGROUND

[0002] Identifying the identity of the same person under different cameras or different scenes is a basic but important task in security monitoring projects. Pedestrian re-identification aims to detect the location of a specific target in different cameras and identify its identity, and then track it. The pedestrian re-identification system analyzes the pedestrian identity information in the video stream in real time and automatically, and under the condition that most faces are not clear in actual real scenes, it can realize the tracking and timely positioning of a specific person across time and space, and plays an important role in security monitoring, intelligent transportation and smart city, etc.

[0003] Most current researches are based on the assumption that pedestrians appear under different cameras in a relatively short period of time, so that the clothes of pedestrians do not change in different camera ranges. Based on this assumption, short-term pedestrian re-identification has been widely studied. However, short-term pedestrian re-identification does not take into account the fact that in real-life scenarios, pedestrians may change their appearance significantly due to clothing changes, so more attention is paid to clothing-changing pedestrian re-identification in real-time scenarios.

[0004] In order to extract robust features independent of clothes, some existing methods introduce a face recognition model to obtain pedestrian facial features. However, this method requires an additional face recognition model and is only applicable to scenes where the face is clearly visible, which greatly limits its practical application. Other research methods focus on extracting features independent of clothes from auxiliary modes (outline, gait, skeleton, 3D shape), but these methods do not take into account the huge differences between RGB features and other modal features, and only use relatively simple interactions, which cannot fully utilize the rich multi-modal information. SUMMARY

[0005] The purpose of the present application is to provide a multi-modal fusion clothing-changing pedestrian re-identification method in a multi-angle intelligent monitoring scene, which can extract features independent of pedestrians and their clothes under multiple cameras, and then quickly match the pedestrian features under multiple cameras to extract the target pedestrian object to be searched.

[0006] In order to achieve the above purpose, the present application provides a multi-modal fusion clothing-changing pedestrian re-identification method in a multi-angle intelligent monitoring scene, comprising the following steps:

[0007] S1: Collect image data, use monitoring equipment to collect image data, and require that the image data must contain pedestrian images as input image data I, and use P*M pictures as batch input, where P represents pedestrian category and M represents the number of images corresponding to each pedestrian in different environments of multiple cameras;

[0008] Let the input image data I be:

[0009] I = {I0, …, I i , I P×M}

[0010] In the formula: {0, …, i, P x M} represents the image sequence number;

[0011] S2: Extract global image features F a and skeletal features F k from pedestrian images respectively through image feature extraction networks and skeletal feature extraction networks, and HRNet outputs in the form of a skeletal heat map;

[0012] S3: The skeletal head enhancement module uses the head region of the skeletal heat map extracted by the skeletal feature extraction network in step S2 as a mask processing to strengthen the head features of the global image features F a ;

[0013] S4: Align and reconstruct the enhanced global image features F and the features fs of the skeletal features F k after self-attention encoding to obtain the aligned and reconstructed image features F′ G and skeletal features F′ K ;

[0014] S5: Extract image feature prototypes C p and skeletal feature prototypes K p from the aligned and reconstructed image features F′ G and skeletal features F′ K respectively, and then perform loss on the image feature prototypes C p and skeletal feature prototypes K p and the aligned and reconstructed image features F′ G and skeletal features F′ K , thereby narrowing the feature distance between the same person, and the learned image features after the image feature prototypes C p are represented by f a , and the learned skeletal features after the skeletal feature prototypes K p are represented by f s ;

[0015] S6: For the image features f a and skeletal features f s obtained after step S5,a and skeletal features f s Then, the fusion feature F is obtained by using a cross-attention mechanism. f and will fuse feature F f As a matching feature for clothing change re-identification.

[0016] Step S2 of this invention specifically involves: inputting pedestrian image data into two branches, wherein the first branch extracts global image features F from the pedestrian image through an image appearance branch network. a Using ResNet50 as the backbone, the stride of the first convolutional layer in the res4 block is set to 1 to improve feature resolution; the second branch extracts skeletal features F through the HRNet skeletal feature extraction network of the body structure feature branch. k HRNet outputs a skeleton heatmap to represent the location of key points, including 17 key points that represent 17 key regions of the human body.

[0017] Step S3 of the present invention specifically includes:

[0018] For the extracted skeletal features F k After separating the head skeletal features, the output dimension of HRNet is [B, C, H, W], where B represents the batch size, C represents the number of channels, and also represents the 17 key regions of the human body. The skeletal head features can be extracted from the 5 key points representing the head. Specifically, this only requires extracting the first 5 dimensions of dimension C (representing channels) as the skeletal output. This results in the separated skeletal head features containing the 5 head key points. Dimensionality reduction and reshaping are performed using 1x1 convolutional layers, followed by matrix multiplication to compute skeletal head features. and global image features F a The correlation between the weights is analyzed, and the Softmax layer normalizes the weights so that the sum of the weights in each row is 1. A global max pooling layer is used to extract the maximum weight and reduce computational complexity, thus obtaining the skeletal head features. and global image features F a The weight coefficients between them, i.e., the similarity mask, are then compared with the global image features F. a Multiply to generate a weighted feature mask * F a To achieve global image feature F a The weighted average makes the global image features F a While preserving image features, it can focus more on the head region, and finally combine the weighted features and global image features F. a The fusion yields enhanced global image features guided by the skeletal head. For skeletal feature F kThe encoding output is performed using a self-attention layer, and the purpose is to capture the relationship between different skeletal key points;

[0019] The enhanced global image feature guided by the skeletal head obtained in step S3 The formula is as follows:

[0020]

[0021] In the formula, C1 and C2 represent 1x1 convolution layers of different convolution kernels;

[0022] represents the separated skeletal head feature;

[0023] F a represents the global image feature;

[0024] GMP represents a global maximum pooling layer;

[0025] S represents a similarity mask mask of the calculated skeletal head feature and the global image feature F a

[0026] represents the enhanced global image feature.

[0027] The step S4 of the present application is specifically:

[0028] The enhanced global image feature is first mapped to the same latitude feature through a fully connected layer, then mapped to a deeper feature space using an embedding layer, and then the deep skeletal feature (embedding layer output) and the shallow image feature (fully connected layer output) are aligned through the minimization of the maximum mean difference loss MMD. Specifically, only the distance between the deep skeletal feature and the shallow image feature needs to be calculated, and the alignment is achieved by minimizing the distance through training, so as to capture the clothing-independent features through the image and filter the redundant and interfering information that does not match. The embedding layer uses an FC fully connected layer to map the high-dimensional feature to a low-dimensional space;

[0029] In order to recover the information loss caused by forced alignment, the present application further punishes reconstruction through a reconstruction layer and a recon loss function to ensure that the aligned features retain their own features. The reconstruction layer uses an FC fully connected layer to reconstruct the original output feature, and the minimization of the maximum mean difference loss MMD and the recon loss function formula is:

[0030]

[0031] In the formula, u(·) and σ(·) represent the mean and variance of the transformed feature.​

[0032] L MMD represents the minimum maximum mean difference loss MMD;

[0033] g represents the feature before the reconstruction layer;

[0034] represents the feature after the reconstruction layer;

[0035] L recon represents the recon loss.

[0036] The step S5 of the present application is specifically:

[0037] Based on the image feature prototype method, for each batch of P x M image data, after the image features S img ={x1,…,x m} of the same person are extracted using the above steps S1-S4, the image features of the same person are constructed into a prototype centroid by averaging, obtaining P image prototype centroids C P ={c1,…,c p} of the same person, the prototype in each batch can capture more inherent features of the same person from the perspective of a larger data set, and has better robustness to clothing changes, and then the constraint on the features is realized by minimizing the distance between the image instances and the corresponding image prototypes of the same person, and in the process of realizing the constraint, the inherent and clothing-independent features of the same person are learned by narrowing the gap between the image instances and the image prototypes;

[0038] The formula for calculating the image prototype (averaging method) and the distance formula (distance between instances and corresponding prototypes) are as follows:

[0039]

[0040] In the formula, x i represents the image features extracted from each picture, i.e., the image features F' G obtained after step S4.

[0041] M represents M pictures of each person, i.e., M features of each person obtained subsequently.

[0042] represents the i-th image instance feature of the p-th person;

[0043] p represents different persons;

[0044] c p represents the image prototype of the p-th person;

[0045] The method based on the bone feature prototype, for each batch of P x M image data, after the image features S of the same person are extracted by using the method of steps S1-S4 k ={y1,...,y m}, the bone features of the same person are constructed by averaging to obtain the image prototype centroid K of P persons P ={k1,...,k p}, and then the constraint on the features is realized by minimizing the distance between the bone instance and the corresponding bone prototype, and in the process of realizing the constraint, the inherent and clothing-independent features of the same person are learned by narrowing the gap between the bone instance and the bone prototype;

[0046] The formula for calculating the bone prototype (averaging method) and the distance formula (distance between instance and corresponding prototype) are as follows:

[0047]

[0048] In the formula, y i represents the bone feature extracted from each picture, that is, the bone feature F' K obtained after step S4.

[0049] M represents M pictures of each person, that is, M features of each person obtained subsequently;

[0050] represents the i-th bone instance feature of the p-th person;

[0051] p represents different persons;

[0052] k p represents the bone prototype of the p-th person.

[0053] The step S6 of the present application is specifically:

[0054] The attention cross module of step S6 takes the image features and bone features as an example, first introduces the multi-head cross attention mechanism to interact by calculating the dot product similarity between the image features and the bone features, the similarity matrix is normalized by scaling and softmax, and the result is regarded as the attention weight to perform the weighted sum of the features, so that for each image feature, the body bone information with similar response can be found therein; specifically, the image features are taken as Q, the bone features are taken as K and V, and then the image features integrated with the bone features f s→a are calculated by the attention module; for the bone-image features on the other side, corresponding calculation is performed to obtain f a→s , and finally the interacted bone features and image features are spliced to obtain the final features with robustness and discriminability.

[0055] Image features f, which integrate skeletal features, are calculated using an attention module. s→a The formula is:

[0056]

[0057] In the formula: Indicates the normalization layer;

[0058] Skeletal features f, which integrate image features, are computed using an attention module. a→s The formula is:

[0059]

[0060] Compared with existing technologies, this invention proposes a method for re-identifying pedestrians changing clothes based on skeletal points and RGB multimodal feature fusion in multi-angle scenarios of intelligent surveillance. By extracting skeletal features and global image features from the original pedestrian image, and without introducing an additional face model, the extracted skeletal features are used to enhance the skeletal head features, thereby obtaining the skeletal head features. and global image features F a The weight coefficients between them, i.e., the similarity mask, are then compared with the global image features F. a Multiply to generate a weighted feature mask * F a To achieve global image feature F a The weighted average makes the global image features F a While preserving image features, it can focus more on the head region, and finally combine the weighted features and global image features F. a The fusion yields enhanced global image features guided by the skeletal head. Enhanced global image features after skeletal head enhancement and skeletal features F k After self-attention encoding, the feature fs is aligned and reconstructed, effectively extracting appearance and body structure information. By aligning the features of each instance with its prototype, the distance between the same pedestrians is narrowed, and the differences between different people are amplified. Furthermore, the appearance features are refined with the help of body structure features unrelated to clothing, while the body structure features supplement the aligned appearance features that are robust to clothing changes. Through the above method, this invention can extract pedestrians and their clothing-independent human body features in multi-camera and multi-scene monitoring scenarios. Then, by matching the re-identification features of pedestrians changing clothes extracted from multiple cameras, it can quickly match the target pedestrian object under conditions of multiple cameras, changing environment, and changed clothing. Attached Figure Description

[0061] Figure 1Flow chart of the method of the invention;

[0062] Figure 2 Structure diagram of the invention based on skeleton head feature enhanced image feature;

[0063] Figure 3 Structure diagram of the invention of alignment reconstruction of skeleton features and RGB image features;

[0064] Figure 4 Schematic diagram of the prototype module of the invention for extracting image features;

[0065] Figure 5 Framework diagram of the cross-attention module of the invention;

[0066] Figure 6 Framework diagram of the clothing-changing pedestrian re-identification in the actual monitoring application scenario. DETAILED DESCRIPTION

[0067] The invention will be further described below with reference to the accompanying drawings.

[0068] As shown in Figures 1-5 , a multi-modal fusion clothing-changing pedestrian re-identification method in a multi-angle intelligent monitoring scene includes the following steps:

[0069] S1: Collect image data, collect image data through intelligent monitoring devices at multiple different angles, and require that the image data must contain pedestrian images, which are taken as input image data I. In order to better aggregate the feature attributes between the same person and pull apart the features between different people, P*M pictures are used as batch input, where P represents the number of pedestrian categories, and M represents the number of images corresponding to each pedestrian;

[0070] Let the input image data I be:

[0071] I={I0,…,I i , I P×M}

[0072] In the formula: {0,…,i,P×M} represents the image sequence number;

[0073] S2: Extract global image features F a and skeleton features F k from the pedestrian image through the image feature extraction network and the skeleton feature extraction network respectively. HRNet outputs in the form of a skeleton heat map;

[0074] Step S2 is specifically: input the pedestrian image data into two branches, where the first branch is to extract global image features F aUsing ResNet50 (deep convolutional neural network) as the backbone, the stride of the first convolutional layer in the res4 block is set to 1 to improve feature resolution; the second branch extracts skeletal features F through the HRNet skeletal feature extraction network of the body structure feature branch. k HRNet outputs a skeleton heatmap to represent the location of key points, including 17 key points that represent 17 key regions of the human body.

[0075] S3: Head Enhancement Module, which uses the head region of the skeletal heatmap extracted by the skeletal feature extraction network in step S2 as a mask to enhance global image features F. a Head features;

[0076] Step S3 is as follows:

[0077] For the extracted skeletal features F k After separating the head skeletal features, the output dimension of HRNet is [B, C, H, W], where B represents the batch size, C represents the number of channels, and also represents the 17 key regions of the human body. The skeletal head features can be extracted from the 5 key points representing the head. Specifically, this only requires extracting the first 5 dimensions of dimension C (representing channels) as the skeletal output. This results in the separated skeletal head features containing the 5 head key points. Dimensionality reduction and reshaping are performed using 1x1 convolutional layers, followed by matrix multiplication to compute skeletal head features. and global image features F a The correlation between the weights is analyzed, and the Softmax layer normalizes the weights so that the sum of the weights in each row is 1. A global max pooling layer is used to extract the maximum weight and reduce computational complexity, thus obtaining the skeletal head features. and global image features F a The weight coefficients between them, i.e., the similarity mask, are then compared with the global image features F. a Multiply to generate a weighted feature mask * F a To achieve global image feature F a The weighted average makes the global image features F a While preserving image features, it can focus more on the head region, and finally combine the weighted features and global image features F. a The fusion yields enhanced global image features guided by the skeletal head. For skeletal feature F k The purpose of using a self-attention layer for encoding output is to capture the relationships between different skeletal keypoints.

[0078] The enhanced global image features guided by the skeletal head are obtained. The formula is as follows:

[0079]

[0080] C1, C2 represent 1x1 convolutional layers of different convolutional kernels;

[0081] represents the separated skeletal head feature;

[0082] F a represents the global image feature;

[0083] GMP represents a global max pooling layer;

[0084] S represents a similarity mask mask of the calculated skeletal head feature and the global image feature F a ;

[0085] represents an enhanced global image feature.

[0086] The enhanced global image feature after the skeletal head enhancement is aligned and reconstructed with the skeletal feature F k after self-attention encoding, to obtain the aligned and reconstructed image feature F' G and the skeletal feature F' K ;

[0087] Step S4 is specifically:

[0088] The enhanced global image feature is first mapped to the same latitude feature through a fully connected layer, and then mapped to a deeper feature space using an embedding layer. Then, the deep skeletal feature (embedding layer output) is aligned with the shallow image feature (fully connected layer output) by minimizing the maximum mean difference loss MMD. Specifically, only the distance between the deep skeletal feature and the shallow image feature needs to be calculated, and the alignment is achieved by minimizing the distance through training. Thus, the clothes-irrelevant features captured by the image are filtered while redundant and interfering information that does not match is filtered out. The embedding layer uses an FC fully connected layer to map high-dimensional features to a low-dimensional space.

[0089] In order to recover the information loss caused by forced alignment, further punishment reconstruction is performed through a reconstruction layer and a recon loss function to ensure that the aligned features retain their original features. The reconstruction layer uses an FC fully connected layer to reconstruct the original output features. The minimum maximum mean difference loss MMD and the recon loss function formula are:

[0090]

[0091] wherein u(·) and σ(·) represent the mean and variance of the transformed features;

[0092] L MMD represents the minimization of the maximum mean discrepancy loss MMD;

[0093] g represents the features before the reconstruction layer;

[0094] represents the features after the reconstruction layer;

[0095] L recon represents the recon loss.

[0096] S5: For the aligned reconstructed image features F′ G and the skeleton features F′ K , their image feature prototypes C p and skeleton feature prototypes K p are extracted respectively, and the image feature prototypes C p and the skeleton feature prototypes K p are respectively made loss with the aligned reconstructed image features F′ G and the skeleton features F′ K , so as to narrow the feature distance between the same person, and the learned image features after the image feature prototypes C p are denoted as f α , and the learned skeleton features after the skeleton feature prototypes K p are denoted as f s ;

[0097] The step S5 is specifically:

[0098] The method based on the image feature prototypes, for P x M image data in each batch, after the image features S img ={x1, …, x m} of the same person are extracted by using the above steps S1-S4 (for the convenience of this step, the image features F′ G obtained after step S4 are denoted as x i ), the image features of the same person are constructed by the prototype centroid through the way of averaging, and the image prototype centroid C P ={c1, …, c p} of P persons is obtained, the prototype in each batch can capture more inherent features of the same person from the perspective of a larger data set, and has better robustness to clothing changes, and then the constraint on the features is realized by minimizing the distance between the image instances and the corresponding image prototypes of the same person, and in the process of realizing the constraint, the inherent and clothing-independent features of the same person are learned by narrowing the gap between the image instances and the image prototypes.

[0099] The formula for obtaining the image prototype (average method) and the distance formula (distance between the instance and the corresponding prototype) are:

[0100]

[0101] wherein x i represents the image feature extracted from each picture, i.e., the image feature F' G obtained after step S4;

[0102] M represents the M pictures of each person, i.e., the M features of each person obtained subsequently;

[0103] represents the i-th image instance feature of the p-th person;

[0104] p represents different persons;

[0105] c p represents the image prototype of the p-th person;

[0106] Based on the method of the skeletal feature prototype, for each batch of P x M image data, after the image feature S k = {y1,..., y m} of the same person is extracted by using the above steps S1-S4 (for the convenience of the present step, the skeletal feature F' K obtained after step S4 is denoted as y i ), the skeletal feature of the same person is used to construct the prototype centroid by the average method, and the image prototype centroid K P = {k1,..., k p} of the P persons is obtained, and then the constraint on the feature is realized by minimizing the distance between the skeletal instance and the corresponding skeletal prototype, and in the process of realizing the constraint, the inherent and clothing-independent feature of the same person is learned by narrowing the gap between the skeletal instance and the skeletal prototype;

[0107] The formula for obtaining the skeletal prototype (average method) and the distance formula (distance between the instance and the corresponding prototype) are:

[0108]

[0109] wherein y i represents the skeletal feature extracted from each picture, i.e., the skeletal feature F' K obtained after step S4;

[0110] M represents the M pictures of each person, i.e., the M features of each person obtained subsequently;

[0111] represents the i-th skeletal instance feature of the p-th person;

[0112] p represents different persons;

[0113] k p represents the bone prototype of the pth person.

[0114] S6: for the image feature f a and the bone feature f s , the integrated fusion feature F is obtained again by using the cross attention mechanism f , and the fusion feature F f is taken as the matching feature of clothes changing re-identification.

[0115] Step S6 is specifically:

[0116] The attention cross module of step S6 takes the image feature and the bone feature as an example. First, the multi-head cross attention mechanism is referenced to interact by calculating the dot product similarity between the image feature and the bone feature. The similarity matrix is normalized by scaling and softmax, and the result is taken as the attention weight to perform the weighted sum of the features. In this way, for each image feature, the body bone information with similar response can be found therein; specifically, the image feature is taken as Q, and the bone feature is taken as K and V, and then the image feature f s→a of the integrated bone feature is calculated by the attention module; for the bone-image feature on the other side, the corresponding calculation is performed to obtain f a→s , and finally the interacted bone feature and the image feature are spliced to obtain the final feature with robustness and distinguishability.

[0117] The formula for calculating the image feature f s→a of the integrated bone feature by the attention module is:

[0118]

[0119] In the formula: represents a normalization layer;

[0120] The formula for calculating the bone feature f a→s of the integrated image feature by the attention module is:

[0121]

[0122] The deep convolutional neural network, matrix multiplication, normalization processing, global maximum pooling layer calculation weight coefficient, self-attention layer, self-attention encoding and other related operations used in the application all belong to the routine operations of those skilled in the art, and will not be described here.

[0123] Use case of the application is given

[0124] The overall workflow of clothes changing re-identification in an intelligent monitoring multi-camera system is shown in FIG. 30. Figure 6 In actual application reasoning, first, a target person picture is input, which is regarded as a query picture, then a pre-acquired pedestrian query picture is subjected to pedestrian detection (pedestrian detection can be replaced by any target detection algorithm, the higher the target detection accuracy, the more accurate the subsequent pedestrian re-identification), after the pedestrian in the picture is detected, the method of the present application is used to extract the pedestrian feature in the region where the pedestrian is located, which is recorded as query feature (i.e. fusion feature F f );

[0125] In an intelligent monitoring system, the number of cameras is usually not a single camera, but multiple cameras, and multiple scenes and multiple angles are involved, so in actual application, picture data can be directly specified as a gallery data set, or multiple rtsp video streams can be real-time frame-splitting, then the pictures split at the same time are recorded as a gallery data set, each picture in the gallery data set is subjected to pedestrian detection using a pedestrian detection algorithm, then the clothes changing pedestrian feature is extracted using the method of the present application, which is recorded as gallery feature (i.e. fusion feature F f );

[0126] Then the distance between the query feature and the gallery feature is calculated (the distance can be calculated by using common cosine distance or Euclidean distance), if the distance is less than a set threshold, it is recorded as a matching success and the pedestrian ID is marked in the picture, otherwise it is recorded as a matching failure.

Claims

1. A multimodal fusion method for re-identifying pedestrians changing clothes in a multi-angle intelligent monitoring scenario, characterized in that, Includes the following steps: S1: Collect image data. The image data must include pedestrian images, which are used as input image data I. At the same time, P*M images are used as batch input, where P represents the pedestrian category and M represents the number of images of each pedestrian from multiple cameras in different environments. Let the input image data I be: I={I0,…,I i ,I P×M } In the formula: {0,…,i,P×M} represents the image sequence number; S2: Extract global image features F from pedestrian images using an image feature extraction network. a Extracting skeletal features F using a skeletal feature extraction network k HRNet outputs the data in the form of a skeleton heatmap; S3: Head Enhancement Module, which uses the head region of the skeletal heatmap extracted by the skeletal feature extraction network in step S2 as a mask to enhance global image features F. a Head features; S4: Enhanced global image features after bone head enhancement and skeletal features F k After self-attention encoding, the features fs are aligned and reconstructed to obtain the aligned and reconstructed image features F'. G and skeletal features F' K ; S5: For the aligned and reconstructed image features F' G and skeletal features F' K Extract their image feature prototypes C respectively p and skeletal feature prototype K p Image feature prototype C p and skeletal feature prototype K p Then, align and reconstruct the image features F' respectively. G and skeletal features F' K By applying loss, the feature distance between the same person can be reduced, and the image feature prototype C is processed. p The learned image features are f a This indicates that, after passing through the skeletal feature prototype K p Learned skeletal features with f s express; S6: For the image features f obtained in step S5 above... a and skeletal features f s Then, the fusion feature F is obtained by using a cross-attention mechanism. f and will fuse feature F f As a matching feature for clothing change re-identification.

2. The method for re-identifying pedestrians changing clothes in a multi-modal fusion mode under multi-angle intelligent monitoring scenarios according to claim 1, characterized in that, Step S2 specifically involves feeding pedestrian image data into two branches. The first branch extracts global image features F from the pedestrian image through an image appearance branch network. a Using ResNet50 as the backbone, the stride of the first convolutional layer in the res4 block is set to 1; the second branch extracts skeletal features F through the HRNet skeletal feature extraction network of the body structure feature branch. k HRNet outputs a skeleton heatmap to represent the location of key points, including 17 key points that represent 17 key regions of the human body.

3. The method for re-identifying pedestrians changing clothes in a multi-modal fusion mode under multi-angle intelligent monitoring scenarios according to claim 2, characterized in that, Step S3 is as follows: For the extracted skeletal features F k The head skeletal features are separated, and the output dimension of HRNet is [B, C, H, W], where B represents the batch size and C represents the number of channels. The five keypoint features representing the head are extracted as the skeletal head features, and the first five dimensions of dimension C (representing the number of channels) are extracted as the skeletal output. This separates the skeletal head features containing the five keypoints. Dimensionality reduction and reshaping are performed using 1x1 convolutional layers, followed by matrix multiplication to compute skeletal head features. and global image features F a The correlation between the weights is analyzed, and the Softmax layer normalizes the weights so that the sum of the weights in each row is 1. A global max pooling layer is used to extract the maximum weight and reduce computational complexity, thus obtaining the skeletal head features. and global image features F a The weight coefficients between them, i.e., the similarity mask, are then compared with the global image features F. a Multiply to generate a weighted feature mask * F a To achieve global image feature F a The weighted average makes the global image features F a While preserving image features, it can focus more on the head region, and finally combine the weighted features and global image features F. a The fusion yields enhanced global image features guided by the skeletal head. For skeletal feature F k Self-attention layers are used to encode the output, capturing the relationships between different skeletal keypoints.

4. The method for re-identifying pedestrians changing clothes in a multi-modal fusion mode under multi-angle intelligent monitoring scenarios according to claim 3, characterized in that, In step S3, enhanced global image features guided by the skeletal head are obtained. The formula is as follows: In the formula: C1 and C2 represent 1x1 convolutional layers with different kernels; This indicates the skeletal head features after separation; F a Represents global image features; GMP stands for Global Max Pooling Layer; S represents the calculated skeletal head features. and global image features F a Similarity mask; This indicates enhancement of global image features.

5. The method for re-identifying pedestrians changing clothes in a multi-modal fusion mode under multi-angle intelligent monitoring scenarios according to claim 3, characterized in that, Step S4 is as follows: This will enhance global image features. The skeletal features fs, after self-attention encoding, are first processed through a fully connected layer to obtain features of the same dimension. Then, an embedding layer is used to map them into a deeper feature space. The deep skeletal features and the shallow image features are then aligned by minimizing the maximum mean difference loss (MMD). Specifically, it is only necessary to calculate the distance between the deep skeletal features and the shallow image features and achieve alignment by training to minimize the distance. This allows the capture of features unrelated to clothing from the image while filtering out mismatched redundant and interfering information. The embedding layer uses a fully connected (FC) layer to map high-dimensional features to a low-dimensional space.

6. The method for re-identifying pedestrians changing clothes in a multi-modal fusion mode under multi-angle intelligent monitoring scenarios according to claim 5, characterized in that, Penalized reconstruction is performed using a reconstruction layer and a recon loss function. The reconstruction layer uses a fully connected (FC) layer to reconstruct the original output features. The formulas for minimizing the maximum mean difference loss (MMD) and the recon loss function are as follows: In the formula: u(·) and σ(·) represent the mean and variance of the transformed feature; L MMD This represents minimizing the maximum average difference loss (MMD). g represents the features before the reconstruction layer; Indicates the characteristics of the reconstructed layer; L recon This indicates recon loss.

7. The method for re-identifying pedestrians changing clothes in a multi-modal fusion mode under multi-angle intelligent monitoring scenarios according to claim 5, characterized in that, Step S5 is as follows: Based on the image feature prototype method, for each batch of P×M image data, the image features S of the same person are extracted using the methods described in steps S1-S4 above. img ={x1,…,x m Afterwards, the image features of the same person are averaged to construct the prototype centroid, resulting in the image prototype centroid C for P individuals. P ={c1, ..., c p Each batch of prototypes captures more intrinsic features of the same person from a larger dataset perspective, exhibiting better robustness to clothing changes. Then, features are constrained by minimizing the distance between image instances and corresponding prototypes of the same person. In implementing this constraint, the inherent, clothing-independent features of the same person are learned by narrowing the gap between image instances and prototypes. The formulas for calculating image prototypes and the distance between instances and corresponding prototypes are as follows: In the formula: x i This represents the image features extracted from each image, i.e., the image features F' obtained after step S4. G ; M represents the M images of each person, which are also the M features of each person obtained later. Represents the feature of the i-th image instance of the p-th person; p represents different people; c p This represents the image prototype of the p-th person. Based on the skeletal feature prototype method, for each batch of P×M image data, the image features S of the same person are extracted using the methods described in steps S1-S4 above. k ={y1,…,y m Afterwards, the skeletal features of the same person are averaged to construct the prototype centroid, resulting in the image prototype centroid K for person P. P ={k1, ..., k p Then, the feature is constrained by minimizing the distance between the skeletal instance and the corresponding skeletal prototype of the same person. In the process of implementing the constraint, the inherent clothing-independent features of the same person are learned by narrowing the gap between the skeletal instance and the skeletal prototype. The formulas for obtaining the skeletal prototype and the distance between an instance and its corresponding prototype are as follows: In the formula: y i This represents the skeletal features extracted from each image, specifically the skeletal features F' obtained after step S4. K ; M represents the M images of each person, which are also the M features of each person obtained later. This represents the feature of the i-th skeleton instance of the p-th person; p represents different people; k p This represents the skeletal prototype of the p-th person.

8. The method for re-identifying pedestrians changing clothes in a multi-modal fusion mode under multi-angle intelligent monitoring scenarios according to claim 5, characterized in that, Step S6 is as follows: Step S6, the attention cross-attention module, taking the image features and skeletal features side as an example, first employs a multi-head cross-attention mechanism to interact by calculating the dot product similarity between image features and skeletal features. The similarity matrix is ​​normalized through scaling and softmax, and the result is used as attention weights to perform a weighted sum of features. In this way, for each image feature, body skeletal information with similar responses can be found. Specifically, the image features are represented as Q, and the skeletal features are represented as K and V. Then, the attention module calculates the image feature f that integrates the skeletal features. s→a For the bone-image features on the other side, perform corresponding calculations to obtain f. a→s Finally, the skeletal features and image features after interaction are stitched together to form the final robust and discriminative features.

9. The method for re-identifying pedestrians changing clothes in a multi-modal fusion mode under multi-angle intelligent monitoring scenarios according to claim 8, characterized in that, Image features f, which integrate skeletal features, are calculated using an attention module. s→a The formula is: In the formula: Indicates the normalization layer; Skeletal features f, which integrate image features, are computed using an attention module. a→s The formula is:

Images