Pedestrian re-identification method, device and system based on attitude estimation and storage medium

Abstract

The invention discloses a pedestrian re-recognition method, device and system based on attitude estimation and a storage medium, and relates to the technical field of image recognition. The method comprises the following steps: acquiring original modal data, and extracting exclusive features to obtain a visible light modal feature map and an infrared modal feature map; extracting common features based on a multi-head self-attention mechanism to obtain a shared feature map; processing the original modal data based on attitude estimation to obtain a key point thermodynamic diagram of the original modal data; performing score diagram processing on the key point thermodynamic diagram to obtain a score diagram and score confidence of the key point thermodynamic diagram; performing multi-head batch normalization weighting processing on the score map, the score confidence coefficient and the shared feature map to obtain a feature vector set; and carrying out distance measurement matching on the feature vector set to obtain a pedestrian re-identification result. According to the embodiment of the invention, the robustness and adaptability of the system are improved, the capability of processing the shielding problem is improved, and the accuracy and efficiency of pedestrian re-identification are improved.

Description

Technical Field

[0001] The present application belongs to the field of image recognition technology, and in particular, relates to a method, device, system and storage medium for pedestrian re-identification based on posture estimation. Background Art

[0002] Target re-identification aims to match the query image with the same identity image in the gallery, which is crucial for intelligent monitoring, video analysis, etc. However, the target re-identification effect under a single data source is limited by the discriminable information of the single data source. Among multiple data sources, infrared and visible light are the most widely used. Infrared images can capture more effective information under low-light conditions, while visible light images perform better under normal conditions. The two are complementary. The complementarity of infrared and visible light can improve the effect of target re-identification, and target re-identification in multi-source scenarios is more advantageous. In addition, with the rapid development of intelligent monitoring and video data, infrared-visible light data has been accumulated in large quantities, and the society's demand for infrared-visible light target re-identification is growing. Therefore, visible light-infrared target re-identification shows great research value and application significance.

[0003] However, the existing visible light-infrared pedestrian re-identification methods are not ideal when dealing with occlusion problems, and the recognition accuracy is low. Visible light-infrared pedestrian re-identification can be mainly divided into two types according to the method of reducing modal differences: image level and feature level.

[0004] The basic idea of ​​image-level methods is to reduce modal differences at the image level. They convert the data of two modalities into each other or generate intermediate modal data to reduce modal differences. However, in the conversion process, information loss is inevitable, and the generated image is inevitably noisy, and the difference between modalities cannot be completely eliminated. Feature-level methods reduce data dependency by not focusing on the conversion between image modalities, but they increase additional training costs. Summary of the invention

[0005] The purpose of this application is to provide a pedestrian re-identification method, device, system and storage medium based on posture estimation, which aims to improve the processing capability of occlusion problem and the recognition accuracy based on posture estimation in view of the occlusion problem existing in visible light-infrared target re-identification.

[0006] In order to achieve the above objectives, the solution of this application is:

[0007] In a first aspect, an embodiment of the present application provides a pedestrian re-identification method based on posture estimation, comprising:

[0008] Acquire original modal data, which includes visible light modality and infrared modality;

[0009] The modal features of the exclusive features are extracted for the visible light modality and the infrared modality respectively, and a visible light modal feature map and an infrared modal feature map are obtained;

[0010] Based on the multi-head self-attention mechanism, the modal features of the common features of the visible light modal feature map and the infrared modal feature map are extracted to obtain a shared feature map; the shared feature map integrates the exclusive features of the visible light modal feature map, the exclusive features of the infrared modal feature map, and the common features of the visible light modal feature map and the infrared modal feature map;

[0011] The original modal data is processed based on the posture estimation to obtain the key point heat map of the original modal data; the key point heat map is processed into a score map to obtain the score map and score confidence of the key point heat map;

[0012] The score map, score confidence and shared feature map are subjected to multi-head batch normalization weighted processing to obtain a set of feature vectors; the feature vector set is matched by distance metric to obtain the pedestrian re-identification result.

[0013] According to the above method of the embodiment of the present application, the following additional technical features may also be provided:

[0014] Furthermore, based on the multi-head self-attention mechanism, the shared features of the visible light modal feature map and the infrared modal feature map are extracted to obtain the shared feature map, including:

[0015] N stages of residual processing are performed on the visible light modal feature map and the infrared modal feature map. After each stage of residual processing in the first N-1 stages, multi-scale feature aggregation is performed. After the N-1th multi-scale feature aggregation, feature extraction is performed based on the multi-head self-attention mechanism to obtain the feature extraction result. After diversified feature expansion, the feature extraction result is subjected to the Nth stage of residual processing to obtain a shared feature map.

[0016] Furthermore, feature extraction is performed based on the multi-head self-attention mechanism to obtain feature extraction results, and the feature extraction results are subjected to diversified feature expansion and then subjected to residual processing in the Nth stage to obtain a shared feature map, including:

[0017] The result of the N-1th multi-scale feature aggregation is convolved to obtain three vectors: query, key, and value. The similarity matrix is ​​obtained by calculating the dot product of the query and the key. The similarity matrix is ​​normalized by softmax to obtain the attention weight.

[0018] The weighted feature map is obtained by weighted summing of the attention weight and the value, and the weighted feature map is linearly transformed and residually connected with the result of the N-1th multi-scale feature aggregation to obtain a shared feature map.

[0019] Furthermore, the original modal data is processed based on the posture estimation to obtain the key point heat map of the original modal data, including:

[0020] The original modal data is processed by a posture estimation model. The posture estimation model includes a Pose HRNet model and its pre-trained weights. The Pose HRNet model includes a weighted branch and a parallel branch. The original modal data is input into the PoseHRNet model. N feature maps with different resolutions are obtained through the weighted branch. Information is exchanged through the parallel branch. A high-resolution feature map is generated through multi-scale feature aggregation. Feature extraction is performed on the high-resolution feature map to obtain a key point heat map of the original modal data.

[0021] Furthermore, the key point heat map is processed into a score map to obtain the score map and score confidence of the key point heat map, including:

[0022] The size of the key point heat map is adjusted to the same size as the shared feature map through bilinear interpolation. The key point heat map is divided into groups according to the Pose HRNet model and its pre-trained weights. The channel response value and channel maximum response value of each group of key point heat maps are extracted, and the channel response values ​​of each group of key point heat maps are summed up to obtain the single-channel heat map of each group. The single-channel heat map of each group is connected along the channel dimension and normalized to obtain the score map of the key point heat map. The mean of the channel maximum response value of each group of key point heat maps is calculated respectively, and the mean of the channel maximum response value of each group of key point heat maps is connected along the channel dimension to obtain the score confidence.

[0023] Furthermore, the score map, score confidence and shared feature map are subjected to multi-head batch normalization weighted processing to obtain a set of feature vectors, including:

[0024] Extract the elements in the score map and the shared feature map, multiply the score map and the shared feature map element by element, and sum them in the height and width dimensions to obtain the local feature vector;

[0025] The height and width dimensions of the shared feature map are averaged to obtain an average feature vector, and the shared feature map is processed by adaptive maximum pooling to obtain an adaptive maximum pooling result, and the adaptive maximum pooling result is added to the average feature vector to obtain a global feature vector;

[0026] Multi-head batch normalization is performed on the local feature vectors and the global feature vectors, and the feature vector set is obtained by weighting the local feature vectors and the global feature vectors by the fractional confidence.

[0027] Furthermore, the original modal data is processed by the posture estimation model, including:

[0028] The pose estimation model is optimized by calculating the gradient through the composite loss function. The formula of the composite loss function is as follows:

[0029] L=λ1L ort +λ2L cpm +λ3L cre +λ4L tri +λ5L ide

[0030] Among them, L ort represents the orthogonal loss, L cpm represents the center-guided mining loss, L cre represents the cross entropy loss, L tri represents the triplet loss, L ide represents IDE loss; λ1, λ2, λ3, λ4 and λ5 represent L ort , L cpm , L cre , L tri and L ide The weight of

[0031] Orthogonal loss L ort In order to perform average pooling after diversifying the feature extraction results, we get the orthogonal loss L ort ; After performing diversified feature expansion on the feature extraction results, embedding is generated, and the center-guided mining loss L cpm is the constraint on embedding; cross entropy loss L cre To diversify the feature extraction results, batch normalization is performed and the cross entropy loss L is obtained by calculating the cross entropy loss L with the true label. cre , the true label is obtained by pre-annotation; the triple loss L tri In order to perform batch normalization after diversifying the feature extraction results, similar samples are screened by the true labels to obtain the most difficult positive samples and the easiest negative samples. The triplet loss L is calculated by the most difficult positive samples, the easiest negative samples and the true labels. tri IDE loss L ide It is calculated by the set of feature vectors, fractional confidence and true labels.

[0032] In a second aspect, an embodiment of the present application provides a pedestrian re-identification device based on posture estimation, comprising:

[0033] A data acquisition module is configured to acquire raw modality data, the raw modality data including visible light modality and infrared modality;

[0034] The feature extraction module is configured to extract modal features of exclusive features for the visible light modality and the infrared modality, respectively, to obtain a visible light modal feature map and an infrared modal feature map; extract modal features of common features for the visible light modal feature map and the infrared modal feature map based on a multi-head self-attention mechanism, to obtain a shared feature map; the shared feature map integrates the exclusive features of the visible light modal feature map, the exclusive features of the infrared modal feature map, and the common features of the visible light modal feature map and the infrared modal feature map;

[0035] The posture estimation module is configured to process the original modal data based on the posture estimation to obtain a key point heat map of the original modal data; perform score map processing on the key point heat map to obtain a score map and score confidence of the key point heat map; perform multi-head batch normalization weighted processing on the score map, the score confidence and the shared feature map to obtain a feature vector set;

[0036] The re-identification module is configured to perform distance metric matching on the feature vector set to obtain a pedestrian re-identification result.

[0037] In a third aspect, an embodiment of the present application provides a pedestrian re-identification system based on posture estimation, the system comprising a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the pedestrian re-identification method based on posture estimation as provided in the first aspect of the embodiment of the present application.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement a pedestrian re-identification method based on posture estimation as in the first aspect of the embodiment of the present application.

[0039] Compared with the prior art, the pedestrian re-identification method based on posture estimation provided by the embodiment of the present application has the following beneficial technical effects:

[0040] The embodiment of the present application simultaneously acquires data of visible light modality and infrared modality, making full use of the complementarity of different modality data under conditions such as illumination change and environmental occlusion, thereby improving the robustness and adaptability of the system;

[0041] The embodiment of the present application can capture the unique information of each modality by extracting features from each modality data separately, thus providing a basis for subsequent multimodal fusion; multi-stage residual processing and multi-scale feature aggregation enhance the expressiveness of features; the self-attention mechanism can capture the long-distance dependency between features, further enriching the feature representation; and the multi-head design can learn multiple subspace information of features, thus improving the diversity of features;

[0042] The embodiment of the present application adopts a high-performance posture estimation model (such as Pose HRNet) to accurately extract the position information of the key points of the human body, and maintain high accuracy even in complex scenes; generates a high-resolution feature map by combining weighted branches and parallel branches, retains more spatial details, and provides accurate key point information for subsequent feature fusion and weighting; adjusts the size of the key point heat map to the same as the shared feature map to facilitate subsequent feature fusion; through normalization processing, makes the response values ​​of different key points comparable; generates a score map and a score confidence, which are used to represent the accuracy and reliability of the key point position, respectively, to provide a basis for subsequent feature weighting;

[0043] The embodiment of the present application generates a local feature vector by combining a score map and a shared feature map, and generates a global feature vector by using an average feature vector and an adaptive maximum pooling result, thereby comprehensively capturing local and global information; through multi-head batch normalization processing and fractional confidence weighting, effective fusion and weighting of feature vectors are achieved, thereby improving the discrimination and robustness of feature vectors; using the obtained feature vector set for distance metric matching, pedestrians in different scenarios can be accurately identified, thereby improving the ability to handle occlusion problems and improving the accuracy and efficiency of pedestrian re-identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram showing a flow chart of a pedestrian re-identification method based on posture estimation according to an embodiment of the present application is shown;

[0045] Figure 2 A structural block diagram of a pedestrian re-identification device based on posture estimation according to an embodiment of the present application is shown;

[0046] Figure 3 A structural block diagram of a computer device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some structures related to the present application are shown in the accompanying drawings, rather than all structures. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0048] The terms "including" and "having" and any variations thereof in this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0049] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0050] like Figure 1 As shown, the embodiment of the present application provides a pedestrian re-identification method based on posture estimation, comprising the following steps:

[0051] Step 101 : acquiring original modal data, where the original modal data includes visible light modality and infrared modality.

[0052] In person re-identification (Re-ID) or related computer vision tasks, obtaining raw modality data is the first step. Raw modality data refers to the unprocessed data directly obtained from the sensor or data source, which contains all the basic information required for the task.

[0053] In the scenarios involving visible light modality and infrared modality in the embodiments of the present application, the most common sources of visible light modality data are various types of cameras, including but not limited to surveillance cameras, smartphone cameras, professional video cameras, etc. These cameras can capture visible light information in the scene and convert it into digital images or videos. Existing image databases may also contain a large amount of visible light modality data, which can be used to train, verify or test pedestrian re-identification models.

[0054] Visible light modal data usually contains rich color, texture and detail information, which is crucial for distinguishing different pedestrians. However, visible light modal data is also easily affected by factors such as lighting changes, shadows, and occlusions.

[0055] Thermal imaging cameras can capture infrared radiation emitted by objects and convert it into visual images. These images are usually presented in grayscale, indicating the temperature differences in different areas. Infrared cameras are similar to thermal imaging cameras, but they have higher resolution and a wider range of application scenarios.

[0056] Infrared modality data is insensitive to lighting variations and shadows, making it particularly useful at night or in low-light conditions. However, infrared images typically lack the color and texture details found in visible light images, which can pose a challenge for certain types of analysis.

[0057] In the embodiments of the present application, in order to effectively fuse the information of the visible light modality and the infrared modality, it is usually necessary to synchronously acquire the data of the two modalities. Specifically, by using a synchronization device, a visible light camera and an infrared camera are installed at the same location, and it is ensured that they can capture images of the same scene at the same time. For the data captured by different devices, they are synchronized by timestamps, that is, the data of the two modalities are ensured to be as close as possible in time.

[0058] After obtaining the raw modality data, some preprocessing steps can also be performed to improve the quality and applicability of the data, including image cropping, resizing, normalization, etc.

[0059] Step 102 , extracting modal features of exclusive features from the visible light modality and the infrared modality, respectively, to obtain a visible light modal feature map and an infrared modal feature map.

[0060] Modal feature extraction of the visible light modality and the infrared modality with their own features is an important step. This step aims to extract useful information from the visible light modality and the infrared modality for subsequent tasks (such as pedestrian recognition, classification or matching).

[0061] For visible light modal data (usually RGB images), feature extraction can be achieved through a variety of methods, including using pre-trained CNN models (such as ResNet, VGG, MobileNet, etc.) as feature extractors, inputting the original RGB image into CNN, and extracting features layer by layer through the network's convolutional layer, pooling layer and other structures, and finally obtaining high-level feature maps. These feature maps usually contain rich color, texture and shape information, which is of great significance for distinguishing different pedestrians.

[0062] In addition to global features, local features (such as SIFT, SURF, etc.) can also be extracted. These features are robust to image transformations such as rotation and scaling. However, for the task of person re-identification, global features extracted by CNN are usually more commonly used and effective.

[0063] The feature extraction process for infrared modality data (usually grayscale images or thermal images) is similar to that for visible light modality, but there are also some special considerations. Infrared images can also be fed directly into a pre-trained CNN model as input for feature extraction, but due to the visual differences between infrared images and visible light images (such as lack of color information), the CNN model may need to be adjusted or retrained to adapt to infrared data.

[0064] When designing or adjusting the CNN model, the characteristics of infrared images, such as temperature distribution, edge information, etc., can be considered to better capture the useful information in infrared images. In some cases, in order to further improve the performance, the features of the infrared modality and the visible light modality are fused, which usually occurs after feature extraction and is fused at the feature vector or feature map level.

[0065] After the above feature extraction process, we can obtain exclusive feature maps corresponding to the visible light modality and infrared modality. These feature maps not only contain the exclusive information of each modality, such as color and texture information of the visible light modality, temperature and edge information of the infrared modality, but also undergo network abstraction and compression, removing redundant information and retaining the key information that is most beneficial to subsequent tasks.

[0066] In subsequent steps, these feature maps will be used for further processing, such as feature fusion, feature weighting, distance measurement, etc., to achieve the goal of pedestrian re-identification.

[0067] Step 103, based on the multi-head self-attention mechanism, the modal features of the common features of the visible light modal feature map and the infrared modal feature map are extracted to obtain a shared feature map; the shared feature map integrates the exclusive features of the visible light modal feature map, the exclusive features of the infrared modal feature map, and the common features of the visible light modal feature map and the infrared modal feature map.

[0068] The modal feature extraction of common features of visible light modal feature map and infrared modal feature map is performed based on the Multi-Head Self-Attention Mechanism, aiming to fuse the complementary information from different modalities to obtain a richer and more robust shared feature map.

[0069] The multi-head self-attention mechanism is one of the core components of the Transformer model. It allows the model to capture rich contextual information in different representation subspaces by processing multiple self-attention subspaces in parallel. Each self-attention head can focus on information at different positions in the input sequence and integrate this information through weighted summation.

[0070] In the embodiment of the present application, the visible light modal feature map and the infrared modal feature map are used as the input of the multi-head self-attention mechanism. Before input, the feature map is converted into a sequence form suitable for self-attention mechanism processing, for example, the two-dimensional feature map is converted into a one-dimensional sequence by reshape operation. In the multi-head self-attention mechanism, each self-attention head independently calculates the similarity of query, key and value, and generates a new representation by weighted summation. These new representations contain information in each modality and realize cross-modal information interaction through the self-attention mechanism.

[0071] After being processed by the multi-head self-attention mechanism, the outputs of different self-attention heads are concatenated and reduced in dimension through a linear transformation layer to obtain a fused shared feature map. This shared feature map not only contains the exclusive features of the visible light modality and the infrared modality, but also captures the common features between the two modalities through the self-attention mechanism. In the process of generating the shared feature map, some additional operations or mechanisms are introduced to enhance the fusion effect of the features, such as attention weighting, residual connection, etc.

[0072] The multi-head self-attention mechanism can process information in multiple subspaces in parallel, improving the computational efficiency and feature expression ability of the model. Through cross-modal information interaction and fusion, the shared feature map can capture more comprehensive and robust target features. This method has potential advantages in dealing with pedestrian re-identification tasks in complex scenes and lighting changes.

[0073] Specifically, N stages of residual processing are performed on the visible light modal feature map and the infrared modal feature map. In each stage, the input feature map is processed through one or more convolutional layers (or other types of layers, such as bottleneck blocks, etc.), and then the processed result is residually connected (i.e., added) with the original input to ensure the flow of information and the propagation of gradients.

[0074] After each stage of residual processing in the first N-1 stages, a multi-scale feature aggregation is performed. The purpose of multi-scale feature aggregation is to extract information from feature maps of different scales and integrate it into the current feature map. It usually involves using convolution kernels or pooling operations of different sizes to capture contextual information of different scales, and then merging this information in some way (such as splicing, addition, etc.).

[0075] After the N-1th multi-scale feature aggregation, the multi-head self-attention mechanism is used for feature extraction. First, the aggregated feature map is convolved to generate three vectors: query, key, and value. These three vectors represent the representation of different positions in the feature map and are used for subsequent self-attention calculations.

[0076] Then, the similarity matrix is ​​obtained by calculating the dot product of the query and the key. Each element in the similarity matrix represents the similarity between two locations in the feature map.

[0077] Next, the similarity matrix is ​​normalized using the softmax function to obtain the attention weights. These weights represent the contribution of each position in the feature map to other positions.

[0078] Finally, the attention weight is weighted and summed with the value to obtain a weighted feature map. Each position in the weighted feature map is obtained by weighted aggregation based on the information of other positions, so it contains richer contextual information.

[0079] The weighted feature map is linearly transformed through a convolutional layer to adjust its dimension and feature representation. Then, the result of this linear transformation is residually connected (i.e., added) with the result of the N-1th multi-scale feature aggregation to obtain the final shared feature map.

[0080] Through the above process, the feature maps of the visible light modality and infrared modality undergo multiple stages of residual processing and multi-scale feature aggregation, and are finally integrated into a shared feature map in the feature extraction stage based on the multi-head self-attention mechanism. This shared feature map not only contains the exclusive features of the two modalities, but also captures the common features across modalities through the self-attention mechanism, and maintains the continuity and stability of the information through residual connections. Such feature maps will have stronger robustness and expressiveness for subsequent tasks such as pedestrian re-identification.

[0081] Step 104, processing the original modal data based on the posture estimation to obtain a key point heat map of the original modal data; performing score map processing on the key point heat map to obtain a score map and score confidence of the key point heat map.

[0082] To process the raw modal data based on pose estimation and generate key point heat maps and their score maps and score confidences, it is necessary to select or train a pose estimation model, such as a deep learning-based model such as OpenPose, PoseNet, HRNet, etc.

[0083] The raw modal data is processed using a posture estimation model to detect key points of the human body in the image, such as the head, shoulders, elbows, wrists, etc. For each detected key point, a heat map is generated. A heat map is usually a two-dimensional array (matrix) that is the same size as the input image or scaled proportionally. The value of each pixel represents the probability or "heat" of that location being a key point, and the heat map is visualized to intuitively see the location of the key point and its uncertainty.

[0084] The score map is a further processing of the heat map, where the score of each key point represents the confidence or accuracy of the key point detection. The calculation method of the score can be determined based on the statistical characteristics of the heat map, such as the peak value, the energy of the area around the peak value, etc. The score of each key point is represented on the image in some form, such as color depth, digital label, etc., to form a score map.

[0085] Confidence is a measure of the accuracy of detected keypoints. A high confidence level indicates that the model is very certain about the keypoint location. Confidence can be calculated in a variety of ways, such as based on the comparison of the peak value of the heatmap with the surrounding pixel values, based on the internal mechanism of the model, such as Softmax output, etc. Based on the confidence level, the detected keypoints can be screened or corrected to improve the accuracy and robustness of subsequent processing, such as action recognition, posture tracking, etc.

[0086] The processed images, including heat maps, score maps, and score confidence levels, are displayed for analysis and evaluation. The results can be further processed as needed, such as smoothing, outlier removal, etc.

[0087] Through the above steps, the original modal data can be processed based on posture estimation, and the key point heat map, score map and its score confidence can be obtained, thereby providing strong support for subsequent analysis and application.

[0088] In an embodiment of the present application, the original modal data is processed through a posture estimation model. The posture estimation model includes a Pose HRNet model and its pre-trained weights. The Pose HRNet model includes a weighted branch and a parallel branch. The original modal data is input into the Pose HRNet model, and N feature maps with different resolutions are obtained through the weighted branches. Information is exchanged through parallel branches, and a high-resolution feature map is generated through multi-scale feature aggregation. Feature extraction is performed on the high-resolution feature map to obtain a key point heat map of the original modal data.

[0089] Specifically, in pose estimation, the Pose HRNet model and its pre-trained weights are used to process the original modality data. The Pose HRNet model is specially designed to maintain high-resolution feature representation and enhance the learning of multi-scale features through parallel branches and weighted branches.

[0090] The Pose HRNet model generates accurate key point heat maps by maintaining a high-resolution representation from the beginning to the end of the network and gradually fusing multi-scale features. The model mainly consists of weighted branches and parallel branches, where the weighted branches are responsible for extracting feature maps of different resolutions from the input data. These feature maps are generated by gradually downsampling (such as using convolution or pooling layers with a step size of 2), and the output of each branch has a different spatial resolution.

[0091] Parallel branches are parallel to weighted branches. Pose HRNet maintains high-resolution feature maps through a series of parallel convolution blocks and exchanges information between these parallel branches. This design allows the model to process information of different scales simultaneously and enhance feature representation through feature fusion.

[0092] In Pose HRNet, information exchange refers to the process of transferring and fusing feature information between parallel branches of different resolutions. Specifically, the model regularly upsamples high-resolution feature maps with lower-resolution feature maps to match the spatial dimensions, and then adds or concatenates them with the original low-resolution feature maps. In this way, the lower-resolution feature maps can obtain high-resolution spatial information, while the high-resolution feature maps can obtain richer semantic information.

[0093] Through the information exchange between weighted branches and parallel branches, Pose HRNet is able to generate feature maps of multiple resolutions. In the later stages of the model, these multi-scale feature maps are aggregated to generate the final high-resolution feature map. This process usually involves upsampling, adding or concatenating the feature maps to ensure that the final feature map contains rich semantic information while maintaining high-resolution spatial details.

[0094] In pose estimation, high resolution usually refers to feature maps with similar or the same spatial resolution as the input image or video frame. Since pose estimation requires pixel-accurate keypoint localization, maintaining high resolution is crucial for generating accurate keypoint heatmaps.

[0095] The final high-resolution feature map is fed into one or more convolutional layers to generate keypoint heatmaps. Each heatmap corresponds to a keypoint on the human body, such as the head, shoulder, wrist, etc., where the pixel value represents the probability or "heat" of the location being the corresponding keypoint.

[0096] Specifically, the input of Pose HRNet is the original image, and the output is the heat map of the key points of the original image. PoseHRNet is based on ResNet for feature extraction. The main feature of Pose HRNet is that it always maintains a high-resolution branch during the feature map extraction process. High resolution means that the feature map size is not reduced. In the four-stage feature extraction process, in addition to using downsampling to generate a feature map with a higher dimension and smaller size (low resolution), the embodiment of the present application also maintains a weighted branch that performs 1×1 convolution, stride 1, and fills 0. This weighted branch keeps the shape of the feature map unchanged, thereby retaining the high-resolution feature map. The embodiment of the present application performs four stages of residual processing, so after the four stages of feature extraction, 4 feature maps with different resolutions will be generated. In addition, when generating feature maps in these four stages, Pose HRNet will exchange information between parallel branches, receive information from parallel branches, and obtain the feature map of the stage output through multi-scale feature fusion. Then, a high-resolution feature map is generated for the last four feature maps with different resolutions through multi-scale feature aggregation, and finally this high-resolution feature is extracted through the last layer of features to obtain the output heat map. The channel dimension of the heat map output after the posture estimation model is processed in the embodiment of the present application is 17.

[0097] In summary, by processing the original modal data through the Pose HRNet model and its pre-trained weights, key point heat maps can be efficiently generated, providing strong support for subsequent posture estimation and analysis.

[0098] Furthermore, the embodiment of the present application adjusts the size of the key point heat map to the same size as the shared feature map through bilinear interpolation, divides the key point heat map into groups according to the Pose HRNet model and its pre-trained weights, sums the channel response values ​​contained in each group of key point heat maps, generates a single-channel heat map of the group, calculates the mean of the maximum channel response values ​​contained in the group, connects each group of single-channel heat maps along the channel dimension, and performs normalization to obtain a score map of the key point heat map, and connects the mean of the maximum channel response values ​​of each group along the channel dimension to obtain the score confidence.

[0099] During the processing of the keypoint heatmap, in order to generate the score map and score confidence, the keypoint heatmap is resized to the same size as the shared feature map through bilinear interpolation. This is to ensure the consistency of the spatial correspondence between feature maps in subsequent processing.

[0100] According to the design of the Pose HRNet model or the category of key points, the key point heat map is divided into groups. Usually, each key point, such as head, shoulder, elbow, etc., or a set of key points, such as limbs, torso, etc., will be divided into a group.

[0101] In the Pose HRNet model, the key point heat map is usually composed of multiple channels, each channel corresponds to the position prediction of a key point. The channel response value refers to the value of each pixel in each channel, which represents the probability or "heat" of the pixel position corresponding to the key point. These values ​​are directly obtained from the output of the Pose HRNet model without additional calculation.

[0102] For each set of keypoint heatmaps, the channel response values ​​contained in it are summed. This operation is usually performed in the spatial dimension, that is, at the pixel level, to merge the information of multiple channels into a single channel heatmap to represent the overall position prediction of the set of keypoints. The summation operation can be implemented by simple element-wise addition.

[0103] After obtaining the single-channel heat map of each group, it is necessary to calculate the maximum response value of each pixel position in the channel contained in the group. This is usually achieved by comparing all channel values ​​at the same position and taking the maximum value as the maximum response value of the position. This operation is performed independently for each pixel in the spatial dimension.

[0104] For all channels in each group, the mean of the maximum response values ​​is calculated. This mean can be used as a confidence indicator for the key point detection of this group, reflecting the overall reliability of the model's prediction of the key points of this group.

[0105] The single-channel heatmaps obtained for each group are concatenated along the channel dimension to form a new multi-channel heatmap, namely the score map. This score map contains the position prediction information of all group key points. Then, the score map is normalized to ensure that its value is within a reasonable range (such as between 0 and 1) for subsequent processing and analysis.

[0106] At the same time, the mean of the maximum channel response values ​​of each group is also concatenated along the channel dimension to obtain the fractional confidence. This fractional confidence is a vector whose length is the same as the number of groups, and each element represents the confidence of the key point detection of the corresponding group.

[0107] Through the above steps, the score map and score confidence are obtained from the key point heat map. This information is very useful for subsequent tasks such as posture estimation and action recognition.

[0108] This part processes the heat map output by Pose HRNet and extracts the key point information in the heat map. The input of this part includes the heat map output by Pose HRNet. The output of this part is the score map, score confidence and key point position of the heat map.

[0109] Specifically, the heat map size is first adjusted to the same size as the backbone network through bilinear interpolation, that is, the process from obtaining the original modal data to outputting the shared feature map, and then the heat map is reshaped to the shape (n, c, h*w), that is, the last two dimensions are flattened as the response value of channel c in order to process the response value of each channel.

[0110] For each heat map, find the index with the largest response value and the maximum response value in each channel and store the column and row position of the maximum response value. Specifically, use the index modulo the width to calculate the column position of the maximum response value in each channel, and use the index integer to divide the width to calculate the row position of the maximum response value in each channel.

[0111] The 17 channels of the heat map are divided into 14 groups, of which 1-4 channels are one group, and the remaining 5-16 channels are divided into 13 groups. This is divided according to the connection point grouping of the data set used by the pre-trained model. The response values ​​of the channels contained in each group in the heat map are summed to generate a single-channel heat map of the group, and the mean of the maximum response values ​​of the channels contained in the group is calculated. Finally, all the corresponding heat maps are generated and connected along the channel dimension for normalization to become a score map, and the mean of the maximum response value of each group is connected along the channel dimension to become the confidence. Finally, the score map, confidence and column and row positions of the maximum response value are output. The position of the maximum response value here is the key point position. Since two branch embedding channels are generated in the backbone network through diversified embedding expansion, they are restored and connected in series with the original feature map, making the batch size after diversified embedding expansion become 3 times the original. In order to adapt to this situation, the input image is connected to itself 3 times in the 0th dimension to obtain a batch size of 3 times, which enables the posture estimation model to generate a heat map with a batch size of 3 times. The downsampling process reduces the resolution of the feature map to half of its original resolution, that is, the width and height are each reduced to half of the original resolution. The upsampling process doubles the resolution through nearest neighbor sampling.

[0112] Step 105, perform multi-head batch normalization weighted processing on the score map, score confidence and shared feature map to obtain a feature vector set; perform distance metric matching on the feature vector set to obtain a pedestrian re-identification result.

[0113] In the pedestrian re-identification task, the score map, score confidence and shared feature map are fused, and the result of pedestrian re-identification can be obtained through multi-head batch normalization weighting and distance metric matching.

[0114] First, the score map, score confidence and shared feature map need to be fused. Since these feature maps or vectors have different properties, such as spatial resolution, number of channels, physical meaning, etc., direct fusion may not be the best choice. Therefore, the score map and score confidence are mapped to the same spatial or channel dimension as the shared feature map through a convolutional layer, a fully connected layer or an embedding layer.

[0115] The fused feature maps are weighted according to the fractional confidence, which is achieved by multiplying the corresponding feature map elements with the fractional confidence as the weight coefficient to enhance the importance of high-confidence features. The weighted feature maps are normalized to ensure that the features between different samples are comparable.

[0116] After the above processing, one or more feature vectors can be obtained, which together constitute a feature vector set. If multiple pedestrians in an image or video frame are processed, each pedestrian will correspond to a feature vector.

[0117] After obtaining the feature vector set, the similarity between different pedestrians is calculated using distance measurement methods such as Euclidean distance and cosine similarity. Generally, the goal of the person re-identification task is to find the same pedestrian that appears under different camera perspectives. Therefore, the feature vector of the query pedestrian can be compared with the feature vectors in the candidate pedestrian set to find the candidate pedestrian that is most similar to the query pedestrian as the re-identification result.

[0118] According to the results of the distance measurement, the candidate pedestrians are sorted, and the pedestrian with the smallest distance or the highest similarity is selected as the re-identification result.

[0119] Furthermore, the score map, score confidence and shared feature map are subjected to multi-head batch normalization weighted processing to obtain a set of feature vectors, including: extracting elements in the score map and the shared feature map, multiplying the score map and the shared feature map element-wise, and summing them in height and width dimensions to obtain a local feature vector; averaging the height and width dimensions of the shared feature map to obtain an average feature vector, and processing the shared feature map through adaptive maximum pooling to obtain an adaptive maximum pooling result, adding the adaptive maximum pooling result to the average feature vector to obtain a global feature vector; performing multi-head batch normalization processing on the local feature vectors and the global feature vectors, and weighting the local feature vectors and the global feature vectors by score confidence to obtain a set of feature vectors.

[0120] The score map, score confidence and shared feature map are combined to generate a set of feature vectors. This process involves the extraction of local features, global features, multi-head batch normalization and weighted processing.

[0121] First, the corresponding elements are extracted from the score map and the shared feature map. Then, the score map and the shared feature map are multiplied at the element level. This multiplication operation can be regarded as a weighting operation, in which the score map provides the importance weight of each position. After that, the sum is performed in the height and width dimensions, that is, the weighted feature values ​​at all spatial positions are accumulated to obtain a one-dimensional local feature vector, which reflects the features in the shared feature map associated with the high response area of ​​the score map.

[0122] The height and width dimensions of the shared feature map are averaged, that is, the average of all pixel values ​​on each channel is calculated to obtain a mean feature vector. This vector provides global statistical information of the shared feature map. Adaptive max pooling is applied to the shared feature map, and its output size is predetermined. This operation extracts the most representative features from the shared feature map, which may not be continuous in spatial position, but have the largest response value in their respective channels. The result of adaptive max pooling is added to the mean feature vector to obtain a global feature vector that combines global statistics and the most representative features.

[0123] The fractional confidence is used to weight the local feature vectors and the global feature vectors. The fractional confidence reflects the reliability of the model in detecting key points, so it can be used as a weighting coefficient to emphasize those more reliable feature vectors. Specifically, the fractional confidence can be used directly as a weight, or through some transformation, such as normalization, nonlinear transformation, etc., as a weight, and multiplied with the corresponding feature vector.

[0124] Finally, the weighted local feature vectors and global feature vectors are combined to form a feature vector set. This set contains the feature representation extracted from the original data, which integrates local and global information and is weighted, and can be used in subsequent pedestrian re-identification tasks.

[0125] Specifically, this part combines the extracted key point information with the feature map of the backbone network by weight. The input includes the score map extracted from the heat map, the confidence, and the feature map output by the backbone network. _ maps. This part uses the score map and confidence as weights to perform multi-head batch normalization weighted processing to convert the feature _ Maps are processed into a feature vector set output. The first 13 of the output feature vector sets are designed as local feature vectors.

[0126] For the local feature vector to be processed: extract the score of the i-th channel _map and add a new dimension by unsqueeze(1) to make it shape (batch_size, 1, height, width). Use repeat() to repeat it to (batch_size, 2048, height, width), matching the number of channels and dimensions of feature_maps. The score map is element-wise multiplied with the feature map and then summed over the height and width dimensions to get a shape of (batch_size, 1, height, width). _ size, 2048); the last one in the feature vector output is designed as the global feature vector.

[0127] Processing global feature vectors: _ The height and width dimensions of the maps are averaged to get a batch shape of ( _ size, 2048). The feature_maps are reduced to shape (batch_size, 2048, 1, 1) by adaptive max pooling. Squeeze(-1) is used to change its shape to (batch_size, 2048, 1, 1). _ size, 2048). The adaptive max pooling result is added to the previously calculated average feature vector to obtain the global feature vector. Multi-head batch normalization is used on the vector set and the local and global feature vectors are weighted by the fractional confidence to obtain the final output feature vector set.

[0128] In the process of model training, in order to improve the generalization ability and robustness of the model, the embodiment of the present application adopts the following data enhancement and preprocessing strategies. First, before loading the image, uniform identity sampling is used to ensure that the distribution of pedestrian identities in each batch is roughly the same to avoid the influence of sample bias on model training. Then, the loaded image will be filled with 10 pixels around the image, and the image will be adjusted to a size suitable for the input model by random cropping (Random Crop), thereby retaining more original image information. To further enhance the training data set, the image will be randomly flipped horizontally (Random Horizontal Flip), which increases the diversity of samples by randomly flipping the image, helping the model to better learn different image transformations. After the image is converted into a tensor, it will be normalized so that the channel values ​​of the image data are adjusted to a specific range, thereby accelerating the convergence of model training. After normalization, the random erase (Random Erase) method is applied to simulate the situation where the target is blocked or the image is incomplete in actual shooting. This operation erases a rectangular area at a random position in the image, which can effectively enhance the robustness of the model to local missing information and ensure that the data distribution after normalization is not destroyed. In addition, random channel erasure (ChannelRandom Erase) is introduced. This operation is similar to random erasure, but it processes a certain channel of the image. By specifying the probability to select a rectangular block, a channel in it is replaced with a specific value, thereby reducing the difference between the grayscale single channel of the infrared image and the RGB three channels of the visible light image, ensuring that the color information is not over-utilized. Finally, through channel replacement (Channel Exchange), a channel is randomly selected and the values ​​of the remaining channels are all replaced with the values ​​of the selected channel. This operation aims to increase the robustness of the model to information from different channels, thereby completing the entire training data processing process.

[0129] In the forward propagation process, the image is input into the model to extract features. Since the loss function must be calculated with the true label label during training, the output results during training cannot have the same output features as those during testing. It is necessary to obtain the probability classifications of each identity in the image through a linear activation layer, and calculate the loss with the true label of the input image using the loss function. The model is optimized by calculating the gradient of the loss through back propagation.

[0130] The training of the model is mainly based on the calculation of gradients by loss functions for optimization. The main loss functions used in model training are triplet loss and cross entropy loss. Triplet loss (Triplet Loss), for different images of the same identity, the distance between features should be as small as possible; for different images of different identities, the distance between features should be as large as possible. Cross-Entropy Loss (Cross-Entropy Loss), the features of the model should make the predicted probability distribution similar to the true (or dataset) probability distribution at a macro level. In order to improve the simulation effect, calculating the cross entropy loss can adjust the optimization direction of the model at a macro level. In actual use, the triplet loss is calculated for the most difficult positive samples and the easiest negative samples mined by various probabilities and true labels, the cross entropy loss is calculated for the feature map and the true label, and the branch embedding is separated from the feature map to calculate the CPM Loss.

[0131] Specifically, the embodiment of the present application processes the original modal data through the posture estimation model, including optimizing the posture estimation model by calculating the gradient through the composite loss function, and the composite loss function includes orthogonal loss, center guided mining loss, cross entropy loss, triple loss and IDE loss, wherein the orthogonal loss L ort In order to perform average pooling after diversifying the feature extraction results, we get the orthogonal loss L ort ; After performing diversified feature expansion on the feature extraction results, embedding is generated, and the center-guided mining loss L cpm is the constraint on embedding; cross entropy loss L cre To diversify the feature extraction results, batch normalization is performed and the cross entropy loss L is obtained by calculating the cross entropy loss L with the true label. cre ; Triplet loss L tri In order to perform batch normalization after diversifying the feature extraction results, similar samples are screened by the true labels to obtain the most difficult positive samples and the easiest negative samples. The triplet loss L is calculated by the most difficult positive samples, the easiest negative samples and the true labels. tri IDE loss L ide It is calculated by the set of feature vectors, fractional confidence and true labels.

[0132] Specifically, the purpose of orthogonal loss is to promote orthogonal or decorrelated features generated by the feature extractor, which helps the model learn more diverse feature representations. After feature extraction, diversified feature expansion is performed, followed by average pooling operations to reduce spatial dimensions and retain important features. Orthogonal loss usually calculates the covariance matrix between the expanded features and encourages it to be close to the unit matrix, that is, the features are as orthogonal as possible.

[0133] The purpose of the center guided mining loss is to impose certain constraints on the embedding vector to improve certain properties of the embedding space. The embedding vector is generated after feature expansion, and the specific form of the center guided mining loss depends on the desired constraints.

[0134] The constraint attributes in the embodiment of the present application include:

[0135] Property 1: The generated embeddings should be as diverse as possible to effectively learn informative feature representations. This means that the distance between the generated embeddings and the original embeddings needs to be widened to learn diverse features and mine diverse cross-modal clues.

[0136] Property 2: The generated embedding should help reduce the modality difference between visible and infrared images. This means that the distance between the embedding generated from the visible light modality and the original infrared embedding needs to be shortened. At the same time, the distance between the embedding generated from the infrared modality and the original visible light embedding also needs to be shortened.

[0137] Property 3, the intra-class distance should be smaller than the inter-class distance. According to Property 2, it narrows the distance between the generated embedding and the original embedding, which may cause the embeddings of different classes to become close. Therefore, it is necessary to keep the intra-class distance smaller than the inter-class distance.

[0138] The purpose of cross entropy loss is to measure the difference between the model prediction and the true label. It is one of the most commonly used loss functions in classification tasks. Batch normalization is performed after feature expansion, and then the normalized features are compared with the true labels to calculate the cross entropy loss. The true label is usually obtained through the labeling process, that is, manually specifying the category to which each sample belongs, where the true label is obtained through the data labeling process.

[0139] The purpose of triplet loss is to learn an embedding space so that the distance between samples of the same class is smaller than the distance between samples of different classes. Batch normalization is performed after feature expansion, and then samples of the same class are filtered according to the true labels, and the hardest positive samples (different samples that are most similar to the anchor samples but belong to the same class) and the easiest negative samples (different class samples that are most different from the anchor samples) are found. Then the triplet loss is calculated, which maximizes the distance between the anchor and the negative samples, while minimizing the distance between the anchor and the positive samples. The true labels are used to determine which samples belong to the same class, so that the positive and negative sample pairs can be correctly selected.

[0140] The purpose of IDE loss is to improve the model's ability to distinguish different individuals, which is a loss function designed for specific tasks such as person re-identification. The specific form of IDE loss may vary from task to task, but usually involves the calculation of feature vector sets, score confidences, and true labels. It may be combined with cross-entropy loss and other task-specific constraints to better suit the needs of tasks such as person re-identification. In IDE loss, the true labels are used to ensure that the model can correctly learn the distinction between different individuals and optimize its output to match the true labels.

[0141] To summarize, the formula for the composite loss function is as follows:

[0142] L=λ1L ort +λ2L cpm +λ3L cre +λ4L tri +λ5L ide

[0143] Among them, L ort represents the orthogonal loss, L cpm represents the center-guided mining loss, L cre represents the cross entropy loss, L tri represents the triplet loss, L ide represents IDE loss; λ1, λ2, λ3, λ4 and λ5 represent L ort , L cpm , L cre , L tri and L ide The weight of each part depends on the importance of each loss. The weight ratio taken in the embodiment of the present application is λ1:λ2:λ3:λ4:λ5=1:1:1:1:1.

[0144] In the embodiment of the present application, the proposed method is evaluated using the dataset SYSU-MM01 and compared with the two-stream model of similar feature-level methods in recent years. The experimental results are shown in the following table:

[0145] Table 1

[0146]

[0147] From the experimental results, the re-identification method of the embodiment of the present application has a leading level among similar methods in terms of Rank-1 and mAP indicators.

[0148] Effect on occlusion problem: Occlusion is defined as the situation where the target is blocked by the environment or other targets or does not fall completely in the image. In order to verify the effectiveness of the proposed method for occlusion problem, the SYSU-MM01 test set is used to query identity images and divide them into occlusion, side, and back. In the case that it is obviously not the front view, the back and side are distinguished by whether the shoulders can be seen in the image. If the shoulders can be seen, it is classified as a back image, otherwise it is classified as a side image. The experimental results are shown in the following table:

[0149] Table 2

[0150]

[0151] In order to verify the effectiveness of the method of the embodiment of the present application for the actual occlusion situation in the SYSU-MM01 data set, the images with occluded Part, Side, and Back angles are queried. From the results, it can be seen that for occluded situations, the performance of the baseline method is significantly reduced compared to all image types, while the method of the embodiment of the present application has improved performance on the occlusion problem compared to the baseline method. Specifically, the method of the embodiment of the present application has a 0.30% performance improvement in Rank-1 and a 0.10% performance improvement in mAP compared to the baseline method, verifying the effectiveness of the proposed method in dealing with occlusion problems. It can also be seen that the proposed method has improved performance for different shooting angles. Note that when querying the filtered images, the mAP indicators of the baseline method and the proposed method are both high. The possible reason is that there are fewer images with the same identity in the filtered gallery.

[0152] Effect on different occlusion levels: In order to verify the effectiveness of the method of the embodiment of the present application on different occlusion levels, the random erasing method is used to randomly erase all images in the query set and the gallery, and the size of the erasure block is controlled to simulate different degrees of occlusion. The erasure rate is the ratio of the length and width of the erasure block to the length and width of the image. Random erasing is used on the SYSU-MM01 dataset to simulate experiments with erasure rates of 0%, 10%, 20%, 30%, 40% and 50% to simulate different occlusion levels. The experimental results are shown in the following table:

[0153] Table 3

[0154]

[0155] As can be seen from the table, for Rank-1, the performance improvement of the method of the embodiment of the present application over the baseline method at 0%, 10%, 20%, 30%, 40%, and 50% erasure rates is 0.13%, 0.20%, 0.24%, 0.37%, 0.26%, and 0.72% respectively; for mAP, the performance improvement at 0%, 10%, 20%, 30%, 40%, and 50% erasure rates is 0.01%, 0.05%, 0.25%, 0.40%, 0.60%, and 0.82% respectively. From the above results, it can be seen that in the case of severe occlusion, the performance improvement of the method of the embodiment of the present application over the baseline method is more obvious.

[0156] like Figure 2 As shown, the embodiment of the present application provides a pedestrian re-identification device based on posture estimation, including a data acquisition module 201, a feature extraction module 202, a posture estimation module 203 and a re-identification module 204, wherein:

[0157] The data acquisition module 201 is configured to acquire raw modality data, where the raw modality data includes visible light modality and infrared modality;

[0158] The feature extraction module 202 is configured to perform modal feature extraction of exclusive features on the visible light modality and the infrared modality, respectively, to obtain a visible light modal feature map and an infrared modal feature map; perform modal feature extraction of common features on the visible light modal feature map and the infrared modal feature map based on a multi-head self-attention mechanism, to obtain a shared feature map; the shared feature map integrates the exclusive features of the visible light modal feature map, the exclusive features of the infrared modal feature map, and the common features of the visible light modal feature map and the infrared modal feature map;

[0159] The posture estimation module 203 is configured to process the original modal data based on the posture estimation to obtain a key point heat map of the original modal data; perform score map processing on the key point heat map to obtain a score map and score confidence of the key point heat map; perform multi-head batch normalization weighted processing on the score map, the score confidence and the shared feature map to obtain a feature vector set;

[0160] The re-identification module 204 is configured to perform distance metric matching on the feature vector set to obtain a pedestrian re-identification result.

[0161] The pedestrian re-identification device based on posture estimation in the embodiment of the present application can be a computer device, or a component in the computer device, such as an integrated circuit or a chip. The computer device can be a terminal, or a device other than a terminal. Exemplarily, the computer device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted computer device, a mobile Internet device (Mobile Internet Device, MID), an ultra-mobile personal computer (Ultra-Mobile Personal Computer, UMPC), a netbook or a personal digital assistant (Personal Digital Assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (Personal Computer, PC), etc., which is not specifically limited in the embodiment of the present application.

[0162] The pedestrian re-identification device based on posture estimation provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented in the embodiment of the pedestrian re-identification method based on posture estimation are not described here.

[0163] The present application also provides a computer device, such as Figure 3 As shown, the computer device includes a processor 301 and a memory 302. The memory 302 stores programs or instructions that can be executed on the processor 301. When the program or instructions are executed by the processor 301, the various steps of the above-mentioned pedestrian re-identification method based on posture estimation are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0164] It should be noted that the computer device in the embodiment of the present application includes the above-mentioned mobile computer device and non-mobile computer device.

[0165] The memory 302 can be used to store software programs and various data. The memory 302 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 302 may include a volatile memory or a non-volatile memory, or the memory 302 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 302 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0166] The processor 301 may include one or more processing units; optionally, the processor 301 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 301.

[0167] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned pedestrian re-identification method embodiment based on posture estimation is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0168] An embodiment of the present application also provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned pedestrian re-identification method embodiment based on posture estimation, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0169] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0170] An embodiment of the present application also provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned pedestrian re-identification method embodiment based on posture estimation, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0171] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0172] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A pedestrian re-identification method based on posture estimation, characterized in that: The method comprises: Acquiring original modality data, wherein the original modality data includes a visible light modality and an infrared modality; Performing modal feature extraction of exclusive features on the visible light modality and the infrared modality respectively to obtain a visible light modal feature map and an infrared modal feature map; Based on the multi-head self-attention mechanism, modal features of common features of the visible light modal feature map and the infrared modal feature map are extracted to obtain a shared feature map; the shared feature map integrates the exclusive features of the visible light modal feature map, the exclusive features of the infrared modal feature map, and the common features of the visible light modal feature map and the infrared modal feature map; The original modal data is processed based on the posture estimation to obtain a key point heat map of the original modal data; the key point heat map is processed by score map to obtain a score map and score confidence of the key point heat map; The score map, the score confidence and the shared feature map are subjected to multi-head batch normalization weighted processing to obtain a feature vector set; and the feature vector set is subjected to distance metric matching to obtain a pedestrian re-identification result.

2. The method for pedestrian re-identification based on posture estimation according to claim 1, characterized in that: The method of extracting shared features of the visible light modal feature map and the infrared modal feature map based on a multi-head self-attention mechanism to obtain a shared feature map includes: N stages of residual processing are performed on the visible light modal feature map and the infrared modal feature map, and multi-scale feature aggregation is performed after each stage of residual processing in the first N-1 stages. After the N-1th multi-scale feature aggregation, feature extraction is performed based on a multi-head self-attention mechanism to obtain a feature extraction result, and diversified feature expansion is performed on the feature extraction result, followed by the Nth stage of residual processing to obtain a shared feature map.

3. The method for pedestrian re-identification based on posture estimation as claimed in claim 2, characterized in that: The feature extraction is performed based on the multi-head self-attention mechanism to obtain a feature extraction result, and the feature extraction result is subjected to the Nth stage of residual processing after diversified feature expansion to obtain a shared feature map, including: The result of the N-1th multi-scale feature aggregation is convolved to obtain three vectors: query, key, and value. The similarity matrix is ​​obtained by calculating the dot product of the query and the key. The similarity matrix is ​​normalized by softmax to obtain the attention weight. The attention weight and the value are weightedly summed to obtain a weighted feature map, the weighted feature map is linearly transformed, and a residual connection is performed with the result after the N-1th multi-scale feature aggregation to obtain a shared feature map.

4. The method for pedestrian re-identification based on posture estimation as claimed in claim 2, characterized in that: The processing of the original modal data based on the posture estimation to obtain a key point heat map of the original modal data includes: The original modal data is processed by a posture estimation model, the posture estimation model includes a Pose HRNet model and its pre-trained weights, the Pose HRNet model includes a weighted branch and a parallel branch, the original modal data is input into the Pose HRNet model, N feature maps with different resolutions are obtained through the weighted branches, exchanged through the parallel branches, a high-resolution feature map is generated through multi-scale feature aggregation, and features are extracted from the high-resolution feature map to obtain a key point heat map of the original modal data, and the high-resolution feature map is a feature map that does not reduce the size of the feature maps with different resolutions.

5. The method for pedestrian re-identification based on posture estimation as claimed in claim 4, characterized in that: The step of performing score map processing on the key point heat map to obtain a score map and score confidence of the key point heat map includes: The size of the key point heat map is adjusted to the same size as the shared feature map through bilinear interpolation, the key point heat map is divided into groups according to the Pose HRNet model and its pre-trained weights, the channel response value and the channel maximum response value of each group of key point heat maps are extracted, the channel response values ​​of each group of key point heat maps are summed respectively to obtain a single-channel heat map of each group, the single-channel heat map of each group is connected along the channel dimension, and normalized to obtain a score map of the key point heat map, the mean of the channel maximum response value of each group of key point heat maps is calculated respectively, and the mean of the channel maximum response value of each group of key point heat maps is connected along the channel dimension to obtain the score confidence.

6. The method for pedestrian re-identification based on posture estimation according to claim 1, characterized in that: The multi-head batch normalization weighted processing is performed on the score map, the score confidence and the shared feature map to obtain a feature vector set, including: Extracting elements from the score map and the shared feature map, multiplying the score map and the shared feature map by the elements, and summing them in height and width dimensions to obtain a local feature vector; Averaging the height and width dimensions of the shared feature map to obtain an average feature vector, processing the shared feature map through adaptive maximum pooling to obtain an adaptive maximum pooling result, and adding the adaptive maximum pooling result to the average feature vector to obtain a global feature vector; Multi-head batch normalization is performed on the local feature vector and the global feature vector, and the local feature vector and the global feature vector are weighted by the fractional confidence to obtain a feature vector set.

7. The method for pedestrian re-identification based on posture estimation according to claim 4, characterized in that: The processing of the original modal data by the posture estimation model includes: The posture estimation model is optimized by calculating the gradient through a composite loss function, and the formula of the composite loss function is as follows: L=λ1L ort +λ2L cpm +λ3L cre +λ4L tri +λ5L ide Among them, L ort represents the orthogonal loss, L cpm represents the center-guided mining loss, L cre represents the cross entropy loss, L tri represents the triplet loss, L ide represents IDE loss; λ1, λ2, λ3, λ4 and λ5 represent L ort , L cpm , L cre , L tri and L ide The weight of The orthogonality loss L ort In order to perform average pooling after diversifying the feature extraction results, we get the orthogonal loss L ort ; After performing diversified feature expansion on the feature extraction results, embedding is generated, and the center guided mining loss L cpm is the constraint on the embedding; the cross entropy loss L cre To diversify the feature extraction results, batch normalization is performed and the cross entropy loss L is obtained by calculating the cross entropy loss L with the true label. cre , the true label is obtained by pre-labeling; the triple loss L tri In order to perform batch normalization after diversifying the feature extraction results, similar samples are screened by the true labels to obtain the most difficult positive samples and the easiest negative samples. The triplet loss L is calculated by the most difficult positive samples, the easiest negative samples and the true labels. tri ; The IDE loss L ide It is calculated by the set of feature vectors, fractional confidence and true labels.

8. A pedestrian re-identification device based on posture estimation, characterized in that: The device comprises: A data acquisition module is configured to acquire raw modality data, wherein the raw modality data includes a visible light modality and an infrared modality; A feature extraction module is configured to perform modal feature extraction of exclusive features on the visible light modality and the infrared modality, respectively, to obtain a visible light modal feature map and an infrared modal feature map; perform modal feature extraction of common features on the visible light modal feature map and the infrared modal feature map based on a multi-head self-attention mechanism, to obtain a shared feature map; the shared feature map integrates the exclusive features of the visible light modal feature map, the exclusive features of the infrared modal feature map, and the common features of the visible light modal feature map and the infrared modal feature map; The posture estimation module is configured to process the original modal data based on the posture estimation to obtain a key point heat map of the original modal data; perform score map processing on the key point heat map to obtain a score map and score confidence of the key point heat map; perform multi-head batch normalization weighted processing on the score map, score confidence and shared feature map to obtain a feature vector set; The re-identification module is configured to perform distance metric matching on the feature vector set to obtain a pedestrian re-identification result.

9. A pedestrian re-identification system based on posture estimation, the system comprising a processor and a memory, wherein a computer program is stored in the memory, characterized in that: The computer program is loaded and executed by the processor to implement the pedestrian re-identification method based on posture estimation as described in any one of claims 1 to 7.

10. A computer-readable storage medium, wherein a computer program is stored in the storage medium, characterized in that: When the computer program is executed by a processor, it is used to implement the pedestrian re-identification method based on posture estimation as described in any one of claims 1 to 7.

Images