A video visible light-infrared pedestrian re-identification method based on human parsing

Abstract

The application discloses a video visible light-infrared pedestrian re-identification method based on human body analysis, and belongs to the technical field of pedestrian re-identification. The method extracts the body shape features which are invariant to modalities through human body analysis, models the high-order structure information of different parts of the pedestrian, and introduces a similarity orthogonal decomposition mechanism to make the appearance and body shape features complementary and collaborative. Compared with the traditional method which only relies on the appearance information which is easily affected by camouflage, view angle and illumination, the system models the fine-grained body dynamic features such as height, length, girth, stride length and center of gravity movement of the individual. On this basis, the appearance and body shape features are decoupled and fused, so that the model has both discriminative texture expression ability and robust structure recognition ability, thereby significantly improving the cross-modal video pedestrian retrieval accuracy. The experimental results verify that the application has superior performance in the cross-device and all-weather pedestrian matching task in the complex monitoring environment, and provides more abundant and stable identity recognition clues for practical application.

Description

Technical Field

[0001] This invention relates to the field of visible light-infrared pedestrian re-identification in video, specifically to a method for visible light-infrared pedestrian re-identification based on human body analysis. Background Technology

[0002] With the widespread adoption of intelligent surveillance equipment and the increasing demand for public safety, numerous camera networks have been deployed in locations such as airports, communities, streets, and campuses. These camera networks typically cover large, non-overlapping areas, generating massive amounts of surveillance video data daily. Analyzing this data not only helps track pedestrian activity patterns to support applications such as target detection, multi-camera target tracking, and crowd behavior analysis, but it can also be used to find lost children. Among these technologies, Person Re-identification (ReID), a key technology, can be traced back to the research scope of multi-target, multi-camera tracking. Through ReID technology, we can achieve more efficient cross-device pedestrian identity matching in complex surveillance scenarios, thereby contributing to the construction of intelligent surveillance and public safety systems.

[0003] Person re-identification (ReID) is a technique that uses computer vision to determine the presence of a specific pedestrian in an image or video sequence. Specifically, ReID aims to identify and match the same pedestrian across multiple video sequences within the non-overlapping field of view of cameras. As an important branch of image retrieval, ReID features cross-camera matching. Its main challenges lie in the differences in image quality from different cameras, variations in viewing angle, inconsistent lighting conditions, and the similarity of pedestrian appearances. In practical applications, such as surveillance videos, high-quality facial images are often unavailable due to low camera resolution or shooting angle limitations. This makes ReID an important alternative when face recognition fails. In recent years, with the rapid development of deep learning technology, the ReID field has achieved significant breakthroughs in feature extraction and matching algorithms, providing strong support for improving the accuracy of cross-camera person matching.

[0004] However, traditional pedestrian re-identification is mainly used in well-lit scenes. In low-light or nighttime conditions, the performance of visible light cameras degrades significantly, or even fails completely. Modern cameras, on the other hand, are generally equipped with dual-modal capabilities (visible light and infrared), with infrared cameras able to capture infrared image information of pedestrians both day and night. This characteristic provides excellent research conditions and technical support for cross-modal pedestrian re-identification. Therefore, in cross-modal ReID tasks, the goal is to take a query image (probe) as input and match it not only in RGB images but also in infrared images, and vice versa. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a video visible light-infrared pedestrian re-identification method based on human body analysis. Existing technologies focus solely on identifying pedestrians' appearance and clothing, information easily disguised or affected by changes in viewing angle and lighting. This invention models body features such as arm swings and shifts in the center of gravity during pedestrian walking videos, effectively utilizing modally invariant body shape features to solve the visible light-infrared modal difference problem. Simultaneously, similarity orthogonal decomposition allows appearance and body shape features to achieve a mutually reinforcing synergistic effect.

[0006] To achieve the above technical solution, the specific steps are as follows:

[0007] S1. Construct a training dataset using pedestrian video sequences in visible light and infrared modes;

[0008] The training dataset includes: a video training dataset and a silhouette image sequence training dataset;

[0009] The construction steps are as follows:

[0010] S1.1 Input pedestrian video sequences in visible light and infrared modes, perform random sampling on the input video sequences to obtain visible light images and infrared images to form a video training dataset;

[0011] The random sampling operation is performed by randomly sampling a preset number of frames from the input pedestrian video sequence in visible light and infrared modes;

[0012] The video training dataset is constructed by randomly sampling a preset number of visible light images. and infrared images The reconstructed visible and infrared mode sequences are used to obtain the video training dataset. ;

[0013] S1.2 Based on the obtained video training dataset, use the human body parsing model to perform processing operations to obtain the silhouette training dataset of the video training dataset;

[0014] The processing method is as follows: using the video training dataset as input, a human body analysis model is used to obtain visible light images. and infrared images Silhouette image;

[0015] The method for obtaining the silhouette images from the video training dataset is as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] and infrared images The training dataset consists of silhouette images that have had their backgrounds removed and whose body shapes are highlighted. ,in, Indicates by The obtained visible light image silhouette sequence, Indicates by The obtained infrared silhouette sequence;

[0016] The specific steps are as follows: [The video training dataset is then...] Each frame of the image is used as input, and initial features are first extracted using a ResNet network. Then, the feature refinement is performed in three layers, progressing from coarse-grained to fine-grained. Each layer uses a graph convolutional network with the same shared structure as the backbone, along with classification networks of different output dimensions. The steps for progressively refining the features from coarse-grained to fine-grained are as follows:

[0017] First, the initial features The input is passed through the first layer, then through a graph convolutional network and a linear layer with a 2-dimensional output, to obtain the class mask. In this invention, the first layer obtains the category mask as foreground and background. For each category (e.g., foreground), max pooling and average pooling are performed within the corresponding mask region and summed. The results of all categories are concatenated along the channel dimension to obtain the category mask of the first layer. ;

[0018] The second and third layers are similar to the first layer, except that the output dimensions of the classification layers are 5 for the second layer and 18 for the third layer; the input of the second layer is... The output is the category mask of the second layer. The input of the third layer is The output is the category mask of the third layer. In this invention, the second layer obtains the category mask as head, torso, arms, legs, and background; the third layer obtains the category mask as face, hat, hair, sunglasses, top, dress, pants, scarf, skirt, belt, bag, left arm, right arm, left leg, right leg, left shoe, right shoe, and background; the final silhouette image is... , , and This was obtained by using convolutional layers to reduce the dimensionality after connecting along the channel dimension.

[0019] S1.3, Training the dataset with videos and silhouette image training dataset Obtain the training dataset.

[0020] S2. Using the training dataset as input, extract appearance features and body shape features using an appearance encoder and a body shape encoder, respectively, as shown in the following expressions:

[0021]

[0022]

[0023] In the formula, Indicates the appearance encoder; Indicates the size encoder; Indicates appearance characteristics; Indicates body shape characteristics; This represents the video training dataset; This represents the silhouette image training dataset;

[0024] Body encoder The ability to learn pedestrian body shape features is determined by the smoothing loss function. The guidance is provided, and the specific expression is:

[0025]

[0026] In the formula, This represents the total number of frames in a video sequence, where t represents the index of the total number of frames. Represents the horizontal Sobel operator; Represents the vertical Sobel operator;

[0027] S3. Based on the extracted appearance and body shape features, similarity orthogonal decomposition is performed using the information collaboration layer to obtain the enhanced feature map;

[0028] The enhanced feature map includes: enhanced appearance features and enhanced body shape features;

[0029] The expression for performing similarity orthogonal decomposition using the information collaboration layer is as follows:

[0030]

[0031]

[0032]

[0033]

[0034] In the formula, Indicated by appearance features Enhancement and body shape characteristics The feature map is obtained after relevant body shape information, where the superscript + indicates enhanced positive correlation information; the superscript T indicates transpose; Indicated by appearance features Enhancement and body shape characteristics The result of augmenting irrelevant information, including superscript Information indicating an enhanced negative correlation; Indicated by body shape characteristics Enhancement and appearance features The feature map obtained after obtaining relevant appearance information; Indicated by body shape characteristics Enhanced appearance features The result after augmenting irrelevant information; ReLU represents the ReLU activation function; This represents matrix multiplication, and the purpose is to find... and The correlation, specifically, with For example, using The similarity matrix was obtained, and then the positive values ​​were decomposed using the ReLU function and compared with... Multiply to get ,in express Enhancement and The feature map obtained after obtaining relevant body shape information, with the superscript + indicating that the enhancement is related to the body shape information. Information of positive correlation; similarly, using The similarity matrix is ​​obtained, and then the negative values ​​are decomposed using the ReLU function. Multiply to get ,in express Enhance the unique modal characteristics and The result of augmenting irrelevant information, superscript The enhancement is related to Information about negative correlation;

[0035] The enhanced feature map is obtained by co-enhancing the intermediate features obtained from the decomposition, as shown in the following expression:

[0036]

[0037]

[0038] In the formula, It is an enhanced appearance feature. It refers to the enhanced body shape characteristics.

[0039] S4. Use the enhanced feature map for final retrieval and identity matching, and use cross-entropy loss and triplet loss for constraints and training to obtain the re-identification result.

[0040] Beneficial effects of the present invention

[0041] (1) By modeling the body features such as arm swing and body center movement in pedestrian walking videos, this invention effectively utilizes modally invariant body shape features to solve the visible light-infrared modal difference problem and solves the problem that existing methods are easily affected by changes in viewing angle and lighting when modeling the appearance texture features of pedestrians.

[0042] (2) By using a three-layer network to gradually refine the features from coarse to fine, the present invention obtains an analytical map that filters background interference and highlights the human body shape of pedestrians. The final analytical map features learn both the overall body shape information representing the height and weight of pedestrians and the fine-grained information representing the length of the pedestrian's stride, the swing of the arms and the movement of the body's center of gravity.

[0043] (3) By using similarity orthogonal decomposition in the information collaboration layer to enable appearance features and body shape features to work together in a mutually reinforcing manner, the method of the present invention learns both discriminative appearance texture features and modality-invariant body shape features, and enables the two to work together to achieve higher retrieval accuracy. Attached Figure Description

[0044] Figure 1 This is a flowchart of the steps of the present invention;

[0045] Figure 2 These are the reconstructed visible light sequence and infrared mode sequence in this invention, wherein part a is the reconstructed visible light sequence and part b is the reconstructed infrared mode sequence;

[0046] Figure 3 This is a sequence of silhouette images that eliminates the background and highlights the human figure in this invention, wherein part a is a reconstructed visible light sequence silhouette image and part b is a reconstructed infrared modal sequence silhouette image;

[0047] Figure 4 This is a visualization of the rank1 search results of this invention; where a and b are the search results from visible light to infrared light (V2I), a is the input (query), and b is the search target (gallery); c and d are the search results from infrared light to visible light (I2V), c is the input (query), and d is the search target (gallery). Detailed Implementation

[0048] The present invention will be further described in detail below with reference to specific embodiments.

[0049] like Figure 1 As shown, a video visible light-infrared pedestrian re-identification method based on human body analysis includes the following steps:

[0050] S1. Construct a training dataset using pedestrian video sequences in visible light and infrared modes;

[0051] The training dataset includes: a video training dataset and a silhouette image training dataset;

[0052] The construction steps are as follows:

[0053] S1.1 Input pedestrian video sequences in visible light and infrared modes, perform random sampling on the input video sequences to obtain visible light images and infrared images to form a video training dataset;

[0054] The random sampling operation is performed by randomly sampling a preset number of frames from the input pedestrian video sequence in visible light and infrared modes; in this invention, the preset number of frames is set to 6.

[0055] The video training dataset is constructed by randomly sampling a preset number of visible light images. and infrared images The reconstructed visible and infrared mode sequences are used to obtain the video training dataset. ,like Figure 2 As shown in parts a and b;

[0056] S1.2 Based on the obtained video training dataset, use the human body parsing model to perform processing operations to obtain the silhouette training dataset of the video training dataset;

[0057] The processing method is as follows: using the video training dataset as input, a human body analysis model is used to obtain visible light images. and infrared images Silhouette image;

[0058] The method for obtaining the silhouette images from the video training dataset is as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] and infrared images The training dataset consists of silhouette images that have had their backgrounds removed and whose body shapes are highlighted. ,in, Indicates by The obtained visible light image silhouette sequence, Indicates by The obtained infrared image silhouette sequence; the specific steps are: to use the video training dataset Each frame of the image is used as input, and initial features are first extracted using a ResNet network. Then, the feature refinement is performed in three layers, progressing from coarse-grained to fine-grained. Each layer uses a graph convolutional network with the same shared structure as the backbone, along with classification networks of different output dimensions. The steps for progressively refining the features from coarse-grained to fine-grained are as follows:

[0059] First, the initial features The input is passed through the first layer, then through a graph convolutional network and a linear layer with a 2-dimensional output, to obtain the class mask. In this invention, the first layer obtains the category mask as foreground and background. For each category (e.g., foreground), max pooling and average pooling are performed within the corresponding mask region and summed. The results of all categories are concatenated along the channel dimension to obtain the category mask of the first layer. ;

[0060] The second and third layers are similar to the first layer, except that the output dimensions of the classification layers are 5 for the second layer and 18 for the third layer; the input of the second layer is... The output is the category mask of the second layer. The input of the third layer is The output is the category mask of the third layer. In this invention, the second layer obtains the category mask as head, torso, arms, legs, and background; the third layer obtains the category mask as face, hat, hair, sunglasses, top, dress, pants, scarf, skirt, belt, bag, left arm, right arm, left leg, right leg, left shoe, right shoe, and background; the final silhouette image is... , , , This was obtained by using convolutional layers to reduce the dimensionality after connecting along the channel dimension.

[0061] like Figure 3 As shown in sections a and b, this three-layer network structure, progressing from coarse-grained to fine-grained, uses a base layer (foreground / background) to locate human body regions, an intermediate layer (head / torso / limbs) to decompose main body parts, and a fine layer (such as left and right limbs, clothing) to refine local components. The input features of each layer inherit from the refined output of the previous layer, forming a progressive inference flow from the whole to the part. The features of each category at each level correspond to the features of each node in the graph neural network, thus enabling bidirectional optimization of feature representation—the lower layer provides structural constraints for the higher layer, and the higher layer injects semantic guidance into the lower layer. This closed-loop design not only strengthens the spatial correlation between parts (such as the connection dependency between the arm and the torso) but also effectively alleviates analytical ambiguities in complex scenarios such as occlusion and deformation through cross-level feature fusion.

[0062] S1.3, Training the dataset with videos and silhouette image training dataset Obtain the training dataset.

[0063] S2. Using the training dataset as input, extract appearance features and body shape features using an appearance encoder and a body shape encoder, respectively, as shown in the following expressions:

[0064]

[0065]

[0066] In the formula, Indicates the appearance encoder; Indicates the size encoder; Indicates appearance characteristics; Indicates body shape characteristics; This represents the video training dataset; This represents the silhouette image training dataset;

[0067] Body encoder The ability to learn pedestrian body shape features is determined by the smoothing loss function. The guidance is provided, and the specific expression is:

[0068]

[0069] In the formula, This represents the total number of frames in a video sequence, where t represents the index of the total number of frames. Represents the horizontal Sobel operator; Represents the vertical Sobel operator;

[0070] S3. Based on the extracted appearance and body shape features, similarity orthogonal decomposition is performed using the information collaboration layer to obtain the enhanced feature map;

[0071] The enhanced feature map includes: enhanced appearance features and enhanced body shape features;

[0072] The expression for performing similarity orthogonal decomposition using the information collaboration layer is as follows:

[0073]

[0074]

[0075]

[0076]

[0077] In the formula, Indicated by appearance features Enhancement and body shape characteristics The feature map is obtained after relevant body shape information, where the superscript + indicates enhanced positive correlation information; the superscript T indicates transpose; Indicated by appearance features Enhancement and body shape characteristics The result of augmenting irrelevant information, including superscript Information indicating an enhanced negative correlation; Indicated by body shape characteristics Enhancement and appearance features The feature map obtained after obtaining relevant appearance information; Indicated by body shape characteristics Enhanced appearance features The result after augmenting irrelevant information; ReLU represents the ReLU activation function; This represents matrix multiplication, and the purpose is to find... and The correlation, specifically, with For example, using The similarity matrix was obtained, and then the positive values ​​were decomposed using the ReLU function and compared with... Multiply to get ,in express Enhancement and The feature map obtained after obtaining relevant body shape information, with the superscript + indicating that the enhancement is related to the body shape information. Information of positive correlation; similarly, using The similarity matrix is ​​obtained, and then the negative values ​​are decomposed using the ReLU function. Multiply to get ,in express Enhance the unique modal characteristics and The result of augmenting irrelevant information, superscript The enhancement is related to Information about negative correlation;

[0078] The enhanced feature map is obtained by co-enhancing the intermediate features obtained from the decomposition, as shown in the following expression:

[0079]

[0080]

[0081] In the formula, It is an enhanced appearance feature. These are enhanced physical characteristics;

[0082] Obtain the enhanced feature map and These features combine the advantages of both appearance and body shape modalities, while reducing the interference of modality-specific information; the enhanced In retention While utilizing the appearance and texture information, and The similarity matrix enhances the body shape information related to pedestrian identity and eliminates modality-specific interference. Similarly, the enhanced... In retention While using body type information, and The similarity enhances the appearance texture information related to pedestrian identity and eliminates modality-specific interference.

[0083] S4. Use the enhanced feature map for final retrieval and identity matching, and use cross-entropy loss and triplet loss for constraints and training to obtain the re-identification result;

[0084] Re-identification results are as follows Figure 4 Sections a, b, c, and d show the visualization of the rank 1 search results. Sections a and b represent the search results from visible light to infrared light (V2I), where a is the input (query) and b is the search target (gallery). Sections c and d represent the search results from infrared light to visible light (I2V), where c is the input (query) and d is the search target (gallery). Figure 4 It can be seen that, under the interference of modal and viewpoint changes, the model can still accurately retrieve the target pedestrian because it decouples and integrates appearance and body features, thus possessing both discriminative texture representation ability and robust structure recognition ability. Instead of treating the overexposure of the infrared modality as white clothing texture.

[0085] To verify the present invention, the test results of the human body analysis-based method proposed in this invention and other visible light-infrared pedestrian re-identification and retrieval methods on the HITSZ-VCM dataset are shown in Table 1.

[0086]

[0087] As shown in Table 1, the method of this invention exhibits the best performance. This invention uses the standard CMC curve and mAP accuracy as evaluation metrics for model performance. Experimental results for other methods are derived from published papers. All experiments for the method of this invention were conducted on a single NVIDIA Quadro RTX 8000 GPU using the PyTorch framework. This invention employs a ResNet50 pre-trained on the ImageNet dataset as the backbone network, with the input image adjusted to 256×128 pixels. During the training phase, this invention uses a learning rate warm-up strategy, with an initial learning rate of 0.1. After 35 and 80 training epochs, the learning rate is reduced to 0.01 and 0.001, respectively. This invention sets the maximum number of training epochs to 200. In this embodiment, to enhance the quality of the input image, this invention employs techniques such as random cropping, random horizontal flipping, random channel erasure, and channel adaptive grayscale.

[0088] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A video visible light-infrared pedestrian re-identification method based on human body analysis, characterized in that, Includes the following steps: S1. Construct a training dataset using pedestrian video sequences in visible light and infrared modes; The training dataset includes: a video training dataset and a silhouette image sequence training dataset; The steps to construct the training dataset are as follows: S1.1 Input pedestrian video sequences in visible light and infrared modes, perform random sampling on the input video sequences to obtain visible light images and infrared images to form a video training dataset; The random sampling operation is performed by randomly sampling a preset number of frames from the input pedestrian video sequence in visible light and infrared modes; The video training dataset is constructed by randomly sampling a preset number of visible light and infrared images to create a reconstructed visible light sequence. and infrared mode sequence Obtain the video training dataset ; S1.2 Based on the obtained video training dataset, use the human body parsing model to perform processing operations to obtain the silhouette training dataset of the video training dataset; The processing method is as follows: using the video training dataset as input, a human body analysis model is used to obtain visible light sequences. and infrared sequence Silhouette image; S1.3 Obtain the training dataset using the video training dataset and the silhouette image training dataset; The method for obtaining the silhouette training dataset of the video training dataset by performing processing operations using a human body parsing model based on the obtained video training dataset is as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Each frame of the image is used as input, and initial features are first extracted using a ResNet network. Then, the feature refinement is performed in three layers, progressing from coarse-grained to fine-grained. Each layer uses a graph convolutional network with the same shared structure as the backbone, along with classification networks of different output dimensions. The steps for progressively refining the features from coarse-grained to fine-grained are as follows: First, the initial features The input to the first layer passes through a graph convolutional network and a linear layer with an output dimension of 2 to obtain a class mask, including foreground and background. For each class mask, max pooling and average pooling are performed within the mask region of the class mask, and the results are summed. All the results are concatenated along the channel dimension to obtain the class mask of the first layer. ; The input to the second layer is the category mask of the first layer. The output of the second layer is The second classification layer outputs 5 dimensions, corresponding to: head, torso, arms, legs, and background; The input to the third layer is the category mask of the second layer. The output of the third layer is The third classification layer outputs 18 dimensions, corresponding to: face, hat, hair, sunglasses, top, dress, pants, scarf, skirt, belt, bag, left arm, right arm, left leg, right leg, left shoe, right shoe, and background; The final silhouette image is , , and This was obtained by using convolutional layers to reduce the dimensionality after connecting along the channel dimension. S2. Using the training dataset as input, extract appearance features and body shape features using an appearance encoder and a body shape encoder, respectively, as shown in the following expressions: ； ； In the formula, Indicates the appearance encoder; Indicates the size encoder; Indicates appearance characteristics; Indicates body shape characteristics; This represents the video training dataset; This represents the silhouette image training dataset; Body encoder The ability to learn pedestrian body shape features is determined by the smoothing loss function. Guided expression: ； In the formula, This represents the total number of frames in a video sequence, where t represents the index of the total number of frames. Represents the horizontal Sobel operator; Represents the vertical Sobel operator; S3. Based on the extracted appearance and body shape features, similarity orthogonal decomposition is performed using the information collaboration layer to obtain the enhanced feature map; The enhanced feature map includes: enhanced appearance features and enhanced body shape features; S4. Use the enhanced feature map for final retrieval and identity matching, and use cross-entropy loss and triplet loss for constraints and training to obtain the re-identification result.

2. The video visible light-infrared pedestrian re-identification method based on human body analysis according to claim 1, characterized in that, Based on the extracted appearance and body shape features, a similarity orthogonal decomposition is performed using an information collaboration layer to obtain the enhanced feature map. The expression for the similarity orthogonal decomposition performed using the information collaboration layer is as follows: ； ； ； ； In the formula, Indicated by appearance features Enhancement and body shape characteristics The feature map is obtained after relevant body shape information, where the superscript + indicates enhanced positive correlation information; the superscript T indicates transpose; Indicated by appearance features Enhancement and body shape characteristics The result of augmenting irrelevant information, including superscript Information indicating an enhanced negative correlation; Indicated by body shape characteristics Enhancement and appearance features The feature map obtained after obtaining relevant appearance information; Indicated by body shape characteristics Enhanced appearance features The result after augmenting irrelevant information; ReLU represents the ReLU activation function; Indicates matrix multiplication; The enhanced feature map is obtained by co-enhancing the intermediate features obtained from the decomposition, as shown in the following expression: ； ； In the formula, It is an enhanced appearance feature. It refers to the enhanced body shape characteristics.

Images