A method, system, device and medium for fully supervised video pedestrian re-identification

Abstract

This invention discloses a fully supervised video person re-identification method, system, device, and medium. The fully supervised video person re-identification method includes the following steps: acquiring a video segment containing a target person and a video segment to be re-identified; based on the acquired video segment containing the target person and the video segment to be re-identified, performing person re-identification processing using a pre-trained person re-identification model, and outputting the person re-identification result. Specifically, the fully supervised video person re-identification method disclosed in this invention is a video person re-identification method based on temporal correlation decomposition. It removes features of non-target persons by using the different relative states of the target person and the occluded person, thus eliminating the influence of the occluded person on the model's feature learning in the case of occlusion. Furthermore, it restores the semantic consistency of video segments by realigning misaligned video frames through a correlation filtering algorithm, thereby improving retrieval accuracy.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically the field of pedestrian re-identification, and particularly relates to a fully supervised video pedestrian re-identification method, system, device, and medium. Background Technology

[0002] The goal of the video pedestrian re-identification task is to retrieve video clips containing the target person from a large number of video segments, given a video of the target person. This task has many practical applications, such as intelligent video surveillance systems, intelligent security, and cross-camera target tracking.

[0003] At present, existing video pedestrian re-identification methods still have the following technical shortcomings, including:

[0004] (1) In densely populated scenes, people will be occluded. When the target person is severely occluded, the model has difficulty learning the target person’s features accurately, resulting in retrieval errors. Especially when the occluded person and the target person have similar appearance features, the model will focus on the wrong part, resulting in retrieval failure.

[0005] (2) Due to the inadequacy of target detection results, video clips of the same pedestrian may have inaccurate positioning, resulting in different semantics at the same spatial location in consecutive frames; when fusing video clip features, the misaligned parts will destroy the final video features, causing a decrease in retrieval accuracy. Summary of the Invention

[0006] The purpose of this invention is to provide a fully supervised video person re-identification method, system, device, and medium to solve one or more of the aforementioned technical problems. Specifically, the fully supervised video person re-identification method disclosed in this invention is a video person re-identification method based on temporal correlation decomposition. It removes features of non-target persons by utilizing the different relative states of the target person and the occluded person, thus eliminating the influence of the occluded person on model feature learning in the case of occlusion. Furthermore, it restores the semantic consistency of video segments by realigning misaligned video frames through a correlation filtering algorithm, thereby improving retrieval accuracy.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] The first aspect of this invention provides a fully supervised video pedestrian re-identification method, comprising the following steps:

[0009] Acquire video clips containing the target person and video clips for pedestrian re-identification;

[0010] Based on the acquired video clips containing the target person and the video clips to be re-identified, a pre-trained person re-identification model is used to perform person re-identification processing, and the person re-identification result is output; wherein...

[0011] The pedestrian re-identification result includes at least whether the video segment to be re-identified contains the target person;

[0012] The pedestrian re-identification model includes:

[0013] The encoder module is used to extract features from raw video frames and output frame-level feature maps; wherein, the encoder module is based on the classic Vision Transformer architecture;

[0014] The feature alignment module is used to perform deviation calculation on the input raw video frames using the kernel correlation filtering algorithm, and output the cross-frame position deviation of the target person.

[0015] The decoder module is used to take the frame-level feature map and the cross-frame positional deviation of the target person as input, perform feature alignment processing, and obtain an aligned frame-level feature map; utilize the different relative states of the target person and the occluder to perform occlusion local feature removal processing on the aligned frame-level feature map, and obtain a frame-level feature map with occlusion local features removed; based on a multi-head self-attention mechanism, perform feature interaction and fusion processing on the frame-level feature map with occlusion local features removed, and output video-level features; wherein, the decoder module is a decoder based on a multi-head self-attention mechanism.

[0016] A further improvement of the method of the present invention is that the training steps of the pre-trained person re-identification model include:

[0017] Obtain a training sample set; wherein each training sample in the training sample set includes a sampled video clip containing pedestrians, and the ID number of the pedestrians contained in the video clip;

[0018] During training, for selected training samples, the sampled video clips containing pedestrians are input into the pedestrian re-identification model to predict the ID numbers of pedestrians in the video clips and use them as prediction results. The prediction results are compared with the ID numbers of pedestrians contained in the video clips in the training samples. The model is trained under supervision using cross-entropy, triplet, and mutual information loss functions and the parameters are updated. After reaching the preset convergence condition, the pre-trained pedestrian re-identification model is obtained.

[0019] A further improvement of the method of the present invention is that, in the feature alignment module, the step of using a kernel correlation filtering algorithm to perform deviation calculation processing on the input original video frames and outputting the cross-frame position deviation of the target person includes:

[0020] The cross-frame positional deviation of the target person in each video segment is calculated using a correlation filtering algorithm; where, for each frame X in the video segment... t Taking the average along the channel dimension transforms into

[0021] Initialize the correlation filter, including: initializing the correlation filter using the first frame X'1, with the expression:

[0022]

[0023]

[0024] In the formula, DFT(·) represents Discrete Fourier Transform, IDFT(·) represents Inverse Discrete Fourier Transform, y is the Gaussian regression objective, λ is the regularization coefficient, and DFT(X'1) * It is the complex conjugate of DFT(X'1), and ⊙ represents the matrix element-wise multiplication operation; α 1 It is the correlation filter calculated on the first frame;

[0025] Calculate the cross-frame positional deviation in the next frame, and then update the filter parameters; specifically, the cross-frame positional deviation in the second frame is calculated using a correlation filter, and the filter parameters are updated using an exponential moving average, as expressed by the following expression:

[0026]

[0027] M2 = IDFT(DFT(k) 1,2 )⊙α t );

[0028] In the formula, This is the response map of the second frame on the correlation filter. By calculating the distance from the maximum response point of M2 to the center, the cross-frame position deviation of the target person in the second frame can be obtained.

[0029] The pixels in the second frame are aligned using a rolling method based on the target person's cross-frame positional deviation. The filter parameters are then updated using the aligned second frame, as expressed in the following expression:

[0030]

[0031]

[0032] α 2 =βα 1 +(1-β)α 2 ;

[0033] In the formula, β is the step size of the exponential moving average, and the filter parameters are updated by the exponential moving average.

[0034] Repeat the above steps until the cross-frame positional deviation of the target person in all frames is calculated; where the updated filter α from the second frame is used. 2 To calculate the target person's cross-frame position deviation in the 3rd frame, and to update the filter α using the 3rd frame after pixel roll alignment. 2 The parameters are calculated repeatedly frame by frame to calculate the cross-frame position deviation of the target person and update the filter parameters until the cross-frame position deviation of the target person in all frames is calculated.

[0035] A further improvement of the method of the present invention is that, in the decoder module, the frame-level feature map and the cross-frame position deviation of the target person are input, and feature alignment processing is performed to obtain an aligned frame-level feature map; utilizing the different relative states of the target person and the occluding person, the occluding person's local features are removed from the aligned frame-level feature map to obtain a frame-level feature map with the occluding person's local features removed; based on a multi-head self-attention mechanism, feature interaction and fusion processing are performed on the frame-level feature map with the occluding person's local features removed to output video-level features. The steps include:

[0036] Based on the obtained cross-frame positional deviation of the target person in adjacent frames, let the response map M t The horizontal and vertical deviations of the maximum response point from the center are: and According to the deviation rolling characteristic diagram Z t Align the feature maps to obtain the aligned feature map Z' t ;

[0037] The aligned feature map Z' t By taking the mean along the time dimension to focus on the relatively static parts of the sequence, a feature map is obtained. The expression is,

[0038] Calculate frame-level feature map Z' t and The cosine similarity is calculated using the following expression:

[0039]

[0040] In the formula, The value is between 0 and 1, representing Z'. t Each pixel block and the averaged feature map The cosine similarity; <·, ·> denote the vector dot product operation;

[0041] The feature of the occluded person is eliminated by adaptively generating a threshold based on cosine similarity, by respectively applying c t The parameters γ and δ are generated by taking the mean along the time dimension and the time-space dimension;

[0042] To eliminate local features with cosine similarity less than the adaptively generated threshold, the expression is:

[0043]

[0044] By mask m t With feature map Z' t Element-level multiplication yields a feature map that eliminates occlusion of local human features.

[0045] Feature maps were analyzed sequentially. This involves temporal and spatial interaction; specifically, local features with the same spatial location along the time dimension interact with each other temporally, expressed as follows:

[0046]

[0047]

[0048] In the formula, W q W k and W v These are the learnable weights of the query, key, and value, respectively. It is a feature map after time interaction;

[0049] For feature maps Spatial interaction is achieved by spatially interacting all local features within the same frame, expressed as follows:

[0050]

[0051]

[0052] In the formula, The feature map is obtained after spatiotemporal interaction with the decoder;

[0053] Global average pooling is used to process the feature maps after spatiotemporal interaction. The features are then fused to obtain the final video-level features.

[0054] A further improvement of the method of the present invention is that, in the training step of the pre-trained person re-identification model, the expression of the loss function is as follows:

[0055] L = L v +λL f In the formula, L is the overall loss function of the model; λ is a hyperparameter that balances the weights of the frame-level feature loss function and the video-level loss function.

[0056] In the formula, L x L represents the triplet loss function. irepresents the cross-entropy loss function, and v represents the category label in the video-level features;

[0057] In the formula, L m Z represents the mutual information loss function. t[cls] This represents the category label in the frame-level features.

[0058] A second aspect of the present invention provides a fully supervised video pedestrian re-identification system, comprising:

[0059] The data acquisition module is used to acquire video clips containing the target person and video clips for pedestrian re-identification;

[0060] The pedestrian re-identification result acquisition module is used to perform pedestrian re-identification processing using a pre-trained pedestrian re-identification model based on the acquired video segment containing the target person and the video segment to be re-identified, and output the pedestrian re-identification result; wherein, the pedestrian re-identification result includes at least whether the video segment to be re-identified contains the target person.

[0061] The pedestrian re-identification model includes:

[0062] The encoder module is used to extract features from raw video frames and output frame-level feature maps; wherein, the encoder module is based on the classic Vision Transformer architecture;

[0063] The feature alignment module is used to perform deviation calculation on the input raw video frames using the kernel correlation filtering algorithm, and output the cross-frame position deviation of the target person.

[0064] The decoder module is used to take the frame-level feature map and the cross-frame positional deviation of the target person as input, perform feature alignment processing, and obtain an aligned frame-level feature map; utilize the different relative states of the target person and the occluder to perform occlusion local feature removal processing on the aligned frame-level feature map, and obtain a frame-level feature map with occlusion local features removed; based on a multi-head self-attention mechanism, perform feature interaction and fusion processing on the frame-level feature map with occlusion local features removed, and output video-level features; wherein, the decoder module is a decoder based on a multi-head self-attention mechanism.

[0065] A third aspect of the present invention provides an electronic device comprising:

[0066] At least one processor; and,

[0067] A memory communicatively connected to the at least one processor; wherein,

[0068] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the fully supervised video pedestrian re-identification method as described in any of the first aspects of the invention.

[0069] The fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the fully supervised video pedestrian re-identification method according to any one of the first aspects of the present invention.

[0070] Compared with the prior art, the present invention has the following beneficial effects:

[0071] This invention provides a fully supervised video person re-identification method, specifically a novel video person re-identification method based on temporal correlation decomposition. This method effectively distinguishes between target people and occluded people, and accurately re-identifies pedestrians. Further, addressing the technical problems of existing person re-identification methods, such as the difficulty in eliminating the impact of occluded people on feature learning in densely populated scenes and semantic inconsistencies caused by insufficient detection results, this invention proposes for the first time to distinguish between occluded people and target people through correlation. It removes local features of occluded people before spatiotemporal interaction fusion and utilizes a correlation filtering algorithm to re-align video frames, restoring semantic consistency between frames and improving retrieval accuracy.

[0072] The model of this invention adopts a classic encoder-decoder structure. It uses Vision Transformer to globally model video-level features, first extracts frame-level features through ViT-Base encoder, and then uses multi-head self-attention layer for spatiotemporal interaction. Distinguishing between target people and occluded people by relative state and restoring semantic consistency by using relevant filters does not introduce additional learnable parameters and does not increase the training burden of the model. The two modules complement each other, enhance the final video-level representation, and improve the accuracy of pedestrian re-identification. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art are briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0074] Figure 1 This is a flowchart illustrating a fully supervised video pedestrian re-identification method provided in an embodiment of the present invention;

[0075] Figure 2 This is a schematic diagram of the model training process in an embodiment of the present invention;

[0076] Figure 3 This is a schematic block diagram of the model structure in an embodiment of the present invention;

[0077] Figure 4 This is a schematic diagram of the visualization results of removing local features of an occluded person through relative states in an embodiment of the present invention;

[0078] Figure 5 This is a schematic diagram illustrating the visualization results of restoring inter-frame semantic consistency using a correlation filter in an embodiment of the present invention;

[0079] Figure 6 This is a schematic diagram comparing the GradCAM heatmap visualization results with the baseline model in an embodiment of the present invention.

[0080] Figure 7 This is a schematic diagram of a fully supervised video pedestrian re-identification system provided in an embodiment of the present invention. Detailed Implementation

[0081] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0082] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0083] The present invention will now be described in further detail with reference to the accompanying drawings:

[0084] Please see Figure 1 The present invention provides a fully supervised video pedestrian re-identification method, comprising the following steps:

[0085] Step 1: Obtain the video clip containing the target person and the video clip to be re-identified as a pedestrian;

[0086] Step 2: Based on the acquired video clip containing the target person and the video clip to be re-identified, perform pedestrian re-identification processing using a pre-trained pedestrian re-identification model and output the pedestrian re-identification result; wherein, the pedestrian re-identification result includes at least the determination result of whether the video clip to be re-identified contains the target person.

[0087] The pre-trained person re-identification model described in this embodiment of the invention includes the following model architecture:

[0088] The encoder module, based on the classic Vision Transformer architecture, is used to extract features from raw video frames and output frame-level feature maps.

[0089] The feature alignment module uses a kernel correlation filtering algorithm to perform deviation calculation on the input original video frames and output the cross-frame position deviation of the target person.

[0090] The decoder module is used to first input the frame-level feature map output by the encoder module and the cross-frame position deviation of the target person output by the feature alignment module for feature alignment processing; then, using the different relative states of the target person and the occluded person, the occluded person's local features are removed from the aligned frame-level feature map; finally, based on the multi-head self-attention mechanism, the frame-level feature map with the occluded person's local features removed is subjected to feature interaction and fusion processing, and finally outputs video-level features.

[0091] In this embodiment of the invention, the training steps specifically include:

[0092] Obtain a training sample set; wherein each training sample in the training sample set includes a sampled video clip containing pedestrians, and the ID number of the pedestrians contained in the video clip;

[0093] During training, for selected training samples, video clips from the samples are input into the model to predict the ID numbers of pedestrians in the video clips. The predicted results are compared with the pedestrian ID numbers in the sample set. Supervised training is performed using cross-entropy, triplet, and mutual information loss functions. The AdamW optimizer is used for updating. When the model has been trained for 90 iterations, a pre-trained pedestrian re-identification model is obtained.

[0094] In summary, this invention discloses a fully supervised video person re-identification method, comprising a VisionTransformer-based encoder, a feature alignment module based on a correlation filtering algorithm, and a decoder based on a multi-head self-attention mechanism. The encoder models frame-level features. Before spatiotemporal interactive fusion, the feature alignment module effectively restores the semantic consistency between video frames without introducing additional learnable parameters, providing aligned frame-level feature maps for the subsequent decoder module. The decoder effectively utilizes the fine-grained features of the target person in the feature map, first erasing the local features of the occluded person in the aligned frame-level feature map using the relative state between the target person and the occluded person, thereby performing spatiotemporal interactive fusion without the influence of occlusion or misalignment to obtain the final video-level representation.

[0095] Please see Figure 2 and Figure 3 Specifically, as exemplified, in a fully supervised video pedestrian re-identification method provided by an embodiment of the present invention, the training model includes the following steps:

[0096] Step 1: Collect pedestrian video clips and pedestrian video clip IDs, including:

[0097] 1.1) Uniformly sample video segments of different lengths to obtain video segments of the same length but different ID numbers;

[0098] 1.2) For the video segments sampled in step 1.1), save their ID numbers as their tags.

[0099] Step 2: Calculate the cross-frame positional deviation of the target person in each video segment using a correlation filtering algorithm, including:

[0100] 2.1) Initialize the correlation filter on the first frame;

[0101] 2.1) Calculate the cross-frame positional deviation of the target person in the second frame using a correlation filter, and update the filter parameters using an exponential moving average;

[0102] 2.2) Similarly, use the updated filter to calculate the cross-frame position deviation of the target person in 3 frames, and update the filter parameters again;

[0103] 2.3) Repeat the above steps until the cross-frame positional deviation of the target person in all frames is calculated.

[0104] Step 3, learn frame-level features using a self-attention encoder, including:

[0105] 3.1) Construct a self-attention encoder based on the Vision Transformer structure;

[0106] 3.2) Input the sampled video clip and use the encoder to learn frame-level features.

[0107] Step 4: Perform spatiotemporal interaction and feature fusion using a self-attention decoder:

[0108] 4.1) Construct a decoder based on a multi-head self-attention mechanism;

[0109] 4.2) Based on the cross-frame positional deviation of the target person calculated in step 2.3), align the frame-level feature maps and restore semantic consistency;

[0110] 4.3) Remove the features of the occluded person from the frame-level feature map by using the different relative states of the target person and the occluded person, thereby eliminating the influence of the occluded person on the feature map;

[0111] 4.4) Perform temporal interaction on frame-level features first, then perform spatial interaction;

[0112] 4.5) Use global average pooling to fuse frame-level features to obtain the final video-level features.

[0113] Step 5, calculate the neural network loss function:

[0114] 5.1) Input the ID number of the pedestrian video segment, and calculate the cross-entropy loss function based on the video-level features obtained in step 4;

[0115] 5.2) Calculate the triplet loss function and other loss functions.

[0116] Step 6: Optimize network parameters to improve the accuracy of video pedestrian re-identification.

[0117] 6.1) Iteratively optimize the neural network parameters based on the loss function obtained in step 5;

[0118] 6.2) After reaching the predetermined number of iterations, the video pedestrian re-identification is achieved through the encoder-decoder obtained in steps 3 and 4.

[0119] In summary, to address the semantic inconsistencies caused by insufficient detection results and the problem of people occlusion in densely populated scenes, a correlation filtering algorithm is introduced to restore the semantic consistency between video frames. This algorithm effectively removes the features of occluded people by analyzing the relative state between the target person and the occluded person, driving the model to accurately achieve video pedestrian re-identification even under conditions of occlusion and insufficient detection results.

[0120] A further specific and exemplary embodiment of the present invention provides a video pedestrian re-identification method based on temporal correlation decomposition, comprising the following steps:

[0121] Step 1: Collect pedestrian video clips and pedestrian video clip IDs, including:

[0122] 1.1) Video segments of different lengths are uniformly sampled into video segments of length 8 frames; wherein, each video segment is uniformly divided into 8 sub-segments of equal length, and then one frame is randomly sampled from each sub-segment, and finally each video segment is sampled into a video segment of length 8 frames, represented as follows. in

[0123] 1.2) Collect video clip IDs as tags.

[0124] Step 2: Calculate the cross-frame positional deviation of the target person in each video segment using a correlation filtering algorithm; whereby, for each frame X in the video segment... t Take the average along the channel dimension and transform it into

[0125] 2.1) Initialize the correlation filter, including: initializing the correlation filter using the first frame X'1:

[0126]

[0127]

[0128] Where DFT(·) represents Discrete Fourier Transform, IDFT(·) represents Inverse Discrete Fourier Transform, y is the Gaussian regression objective, λ is the regularization coefficient, and DFT(X'1) * It is the complex conjugate of DFT(X'1), and ⊙ represents the matrix element-wise multiplication operation; α 1 The correlation filter is calculated on the first frame.

[0129] 2.2) Calculate the cross-frame positional deviation in the next frame, and then update the filter parameters; wherein, the cross-frame positional deviation in the second frame is calculated using a correlation filter, and the filter parameters are updated using an exponential moving average:

[0130]

[0131] M2 = IDFT(DFT(k) 1,2 )⊙α t );

[0132] in, This is the response map of the second frame on the correlation filter. By calculating the distance from the maximum response point of M2 to the center, the cross-frame position deviation of the target person in the second frame can be obtained. Based on the cross-frame position deviation of the target person, the pixels of the second frame are rolled and aligned, and the filter parameters are updated using the aligned second frame.

[0133]

[0134]

[0135] α 2 =βα 1 +(l-β)α 2 ;

[0136] Where β is the step size of the exponential moving average, and the filter parameters are updated by the exponential moving average.

[0137] 2.3) Repeat the above steps until the cross-frame positional deviation of the target person in all frames is calculated; where the updated filter α in the second frame is used. 2 To calculate the target person's cross-frame position deviation in the 3rd frame, and to update the filter α using the 3rd frame after pixel roll alignment. 2 The parameters are calculated. This process continues until the cross-frame positional deviation of the target person across all frames is calculated.

[0138] Step 3, learn frame-level features using a self-attention encoder, including:

[0139] 3.1) Construct a self-attention encoder based on the Vision Transformer structure; where,

[0140] The paper "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale" has achieved good results on various computer vision tasks. Here, we use the VisionTransformer (ViT) proposed in the paper as the backbone network for the video pedestrian re-identification task. It adopts a multi-head self-attention mechanism and can learn fine-grained representations in the global receptive field.

[0141] 3.2) Input the sampled video segment and use the encoder to learn frame-level features; where,

[0142] The ViT encoder is used to extract features from the frames of the video, and the video frame X is... t The data is divided into 16×16 pixel blocks, and feature extraction is performed on a pixel-by-pixel basis. In addition, a learnable vector X is generated. t[cls] As the class label for frame t. Frame X is then processed using a 12-layer, 12-head self-attention layer. t Mapping to feature map Class label X t[cls] Mapped to global feature Z t[cls] Where N represents the number of pixel blocks in a frame: D represents the channel dimension of each pixel block after feature extraction: D = 16 × 16 × C.

[0143] Step 4, using a self-attention decoder for spatiotemporal interaction and feature fusion, including:

[0144] 4.1) Construct a decoder based on a multi-head self-attention mechanism; referring to the ViT encoder structure in step 3.1), use a 2-layer, 12-head self-attention layer as the decoder to process the feature map Z. t To perform spatiotemporal interaction and fusion;

[0145] 4.2) Based on the cross-frame positional deviation of the target person calculated in step 2.3), align the frame-level feature maps and restore semantic consistency;

[0146] Due to insufficient detection results, adjacent frames X t With X t+1 Semantic information may be inconsistent at the same spatial location. In step 2.3), the cross-frame positional deviation of the target person in adjacent frames was calculated, and the response map M... t The horizontal and vertical deviations of the maximum response point from the center are: and According to the deviation rolling characteristic diagram Z t Align the feature maps and let the aligned feature map be Z'. t .

[0147] 4.3) By utilizing the different relative states of the target person and the occluded person, the features of the occluded person in the frame-level feature map are removed, thus eliminating the influence of the occluded person on the feature map; among which,

[0148] Aligned feature maps By taking the mean along the time dimension to focus on the relatively static parts of the sequence, a feature map is obtained.

[0149]

[0150] Calculate frame-level feature map Z' t and Cosine similarity:

[0151]

[0152] in The value is between 0 and 1, representing Z'. t Each pixel block and the averaged feature map The cosine similarity.

[0153] In video person re-identification datasets, the target person is always located at the center of the video frame; even when the target person is moving, they remain relatively stationary in the video. Conversely, when occlusion occurs, the occluding person passes the target person, appearing on one side and disappearing on the other, thus exhibiting relative motion in the video. Feature Map Focusing on relatively static parts of a video clip, the frame-level feature map shows the relationship between these relatively static parts and... The cosine similarity will be high, while the cosine similarity of the relatively moving parts in the frame-level feature map will be low. This distinguishes the target person from the occluded person and eliminates the influence of the occluded person on subsequent spatiotemporal fusion.

[0154] Features that obscure people are eliminated by adaptively generating a threshold based on cosine similarity, by separately... The parameters γ and δ are generated by taking the mean along the time dimension and the time-space dimension:

[0155]

[0156] Then, local features with cosine similarity less than the adaptively generated threshold are eliminated:

[0157]

[0158] Generate the mask m according to the above formula. t By combining the mask with the feature map Z' t Element-level multiplication yields a feature map that eliminates occlusion of local human features.

[0159] 4.4) Perform temporal interaction on frame-level features first, then perform spatial interaction;

[0160] The decoder sequentially processes the feature maps For temporal and spatial interaction, the decoder consists of two layers with 12 self-attention heads. This invention employs a standard self-attention mechanism for feature interaction, with the first layer... To achieve temporal interaction, local features with the same spatial location along the time dimension interact with each other:

[0161]

[0162]

[0163] Among them W q W k and W v These are the learnable weights of the query, key, and value, respectively. This is the feature map after time interaction. Then, the second multi-head self-attention layer processes the feature map. Spatial interaction is achieved by allowing all local features within the same frame to interact spatially.

[0164]

[0165]

[0166] Based on the above formula, the feature map after spatiotemporal interaction by the decoder is obtained.

[0167] 4.5) Use global average pooling to process the feature maps after spatiotemporal interaction. The features are then fused to obtain the final video-level features.

[0168] Extracting feature maps Global features in and the intermediate layer global feature Z obtained in step 3.2). [cls] Each of these is then subjected to temporal pooling and concatenated to form the final video representation v:

[0169] v = Concat(Pool(S) [cls] Pool(Z) [cls] )).

[0170] Step 5, calculate the neural network loss function:

[0171] 5.1) Calculate the cross-entropy loss function and the triplet loss function based on the video-level features obtained in step 4.5), using L... x L represents the triplet loss function. i Represents the cross-entropy loss function;

[0172]

[0173] 5.2) Calculate the triplet loss function and cross-entropy loss function based on the frame-level global features obtained in step 3.2), using L... m Z represents the mutual information loss function. t[cls] This represents the category label for frame t.

[0174]

[0175] The overall loss function of the model is L v and L f Composition: L = L v +λL f ;

[0176] Where λ is a hyperparameter that balances the weights of the frame-level feature loss function and the video-level loss function.

[0177] Step 6: Optimize network parameters to improve the accuracy of video pedestrian re-identification, including:

[0178] 6.1) Iteratively optimize the neural network parameters based on the loss function obtained in step 5; wherein, the AdamW optimizer is used for 90 iterations, a cosine learning strategy is used, and the weight decay is 0.0005;

[0179] 6.2) After reaching the predetermined number of iterations, the video pedestrian re-identification is achieved through the encoder-decoder obtained in steps 3 and 4.

[0180] In summary, this invention proposes an encoder-decoder network based on temporal correlation decomposition for video target re-identification tasks. By leveraging the different relative states of the target person and the occluded person, local features of the occluded person are removed from the frame-level feature map before entering the decoder, generating video-level features unaffected by non-target pedestrian occlusion. Furthermore, a correlation filter is used to restore semantic consistency between frames. To ensure fairness in comparisons, this invention uses only cross-entropy and triplet loss functions to guide model training. The proposed method is qualitatively and quantitatively compared with existing methods on the Mars and LS-VID public datasets to verify its effectiveness.

[0181] Table 1. Comparison of experimental results (%) on Mars and LS-VID datasets

[0182]

[0183]

[0184] mAP, Rank-1, and Rank-5 are commonly used metrics for measuring the accuracy of person re-identification. Higher values ​​indicate higher accuracy. As shown in Table 1, on the Mars dataset, this invention achieves the highest re-identification performance, improving the Rank-1 metric by 0.4% compared to the best method. On the LS-VID dataset, mAP is 1.7% higher than the best method. The Rank-1 metric is 0.4% lower than CAViT because CAViT integrates information from three scales, greatly enhancing the representation ability of video-level features. However, its mAP metric is significantly lower than that of this invention. In conclusion, the method proposed in this invention achieves the best results on existing datasets, fully demonstrating the effectiveness and superiority of the proposed method.

[0185] Figure 4 , Figure 5 , Figure 6 For the visualization results analysis of this invention, Figure 4 This paper presents the results of removing local features of the occluded person on the Mars and iLiDS-VID datasets when non-target person occlusion occurs. It can be seen that this invention effectively removes the features of the occluded person, and even under severe occlusion, effectively preserves the fine-grained features of the target person. Figure 5To visualize the restoration of inter-frame semantic consistency using a correlation filtering algorithm, the first line shows the original video sequence. Due to insufficient detection results, misalignment occurs in the later frames, resulting in semantic inconsistency in the same regions across different frames. The second line shows the video sequence after restoring semantic consistency, demonstrating that the misaligned video frames have been realigned, restoring the semantic consistency of the video sequence. Figure 6 The GradCAM heatmap visualization results of the final video-level features obtained by this method are shown in the first row. The first row shows the original input video sequence, and the second row shows the heatmap results of the baseline model. It can be seen that when occlusion occurs, the model learns the features of the occluding person rather than the features of the target person. As shown in the third row, when using the method proposed in this invention, even in cases of severe occlusion, the model can still focus on the fine-grained features of the target person. In summary, the novel temporal correlation decomposition network proposed in this invention successfully removes the features of the occluding person through relative states while preserving the fine-grained features of the target person. It utilizes correlation filters to solve the semantic inconsistency problem caused by insufficient detection results, successfully restoring the semantic consistency between frames. Qualitative analysis and visualization results fully demonstrate the effectiveness and superiority of this invention, achieving accurate video person re-identification.

[0186] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0187] Please see Figure 7 The present invention provides a fully supervised video pedestrian re-identification system, comprising:

[0188] The data acquisition module is used to acquire video clips containing the target person and video clips for pedestrian re-identification;

[0189] The pedestrian re-identification result acquisition module is used to perform pedestrian re-identification processing using a pre-trained pedestrian re-identification model based on the acquired video segment containing the target person and the video segment to be re-identified, and output the pedestrian re-identification result; wherein, the pedestrian re-identification result includes at least whether the video segment to be re-identified contains the target person.

[0190] The pedestrian re-identification model includes:

[0191] The encoder module is used to extract features from raw video frames and output frame-level feature maps; wherein, the encoder module is based on the classic Vision Transformer architecture;

[0192] The feature alignment module is used to perform deviation calculation on the input raw video frames using the kernel correlation filtering algorithm, and output the cross-frame position deviation of the target person.

[0193] The decoder module is used to take the frame-level feature map and the cross-frame positional deviation of the target person as input, perform feature alignment processing, and obtain an aligned frame-level feature map; utilize the different relative states of the target person and the occluder to perform occlusion local feature removal processing on the aligned frame-level feature map, and obtain a frame-level feature map with occlusion local features removed; based on a multi-head self-attention mechanism, perform feature interaction and fusion processing on the frame-level feature map with occlusion local features removed, and output video-level features; wherein, the decoder module is a decoder based on a multi-head self-attention mechanism.

[0194] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or function. The processor described in this embodiment of the present invention can be used for the operation of a fully supervised video pedestrian re-identification method.

[0195] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the fully supervised video pedestrian re-identification method in the above embodiments.

[0196] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0197] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0198] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0199] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A fully supervised video pedestrian re-identification method, characterized in that, Includes the following steps: Acquire video clips containing the target person and video clips for pedestrian re-identification; Based on the acquired video clip containing the target person and the video clip to be re-identified, a pre-trained pedestrian re-identification model is used to perform pedestrian re-identification processing and output pedestrian re-identification results; wherein, the pedestrian re-identification results include at least whether the video clip to be re-identified contains the target person. The pedestrian re-identification model includes: The encoder module is used to extract features from raw video frames and output frame-level feature maps; wherein, the encoder module is based on the Vision Transformer architecture; The feature alignment module is used to perform deviation calculation on the input raw video frames using the kernel correlation filtering algorithm, and output the cross-frame position deviation of the target person. A decoder module is used to input the frame-level feature map and the cross-frame position deviation of the target person, perform feature alignment processing, and obtain an aligned frame-level feature map; utilize the different relative states of the target person and the occluding person to perform occluding person local feature removal processing on the aligned frame-level feature map, and obtain a frame-level feature map with occluding person local features removed; based on a multi-head self-attention mechanism, feature interaction and fusion processing are performed on the frame-level feature map with occluding person local features removed, and video-level features are output; wherein, the decoder module is a decoder based on a multi-head self-attention mechanism; based on the obtained cross-frame position deviation of the target person in adjacent frames, let the response map... The horizontal and vertical deviations of the maximum response point from the center are: and According to the deviation rolling feature map Align the feature maps to obtain the aligned feature maps. ; Aligned feature maps By taking the mean along the time dimension to focus on the relatively static parts of the sequence, a feature map is obtained. The expression is, ; Calculate frame-level feature maps and The cosine similarity is calculated using the following expression: In the formula, The value ranges from 0 to 1, representing Each pixel block and the averaged feature map Cosine similarity; This represents the vector dot product operation; features that obscure people are eliminated by adaptively generating a threshold based on cosine similarity, by respectively... Parameters are generated by taking the mean along the time and time-space dimensions. and To eliminate local features with cosine similarity less than the adaptively generated threshold, the expression is: By masking With feature map Element-level multiplication yields a feature map that eliminates occlusion of local human features. .

2. The fully supervised video pedestrian re-identification method according to claim 1, characterized in that, The training steps of the pre-trained person re-identification model include: Obtain a training sample set; wherein each training sample in the training sample set includes a sampled video clip containing pedestrians, and the ID number of the pedestrians contained in the video clip; During training, for selected training samples, the sampled video clips containing pedestrians are input into the pedestrian re-identification model to predict the ID numbers of pedestrians in the video clips and use them as prediction results. The prediction results are compared with the ID numbers of pedestrians contained in the video clips in the training samples. The model is trained under supervision using cross-entropy, triplet, and mutual information loss functions and the parameters are updated. After reaching the preset convergence condition, the pre-trained pedestrian re-identification model is obtained.

3. The fully supervised video pedestrian re-identification method according to claim 1, characterized in that, In the feature alignment module, the step of using a kernel correlation filtering algorithm to calculate the deviation of the input original video frames and outputting the cross-frame position deviation of the target person includes: The kernel correlation filter algorithm is used to calculate the cross-frame positional deviation of the target person in each video segment; where, for each frame in the video segment... Taking the average along the channel dimension transforms into ; Initialize the correlation filter, including: using the first frame The correlation filter is initialized using the following expression: ； ； In the formula, Represents the discrete Fourier transform. This represents the inverse discrete Fourier transform. It is a Gaussian regression target. It is the regularization coefficient. yes The complex conjugate, This represents the matrix element-wise multiplication operation; It is the correlation filter calculated on the first frame; Calculate the cross-frame positional deviation in the next frame, and then update the filter parameters; specifically, the cross-frame positional deviation in the second frame is calculated using a correlation filter, and the filter parameters are updated using an exponential moving average, as expressed by the following expression: ； ； In the formula, This is the response map of the second frame on the correlation filter, calculated... The distance from the maximum response point to the center can be used to obtain the cross-frame positional deviation of the target person in the second frame; The pixels in the second frame are aligned using a rolling method based on the target person's cross-frame positional deviation. The filter parameters are then updated using the aligned second frame, as expressed in the following expression: ； ； In the formula, It is the step size of the exponential moving average, and the filter parameters are updated by the exponential moving average. Repeat the above steps until the cross-frame positional deviation of the target person in all frames is calculated; where the filter updated in the second frame is used. To calculate the target person's cross-frame position deviation in the 3rd frame, and to update the filter using the 3rd frame after pixel roll alignment. The parameters are calculated repeatedly frame by frame to calculate the cross-frame position deviation of the target person and update the filter parameters until the cross-frame position deviation of the target person in all frames is calculated.

4. The fully supervised video pedestrian re-identification method according to claim 1, characterized in that, In the decoder module, the steps for performing feature interaction and fusion processing on the frame-level feature map after removing the local features of the occluded person based on a multi-head self-attention mechanism, and outputting video-level features are as follows: Feature maps were analyzed sequentially. This involves temporal and spatial interaction; specifically, local features with the same spatial location along the time dimension interact with each other temporally, expressed as follows: In the formula, , and These are the learnable weights of the query, key, and value, respectively. It is a feature map after time interaction; For feature maps Spatial interaction is achieved by spatially interacting all local features within the same frame, expressed as follows: In the formula, The feature map is obtained after spatiotemporal interaction with the decoder; Global average pooling is used to process the feature maps after spatiotemporal interaction. The features are then fused to obtain the final video-level features.

5. The fully supervised video pedestrian re-identification method according to claim 2, characterized in that, In the training steps of the pre-trained person re-identification model, the expression for the loss function is as follows: In the formula, This is the overall loss function of the model; It is a hyperparameter that balances the weights of the frame-level feature loss function and the video-level loss function; In the formula, Represents the triplet loss function. Represents the cross-entropy loss function. Indicates the category label in video-level features; In the formula, Represents the mutual information loss function. This represents the category label in the frame-level features.

6. A fully supervised video pedestrian re-identification system, characterized in that, include: The data acquisition module is used to acquire video clips containing the target person and video clips for pedestrian re-identification; The pedestrian re-identification result acquisition module is used to perform pedestrian re-identification processing using a pre-trained pedestrian re-identification model based on the acquired video segment containing the target person and the video segment to be re-identified, and output the pedestrian re-identification result; wherein, the pedestrian re-identification result includes at least whether the video segment to be re-identified contains the target person. The pedestrian re-identification model includes: The encoder module is used to extract features from raw video frames and output frame-level feature maps; wherein, the encoder module is based on the Vision Transformer architecture; The feature alignment module is used to perform deviation calculation on the input raw video frames using the kernel correlation filtering algorithm, and output the cross-frame position deviation of the target person. A decoder module is used to input the frame-level feature map and the cross-frame position deviation of the target person, perform feature alignment processing, and obtain an aligned frame-level feature map; utilize the different relative states of the target person and the occluding person to perform occluding person local feature removal processing on the aligned frame-level feature map, and obtain a frame-level feature map with occluding person local features removed; based on a multi-head self-attention mechanism, feature interaction and fusion processing are performed on the frame-level feature map with occluding person local features removed, and video-level features are output; wherein, the decoder module is a decoder based on a multi-head self-attention mechanism; based on the obtained cross-frame position deviation of the target person in adjacent frames, let the response map... The horizontal and vertical deviations of the maximum response point from the center are: and According to the deviation rolling feature map Align the feature maps to obtain the aligned feature maps. ; Aligned feature maps By taking the mean along the time dimension to focus on the relatively static parts of the sequence, a feature map is obtained. The expression is, ; Calculate frame-level feature maps and The cosine similarity is calculated using the following expression: In the formula, The value ranges from 0 to 1, representing Each pixel block and the averaged feature map Cosine similarity; This represents the vector dot product operation; features that obscure people are eliminated by adaptively generating a threshold based on cosine similarity, by respectively... Parameters are generated by taking the mean along the time and time-space dimensions. and To eliminate local features with cosine similarity less than the adaptively generated threshold, the expression is: By masking With feature map Element-level multiplication yields a feature map that eliminates occlusion of local human features. .

7. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the fully supervised video pedestrian re-identification method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the fully supervised video pedestrian re-identification method according to any one of claims 1 to 5.

Images