A video person re-identification method based on region augmentation and location information interaction

Abstract

The present invention claims protection for a video pedestrian re-identification method based on region augmentation and location information interaction, which belongs to the image retrieval method. It includes the following steps: data slicing and preprocessing; then using the channel attention module to reconstruct the channel weights; then extracting the salient features of the previous frame, and further transitioning the focus area of ​​the subsequent frame from salient to comprehensive; then using the location information interaction (LII) module to promote the interaction of location information across all frames to achieve mutual enhancement between frames; finally, designing a loss function based on the model and training it. It is worth noting that in the test phase, the LII module is removed to improve the inference speed. The present invention proposes a lightweight video pedestrian re-identification model that demonstrates strong performance and generalization capabilities on multiple public benchmarks.

Description

Technical Field

[0001] The present invention relates to an image retrieval method, and in particular to a video pedestrian re-identification method based on region expansion and position information interaction. Background Art

[0002] The task of pedestrian re-identification (Re-ID) aims to retrieve specific pedestrians across monitoring areas. It is an important technology in high-level applications such as intelligent monitoring, safe cities and incident detection. The input of the video-based Re-ID task is continuous.

[0003] Video sequences can provide more comprehensive clues based on temporal changes, but complex lighting, backgrounds, and occlusions hinder further development. While CNNs have made significant progress in video person re-identification over the past few years, effectively utilizing the rich temporal information between video frames remains a challenge.

[0004] Recently, some methods have used self-attention mechanisms and graph convolutional networks to promote the flow of information between video frames. Although the frame-level features obtained in this way are richer and more discriminative, and have achieved relatively advanced results in tests, they still have some defects.

[0005] First, for a single image frame, the region of interest is typically concentrated in a prominent but localized area. When a model focuses only on a portion of the pedestrian's area, it loses focus on other fine-grained information, limiting the expressive power of a single feature. Clearly, a robust Re-ID model should avoid this behavior. Expanding the region of interest to capture the complete features of a given person is crucial for further improving the model's discriminative power and robustness.

[0006] Second, these methods treat temporal relationships as mutually reinforcing or homogenous information flows across all frames. This approach results in richer frame-level feature embeddings because they contain mutually reinforcing information from other frames. However, this enhancement leads to increased similarity and even redundancy between individual frames. This similarity and redundancy not only sacrifices inter-frame differences but also limits further improvement in the final temporal fusion stage. Therefore, it is necessary to exploit temporal cues from other perspectives to amplify temporal differences and encourage information transfer across all frames.

[0007] CN112836646B, a video pedestrian re-identification method and application based on a channel attention mechanism, the method comprises the following steps: pre-processing the video sampling image and inputting it into a convolutional neural network, extracting N image feature maps respectively, and obtaining a temporary video-level feature map through temporal pooling; inputting the temporary video-level feature map into a channel attention module and outputting a feature vector; expanding and filling the feature vector in height and width to obtain an attention matrix, and multiplying the attention matrix element-by-element with the original N feature maps to obtain N modified feature maps; and successively performing temporal and spatial pooling to obtain a video-level feature vector, adding a classifier to obtain a classification result, calculating the cross entropy loss for the classification result, and optimizing the entire model training; and applying the trained pedestrian re-identification model to pedestrian re-identification.

[0008] CN112836646B, a video pedestrian re-identification method and application based on a channel attention mechanism. In this patent, all input features are temporally pooled to obtain a temporary video-level feature. If there is severe occlusion or misalignment in the input feature, the single video-level feature will lose its effectiveness and affect the learning of subsequent features. When constructing the channel attention module, the channel feature is reduced and then increased in dimension, which will destroy the direct correspondence between channels. When designing the loss function, the cross-entropy loss is conducive to learning inter-class information because it adopts an inter-class competition mechanism. However, it only cares about the accuracy of the probability of predicting the correct label and ignores the differences in other incorrect labels, resulting in the learned features being relatively scattered. In addition, the overall structure of this patent is simple, and its robustness is poor when encountering complex benchmarks.

[0009] In this patent, the design of the channel attention module makes the channel features pay more attention to the central foreground information through a simple assignment operation, and then establishes the relationship information of adjacent channels through one-dimensional convolution, which is lightweight and more accurate. In the loss function part, the cross entropy loss L is used. cent and triplet loss L tri To jointly guide the training process, in addition, in order to maintain the diversity between frames, the mutual information loss L is used mi To minimize the mutual information between embeddings of different frames, further improving overall performance; in terms of overall design, salient information is used to force the network to focus on more comprehensive areas, obtaining more detailed and accurate features, and finally some modules are removed to ensure the lightweight model. These are all things that the aforementioned patents do not take into account. Summary of the Invention

[0010] The present invention aims to solve the above problems in the prior art. It proposes a video pedestrian re-identification method based on region augmentation and location information interaction. The technical solution of the present invention is as follows:

[0011] A video pedestrian re-identification method based on region augmentation and location information interaction includes the following steps:

[0012] Step 1: Slice the input video benchmark data according to the model settings, and then perform preprocessing operations on the input video frames including cropping, random erasing, normalization, horizontal (vertical) flipping, and random padding;

[0013] Step 2: The original input features are processed using the Channel Attention Layer (CAL). Its purpose is to reconstruct channel weights, filter out errors and meaningless channels, and improve feature expression capabilities.

[0014] Step 3: After CAL, extract the previous frame salient features of the frame, where P is the feature map set from 0 to s after CAL processing, s is the slice position in step 1, and the extracted features will be used to suppress the salient information of subsequent frame-level features;

[0015] Step 4: Use the features generated above to suppress The salient regions that have been activated in P are represented by Q, where Q is the feature map set from s to t-1 after CAL processing, and t is the total number of input frames, so that subsequent frame-level features can focus on broader fine-grained information and obtain more complete feature representation;

[0016] Step 5: Transfer information from Deliver to Where X is the initial input tensor and Y is the final generated feature suppression map to improve the expressiveness of the feature;

[0017] Step 6: Integrate and distribute the global characteristics of frame information, establish connections between any two locations in the spatiotemporal dimension, enable information to be transmitted across all frames and achieve mutual enhancement between frames;

[0018] Step 7: Design a loss function based on the model settings to guide the training process;

[0019] Step 8: During the training phase, the above operations are used to implement information interaction between features and the transition from significant to comprehensive features to assist in the training of the backbone ResNet-50. During the testing phase, some branches are abandoned to improve the inference rate and ensure the lightweight model. Finally, mAP (Mean Average Precision) and Rank are calculated to evaluate the model performance.

[0020] The advantages and beneficial effects of the present invention are as follows:

[0021] The innovation of the present invention is mainly reflected in the following three aspects:

[0022] In the data preprocessing part, in addition to the conventional methods such as random erasing, normalization and cropping, a new random filling method is proposed to further improve the robustness of the model by appropriately increasing the probability of inaccurate pedestrian detection box scenarios.

[0023] In the methodology section, SFT achieves a shift from significant to comprehensive. It innovatively leverages temporal relationships to amplify inter-frame differences, gradually expanding the attention area of ​​consecutive frames. This makes frame-level features more complete and diverse, thereby generating more informative video-level features after temporal fusion. The performance of SFT depends on the richness of frame-level information. To this end, the LII module is introduced to integrate and distribute global information, enabling feature transfer across all frames and promoting information exchange between any two locations in the spatiotemporal dimension.

[0024] Another advantage of LII is that affinity graphs are learnable, automatically discovering useful patterns from the training distribution. Secondly, the order in which matrices of different dimensions are learned is optimized, increasing the flexibility of combining features. Finally, LII establishes connections between arbitrary spatial locations across different frames, greatly facilitating the transfer of information.

[0025] In terms of the overall structure, the complete structure is maintained during the training phase, and some branch structures are removed during the testing phase. This ensures test accuracy while improving the deduction speed and enhancing its feasibility in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The present invention provides a flowchart of a video pedestrian re-identification method based on region augmentation and location information interaction in a preferred embodiment. DETAILED DESCRIPTION

[0027] The following will describe the technical solutions in the embodiments of the present invention in detail with reference to the accompanying drawings. The described embodiments are only a part of the embodiments of the present invention.

[0028] The technical solution of the present invention to solve the above technical problems is:

[0029] As shown in the figure, the present invention provides a video pedestrian re-identification method based on region augmentation and location information interaction, comprising the following steps:

[0030] Step 1: The input of the SFT framework is two items: feature map and segmentation position. For feature map, SFT uses the mid-level feature map with detailed semantic information. Given a slice Contains t frames, using the backbone Res-Net50 network to obtain the intermediate layer feature maps Here c represents the number of channels, h represents the height, and w represents the width.

[0031] For the segmentation position s, it determines which frames in the previous and next frames need to be suppressed. In short, SFT captures X 0,…,s-1 The salient features in the late frame X s,…,t-1 The notable features of .

[0032] The input video frame is then preprocessed, including cropping, random erasing, normalization, horizontal (vertical) flipping, and random padding, a newly proposed data augmentation method. Specifically, the original image size is C×W×H, and the generated noise image size is C×(d×W)×(d×H), where d is a random number generated in the range (1, 1+r) and r is a constant. The original image is then superimposed on a random position of the noise image, and finally a resize operation is used to reduce it to the original image size as input.

[0033] Step 2: The first module of SFT is the channel attention module, which aims to filter out misleading and meaningless channels. Due to the zero padding of the model, some channels may be concentrated in the periphery, tending to select background as salient features. This may mislead the later suppression because salient pedestrians are usually in the center of the input frame. To this end, the channel weights are generated as follows:

[0034]

[0035] in yes Cascade, X no-pad is a subregion of X without the first and last rows / columns, GAP is a global average pooling layer, W c is an estimate of each accessibility centrality, that is, the channel focusing on the central foreground information is assigned a value of 1, and the other channels are assigned a value of 0.

[0036] The channel attention module also uses ECA-Net to establish the relationship information of adjacent channels. The result is a c-dimensional vector W e , represents the weight of each channel.

[0037] By applying W to the input feature map c and W e , CAL will return more focused and meaningful feature maps. For clarity, the returned feature maps are renamed according to the segmentation position s. and

[0038] P i =W c W e ·X i,i=0,…,s-1

[0039] Q i =W c W e ·X i ,i=s,…,t-1

[0040] Step 3: After the channel attention module, SFT generates a k*k (k: kernel size) convolution kernel to include more The specific operations are as follows:

[0041] The input tensor is Depend on Then use the multi-head attention mechanism to weight the importance of each position and return the attention map To further increase the diversity and information richness of attention maps, it is necessary to avoid A P The collapse of k 2 L1-Normalization is used in the s×h×w and s×h×w dimensions respectively, and the feature map and attention map are multiplied by matrices to obtain a convolution kernel of size k*k. For the sake of brevity, P and A P Temporarily reshaped as and Where u = s × h × w.

[0042] Step 4: In this part, use the kernel generated above to suppress Based on the salient regions that have been activated in P, subsequent frames can focus on a wider area and obtain a more complete feature representation.

[0043] First, the input feature Q i Convolution operation is performed with the reconstruction kernel K of size c×k×k.

[0044] R i =softmax(Q i *K)

[0045] Where * is the convolution operation, and the softmax normalization operation is performed on the h×w dimension. The final R i It is an affinity matrix that gives higher weights to the features captured in the previous frame P and maps the affinity to R i Convert to inhibition matrix S i , which is achieved through the following operations.

[0046]

[0047] Where [a] - =min{a,0}, output matrix S iBetween 0 and 1. β is a hyperparameter of the variance of the transformed distribution. A higher β indicates heavier suppression, i.e., giving lower weight to significant features.

[0048] Then, S i With X i Multiply to generate the final suppression feature map: Y i =S i ×X i In this process, the salient regions will be S i It is worth noting that the convolution in the equation actually measures patch-wise affinity rather than point-wise affinity. So the result S i and R i Will be more continuous and smooth. If Y i , which may complicate the modeling of local relationships in subsequent convolutions.

[0049] In general, for the subsequent frames, SFT reduces the weight of the salient regions, forcing the network to pay more attention to other regions with fine-grained information. The attention regions of consecutive frames are therefore shifted from salient to wider regions.

[0050] Step 5: In the above operation, due to the suppression of salient features, the information loss may also deteriorate the frame In order to improve the representation ability, information is encouraged from Deliver to The specific expression is as follows:

[0051]

[0052] Among them, α is used to control the degree of transmission, Z i Indicates integration X i and Y i The discriminant features obtained, GAP means global average pooling.

[0053] Step 6: The performance of the SFT module depends largely on the information richness of the attention region in each frame. The greater the amount of information in each frame, the more powerful the final video-level representation of the SFT will be.

[0054] To enrich frame-level representations, recent methods have primarily employed self-attention mechanisms, treating one position as a similarity-based aggregate to enrich features across all positions. However, non-local behavior is more critical than similarity-based aggregation strategies. Based on this, ensemble and distribution structures are employed to approximate input-agnostic affinity mappings, thereby preserving non-local behavior.

[0055] Specifically, the Location Information Interaction (LII) module integrates key functions / channels and distributes them in a fixed pattern. First, the feature map of the middle layer Reshape into F∈R c×m (m=t×h×w). Then a series of linear transformations are performed on F to achieve the information flow between positions. The form is as follows:

[0056] F′=G D G I F(L I L D +I)

[0057] Where G and L represent linear transformations in c (channel) and m (space-time) dimensions, respectively, and the superscripts I and D represent integration and distribution, respectively. In general, m′≤m, c′≤c. Identity matrix The entire process of LII is similar to the encoder-decoder structure, where the integral operation is used to extract key features, and the distribution operation attempts to use these key features to restore the original feature map. To facilitate integration with the backbone network, the output of LII is in residual form.

[0058] F output =F+F′

[0059] LII can also be viewed from the perspective of an “affinity diagram”. In fact, the matrix L I L D yes The low-rank decomposition of L is an affinity graph that measures the similarity between any two locations. Therefore, L enables information transfer between any two locations in the spatiotemporal dimension, while G represents the channel dimension. In this way, LII establishes connections between any two locations in different frames, or even different channels, achieving mutual enhancement between frames.

[0060] Step 7: Use L cent (CrossEntropy Loss) and L tri (Batch Hard Triplet Loss) is used to guide the training process. In addition, in order to maintain the diversity between frames, the mutual information loss L is used mi To minimize the mutual information of embeddings of different frames. The specific expression is as follows:

[0061] L all =L cent +λ1L tri +λ2L mi

[0062] Step 8: When using the complete structure, its computational cost and additional memory consumption have certain advantages compared to general models. To make the model more lightweight, a flexible design is proposed. The LII module is discarded during the test phase, and the model performance is evaluated by calculating the mAP (Mean Average Precision) and Rank indicators.

[0063] The supervisory signal with spatiotemporal information is back-propagated from the LII back to the CNN. As a result, the CNN feature maps receive feedback from a more diverse set of spatiotemporal locations, especially for shallow layers with smaller receptive fields. Once the model is trained, the entire LII module can be discarded, reducing the computational burden without affecting overall performance.

[0064] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0065] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0066] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0067] The above embodiments should be understood as merely illustrating the present invention and not as limiting the scope of protection of the present invention. After reading the contents of the present invention, technicians may make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A video pedestrian re-identification method based on region augmentation and location information interaction, characterized in that: The following steps are involved: Step 1: Slice the input video benchmark data, and then perform preprocessing operations on the input video frames including cropping, random erasing, normalization, horizontal or vertical flipping, and random padding; Step 2: The original input features are processed using the channel attention layer CAL to reconstruct channel weights and filter out errors and meaningless channels; Step 3: After CAL, extract the previous frame salient features of the frame, where P is the feature map set from 0 to s after CAL processing, s is the slice position in step 1, and the extracted features will be used to suppress the salient information of subsequent frame-level features; Step 4: Use the generated features to suppress The salient regions that have been activated in P are represented by Q, where Q is the feature map set from s to t-1 after CAL processing, and t is the total number of input frames, so that subsequent frame-level features focus on broader fine-grained information and obtain more complete feature representation; Step 5: Transfer information from Deliver to Where X is the initial input tensor and Y is the final generated feature suppression map; Step 6: Integrate and distribute the global characteristics of frame information, establish connections between any two locations in the spatiotemporal dimension, enable information to be transmitted across all frames and achieve mutual enhancement between frames; Step 7: Design a loss function based on the model settings to guide the training process; Step 8: During the training phase, steps 1-7 are used to implement information interaction between features and transition from significant to comprehensive features to assist in the training of the backbone ResNet-50. During the testing phase, some branches are abandoned to improve the inference rate and ensure the lightweight model. Finally, mAP and Rank are calculated to evaluate the model performance. The step 1 is specifically as follows: From significant to comprehensive transition, the SFT framework has two inputs: feature map and segmentation position. For feature map, SFT uses the mid-level feature map with detailed semantic information. Given a slice Contains t frames, using the backbone Res-Net50 network to obtain the intermediate layer feature maps Here c represents the number of channels, h represents the height, and w represents the width; For the segmentation position s, it determines which frames in the previous frame and the next frame need to be suppressed, that is, SFT will capture X 0,…,s-1 The salient features in the suppressed frame X s,…,t-1 The salient features of The input video frame is then preprocessed, including cropping, random erasing, normalization, horizontal or vertical flipping, and random padding. Random padding is a newly proposed data augmentation method. Specifically, the original image size is C×W×H, and the generated noise image size is C×(d×W)×(d×H), where d is a random number generated in the range (1,1+r) and r represents a constant. The original image is then superimposed on a random position of the noise image, and finally a resize operation is used to reduce it to the original image size as input. The step 7 is specifically as follows: Using cross entropy loss L cent and triplet loss L tri to jointly guide the training process; in addition, in order to maintain the diversity between frames, the mutual information loss L is used mi To minimize the mutual information of embeddings of different frames; the specific expression is as follows: L all =L cent +λ1L tri +λ2L mi 。 2. The video pedestrian re-identification method based on region augmentation and location information interaction according to claim 1 is characterized in that: The step 2 is specifically as follows: The first module of SFT is the channel attention module, which aims to filter out misleading and meaningless channels; the channel weights are generated as follows: in yes Cascade, X no-pad is a subregion of X without the first and last rows / columns, GAP is a global average pooling layer, W c It is an estimation of each accessibility centrality, that is, the channel focusing on the central foreground information is assigned a value of 1, and the other channels are assigned 0; The channel attention module also uses ECA-Net to establish the relationship information of adjacent channels; The result is a c-dimensional vector W e , represents the weight of each channel; By applying W to the input feature map c and W e , CAL will return a more focused and meaningful feature map; the returned feature map will be renamed according to the segmentation position s and 3. The video pedestrian re-identification method based on region augmentation and location information interaction according to claim 2 is characterized in that: The step 3 is specifically as follows: After the channel attention module, SFT generates a k*k convolution kernel, where k represents the kernel size to include more The specific operations are as follows: The input tensor is Depend on Then use the multi-head attention mechanism to weight the importance of each position and return the attention map To further increase the diversity and information richness of attention maps, it is necessary to avoid A P The collapse of k 2 L1-Normalization is used in the s×h×w and s×h×w dimensions respectively, and the feature map and attention map are multiplied by matrices to obtain a convolution kernel of size k*k. For the sake of brevity, P and A P Temporarily reshaped as and Where u = s × h × w.

4. The video pedestrian re-identification method based on region augmentation and location information interaction according to claim 3 is characterized in that: The step 4 is specifically as follows: In this section, the kernel generated above is used to suppress For the salient regions that have been activated in P, subsequent frames can focus on a wider area to obtain a more complete feature representation; First, the input feature Q i Perform convolution operation with the reconstruction kernel K of size c×k×k; R i =softmax(Q i *K) Where * is the convolution operation, and the softmax normalization operation is performed on the h×w dimension; the final R i It is an affinity matrix that gives higher weights to the features captured in the previous frame P and maps the affinity to R i Convert to inhibition matrix S i , achieved through the following operations; Where [a] - =min{a,0}, output matrix S i Between 0 and 1; β is a hyperparameter of the variance of the transformed distribution; a higher β indicates heavier suppression, that is, giving lower weight to significant features; Then, S i With X i Multiply to generate the final suppression feature map: Y i =S i ×X i ; In this process, the salient areas will be S i The low-weight multiplication in the equation actually measures the patch-wise affinity rather than the point-wise affinity; so the result S i and R i will be more continuous and smoother; if Y i Discontinuities are exhibited in , which may complicate the modeling of local relationships in subsequent convolutions; In general, for the later frames, SFT reduces the weights of salient regions, forcing the network to pay more attention to other regions with fine-grained information; the attention regions of consecutive frames are therefore shifted from salient to wider regions.

5. The video pedestrian re-identification method based on region augmentation and location information interaction according to claim 4 is characterized in that: The step 5 is specifically as follows: In order to improve the representation ability, information is encouraged to Deliver to The specific expression is as follows: Among them, α is used to control the degree of transmission; Z i Indicates integration X i and Y i The discriminant features obtained, GAP means global average pooling.

6. The video pedestrian re-identification method based on region augmentation and location information interaction according to claim 5 is characterized in that: The step 6 is specifically as follows: The performance of the SFT module depends largely on the information richness of the attention region in each frame. The greater the amount of information in each frame, the more powerful the final video-level representation of the SFT will be. Adopting an integrated and distributed structure to approximate input-agnostic affinity mapping, thus preserving non-local behavior; The location information interaction LII module will integrate key functions / channels and then distribute them through a fixed pattern; first, the feature map of the middle layer Reshape into F∈R c×m (m = t × h × w); then a series of linear transformations are performed on F to achieve the flow of information between positions; the form is as follows: F′=G D G I F(L I L D +I) Where G and L represent the linear transformation in c channels and m space-time dimensions, respectively, and the superscripts I and D represent integration and distribution, respectively; m′≤m, c′≤c, identity matrix The entire process of LII is similar to the structure of encoder-decoder, where the integral operation is used to extract key features, and the distribution operation attempts to use these key features to restore the original feature map. In order to facilitate integration with the backbone network, the output of LII is in residual form; F output =F+F′ Matrix L I L D yes The low-rank decomposition of L is an affinity graph that measures the similarity between any two positions; therefore, L realizes the information transfer between any two positions in the spatiotemporal dimension, and G represents the channel dimension; in this way, LII establishes a connection between any two positions in different frames or even different channels, achieving mutual enhancement between frames.

7. The video pedestrian re-identification method based on region augmentation and location information interaction according to claim 6 is characterized in that: The step 8 is specifically as follows: In order to make the model more lightweight, a flexible design is proposed to discard the LII module in the test phase and evaluate the model performance by calculating the mAP and Rank indicators; The supervisory signal with spatiotemporal information is returned from LII to CNN via back-propagation; thus, the CNN feature maps receive feedback from a more diverse set of spatiotemporal locations.

Images