A video pedestrian re-identification method based on spatio-temporal affine subspace

Abstract

This invention relates to the field of video pedestrian re-identification technology, and in particular to a video pedestrian re-identification method based on a spatiotemporal affine subspace. It extracts deep features from video frames using a visual Transformer and introduces an adaptive spatiotemporal information coupling module. 3D convolution is used to collaboratively fuse spatiotemporal features in local regions, enhancing the model's understanding of the overall video context. Subsequently, the entire video sequence is modeled as an affine subspace composed of an average appearance origin and dynamically changing orthogonal bases, effectively preserving the structured information of pedestrians. Finally, a principal angle metric is used to accurately calculate the similarity between subspaces, achieving efficient and accurate pedestrian retrieval. This application effectively improves the discriminative power and robustness of feature representations, significantly increases the accuracy of video pedestrian re-identification, and reduces recognition errors.

Description

Technical Field

[0001] This invention relates to the field of video pedestrian re-identification technology, and in particular to a video pedestrian re-identification method based on spatiotemporal affine subspace. Background Technology

[0002] Video person re-identification, a key technology in the field of video surveillance, has received considerable attention in recent years. The problem it aims to solve is correctly identifying the same pedestrian in cross-camera scenarios. Currently, this technology is widely used in pedestrian tracking, intelligent security systems, and intelligent monitoring systems.

[0003] Video data offers richer representations than traditional image-based person re-identification, along with temporal features not found in single-frame images. However, it also presents various challenges, such as differences in viewpoint, lighting, and varying degrees of occlusion between different cameras.

[0004] To fully exploit the representations and temporal features inherent in videos and address various challenges, the development of Convolutional Neural Networks (CNNs) has provided a powerful tool for this field, with many previous studies achieving significant results after their introduction. Meanwhile, some studies have employed Graph Convolutional Networks (GCNs) to construct relationships between frames or parts, achieving even better results. Although both CNN-based and GCN-based video re-recognition schemes have been successful, the former is typically limited by a fixed receptive field, while the latter relies on prior or external information and lacks flexibility. The Visual Transformer (ViT), which has been widely explored in recent years, can promote interaction based on a multi-head self-attention mechanism, thereby acquiring complex visual patterns, which effectively alleviates the limitations of traditional CNNs.

[0005] Nevertheless, existing ViT-based methods for re-identifying pedestrians in videos focus on extracting spatial or temporal features, ultimately fusing them into a single feature vector as the overall representation of the pedestrian in the video. These methods process features separately: first extracting spatial features, then performing temporal processing, or vice versa. However, spatiotemporal information is often closely intertwined, especially in video tasks. Therefore, this separate processing approach cannot collaboratively learn the complex relationships inherent in the video, resulting in a lack of overall contextual understanding of the video sequence and consequently low pedestrian re-identification accuracy. Furthermore, no matter how complex existing methods are, they ultimately fuse all spatiotemporal features to obtain a comprehensive video vector feature representation of the person. A single comprehensive video vector feature cannot represent the inherent data structure necessary for video pedestrian re-identification, leading to large re-identification errors and low accuracy. Summary of the Invention

[0006] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the existing technology first separates and obtains time features and spatial features, and then fuses them into comprehensive features, resulting in insufficient feature representation ability and thus low accuracy of pedestrian re-identification.

[0007] To address the aforementioned technical problems, this invention provides a video pedestrian re-identification method based on spatiotemporal affine subspace, comprising:

[0008] Obtain the set of video sequences to be matched and the set of candidate video sequences, and calculate the affine subspace of all video sequences; based on the principal angle metric, calculate the distance between the affine subspace of the video sequence to be matched and each video sequence in the set of candidate video sequences, and sort the video sequences in the set of candidate video sequences in ascending order of distance to obtain the set of target video sequences with descending similarity to the video sequence to be matched.

[0009] The video sequence is input into the re-identification model, passing sequentially through a cascaded feature embedding layer, multiple visual encoders, and an affine subspace modeling module. The output affine subspace of the video sequence includes:

[0010] The video sequence is input into the feature embedding layer, and block convolution is performed on each frame of the video sequence to obtain multiple block feature vectors for each frame. A category feature vector with the same dimension as the block feature vector is set for each frame, and it is concatenated with multiple block feature vectors to form the embedding features of each frame.

[0011] The embedding features of each frame image are input into the first visual encoder to extract and couple spatiotemporal information, and output the spatiotemporal coupling features of each frame image. The output of the current visual encoder is used as the input of the next visual encoder to obtain the spatiotemporal coupling features of each frame image output by the last visual encoder, which are used as the target frame features of each frame image.

[0012] The target frame features of each frame image are input into the affine subspace modeling module. Temporal average pooling is performed on the target frame features of all frames to obtain the origin of the affine subspace. Singular value decomposition is performed on the matrix composed of the target frame features of all frames. The first preset number of column vectors in the left singular vector matrix after decomposition are used as the orthonormal basis. The set of the origin and the preset number of orthonormal bases is used as the affine subspace of the video sequence.

[0013] Preferably, based on the principal angle metric, the distance between the video sequence to be matched and the affine subspace of each video sequence in the candidate video sequence set is calculated, including:

[0014] Obtain the set of orthonormal bases for the affine subspace of the video sequence to be matched. ;

[0015] Obtain the set of orthonormal bases for the affine subspace of any video sequence in the candidate video sequence set. ;

[0016] Singular value decomposition of the orthonormal basis set of the affine subspaces of the two video sequences. , is represented as: ;

[0017] Obtain the diagonal matrix after singular value decomposition , with diagonal matrix diagonal elements, for Cosine values ​​of the principal angles;

[0018] based on Calculate the cosine values ​​of the principal angles. The sum of the squares of the principal angles is used to obtain the distance between the affine subspace of the video sequence to be matched and any video sequence in the candidate video sequence set, expressed as:

[0019] ;

[0020] in, Indicates the first The cosine value of each principal angle, , This represents the total number of orthonormal bases.

[0021] Preferably, the embedding features of each frame image are obtained, including:

[0022] For video sequences Each frame of the image is divided into... Each image block is converted into a... dimensional vector;

[0023] Add a category feature vector to each frame of the image to capture the overall features of the entire image, and... Image patches Concatenate 3D vectors to obtain each frame of the image. Embedding features of dimensions;

[0024] in, This indicates the number of image frames in a video sequence. Indicates the number of image channels. and These represent the height and width of the image, respectively. This represents the dimension of a block in ViT.

[0025] Preferably, the embedding features of each frame image are input into the first visual encoder for spatiotemporal information extraction and coupling, and the spatiotemporal coupling features of each frame image are output, including:

[0026] The embedded features of the image are input through multiple concatenated ViT layers, and the output is the target visual features;

[0027] The target visual features are input into the adaptive spatiotemporal information coupling module, spatiotemporal convolution is performed, and the spatiotemporal coupling features of the image are output.

[0028] Preferably, the embedded features of the image are input through multiple concatenated ViT layers to output target visual features, including:

[0029] The embedded features of the image are input into the first ViT layer, and the self-attention mechanism and feedforward network are used to capture global context dependencies and perform nonlinear transformations to output the visual features of the image; the visual features include a global image feature and multiple local feature vectors.

[0030] The visual features output from the current ViT layer are used as the input to the next ViT layer, and the visual features output from the last ViT layer are obtained as the target visual features.

[0031] Preferably, the target visual features are input into the adaptive spatiotemporal information coupling module, spatiotemporal convolution is performed, and the spatiotemporal coupling features of the image are output, including:

[0032] Divide multiple local feature vectors in the target visual features into Each patch is reconstructed according to the original spatial layout of the image and arranged into... The vector matrix, Indicates the size of the patch;

[0033] Divide the vector matrix into Each region, obtain Each initial region feature is then subjected to a 3D spatiotemporal convolution, and the resulting features are added to the initial region features to obtain the optimized region features for each region.

[0034] By stitching together the optimized regional features of each region, the spatiotemporal coupling features of the image are obtained.

[0035] Preferably, in the visual encoder, the input features input to the visual encoder are compressed after being output through multiple ViT layers, and the compressed features are then input to the adaptive spatiotemporal information coupling module of the visual encoder.

[0036] Preferably, the target frame features of each frame are input into the affine subspace modeling module, and temporal average pooling is performed on the target frame features of all frames to obtain the origin of the affine subspace, represented as:

[0037] , ;

[0038] in, Represents the origin of the affine subspace. This indicates the number of image frames in a video sequence. This represents the set of target frame features for each frame of an image in a video sequence. Indicates the first video sequence Target frame features of the frame image .

[0039] Preferably, singular value decomposition is performed on the matrix composed of target frame features of all frame images, using the first preset number of column vectors in the decomposed left singular vector matrix as an orthonormal basis, including:

[0040] A matrix composed of the target frame features of all frames in the video sequence Singular value decomposition of the transpose , is represented as: ;

[0041] Obtain the left singular vector matrix after decomposition Center front column vectors for A standard orthogonal basis.

[0042] Preferably, the video sequence is preprocessed, including:

[0043] For each frame of the video sequence, data augmentation is performed using random horizontal flipping and random erasure;

[0044] Each frame of the augmented image is converted into tensor data, and the pixel values ​​are scaled to the range [0,1]. Then, the mean or variance of ImageNet is used for normalization to obtain the corresponding RGB image of each frame, forming a preprocessed video sequence.

[0045] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0046] The video person re-identification method based on spatiotemporal affine subspaces described in this invention abandons the approach of compressing the entire video sequence into a single feature vector, and instead uses affine subspaces as the final feature representation of the video. An affine subspace consists of an origin representing the average static appearance and a set of orthonormal bases representing the main dynamic change patterns. This allows for a richer and more structured description of the various poses and viewpoint changes exhibited by pedestrians throughout the video than a single vector, making the model more robust to interference factors such as illumination and viewpoint changes, effectively reducing re-identification errors. Furthermore, this invention uses a principal angle metric to calculate the distance between two affine subspaces to evaluate the similarity between two video sequences. The principal angle metric can geometrically measure the similarity of the structures of two subspaces, ensuring theoretical consistency and efficiency throughout the entire process from feature representation to similarity comparison, further improving the accuracy and reliability of person re-identification.

[0047] The visual encoder of this invention comprises multiple ViT layers and an adaptive spatiotemporal information coupling module. This adaptive spatiotemporal information coupling module is designed as a module with invariant input and output dimensions, thus it can be flexibly inserted into different ViT layers as a plug-and-play unit without disrupting the original architecture. During the mid-stage of feature extraction, the adaptive spatiotemporal information coupling module directly and synchronously aggregates spatiotemporal information in local regions through 3D convolution, and fuses it with the original features through residual connections. This enables the model to simultaneously capture spatial appearance features and short-term motion changes, achieving deep fusion and collaborative learning of spatiotemporal information. The resulting features contain richer dynamic discriminative information, thereby significantly improving the accuracy of person re-identification. Attached Figure Description

[0048] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0049] Figure 1 This is a flowchart of the steps of the video pedestrian re-identification method based on spatiotemporal affine subspace provided by the present invention;

[0050] Figure 2 This is a schematic diagram of the pedestrian re-identification model structure provided by the present invention. Detailed Implementation

[0051] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0052] Reference Figure 1The flowchart shown illustrates the steps of the pedestrian re-identification method based on spatiotemporal affine subspace of the present invention. The specific steps include:

[0053] Obtain the set of video sequences to be matched and the set of candidate video sequences, and calculate the affine subspace of all video sequences; based on the principal angle metric, calculate the distance between the affine subspace of the video sequence to be matched and each video sequence in the set of candidate video sequences, and sort the video sequences in the set of candidate video sequences in ascending order of distance to obtain the set of target video sequences with descending similarity to the video sequence to be matched.

[0054] The video sequence is input into the re-identification model, passing sequentially through a cascaded feature embedding layer, multiple visual encoders, and an affine subspace modeling module. The output affine subspace of the video sequence includes:

[0055] The video sequence is input into the feature embedding layer, and block convolution is performed on each frame of the video sequence to obtain multiple block feature vectors for each frame. A category feature vector with the same dimension as the block feature vector is set for each frame, and it is concatenated with multiple block feature vectors to form the embedding features of each frame.

[0056] The embedding features of each frame image are input into the first visual encoder to extract and couple spatiotemporal information, and output the spatiotemporal coupling features of each frame image. The output of the current visual encoder is used as the input of the next visual encoder to obtain the spatiotemporal coupling features of each frame image output by the last visual encoder, which are used as the target frame features of each frame image.

[0057] The target frame features of each frame image are input into the affine subspace modeling module. Temporal average pooling is performed on the target frame features of all frames to obtain the origin of the affine subspace. Singular value decomposition is performed on the matrix composed of the target frame features of all frames. The first preset number of column vectors in the left singular vector matrix after decomposition are used as the orthonormal basis. The set of the origin and the preset number of orthonormal bases is used as the affine subspace of the video sequence.

[0058] Specifically, based on the principal angle metric, the distance between the video sequence to be matched and the affine subspace of each video sequence in the candidate video sequence set is calculated, including...

[0059] Obtain the set of orthonormal bases for the affine subspace of the video sequence to be matched. ;

[0060] Obtain the set of orthonormal bases for the affine subspace of any video sequence in the candidate video sequence set. ;

[0061] Singular value decomposition of the orthonormal basis set of the affine subspaces of the two video sequences. , is represented as: ;

[0062] Obtain the diagonal matrix after singular value decomposition , with diagonal matrix diagonal elements, for Cosine values ​​of the principal angles;

[0063] based on Calculate the cosine values ​​of the principal angles. The sum of the squares of the principal angles is used to obtain the distance between the affine subspace of the video sequence to be matched and any video sequence in the candidate video sequence set, expressed as: ;

[0064] in, Indicates the first The cosine value of each principal angle, , This represents the total number of orthonormal bases.

[0065] In this embodiment, obtaining the affine subspace of the video sequence includes:

[0066] The target frame features of each frame are input into the affine subspace modeling module. Temporal average pooling is performed on the target frame features of all frames to obtain the origin of the affine subspace, represented as: , ;

[0067] in, Represents the origin of the affine subspace. This indicates the number of image frames in a video sequence. This represents the set of target frame features for each frame of an image in a video sequence. Indicates the first video sequence Target frame features of the frame image .

[0068] A matrix composed of the target frame features of all frames in the video sequence Singular value decomposition of the transpose , is represented as: ;

[0069] Obtain the left singular vector matrix after decomposition Center front column vectors for A standard orthogonal basis;

[0070] Interchange the origin with By splicing together several orthogonal bases, a representation of the affine subspace is obtained. .

[0071] The video person re-identification method based on spatiotemporal affine subspaces described in this invention abandons the approach of compressing the entire video sequence into a single feature vector, and instead uses affine subspaces as the final feature representation of the video. An affine subspace consists of an origin representing the average static appearance and a set of orthonormal bases representing the main dynamic change patterns. This allows for a richer and more structured description of the various poses and viewpoint changes exhibited by pedestrians throughout the video than a single vector, making the model more robust to interference factors such as illumination and viewpoint changes, effectively reducing re-identification errors. Furthermore, this invention uses a principal angle metric to calculate the distance between two affine subspaces to evaluate the similarity between two video sequences. The principal angle metric can geometrically measure the similarity of the structures of two subspaces, ensuring theoretical consistency and efficiency throughout the entire process from feature representation to similarity comparison, further improving the accuracy and reliability of person re-identification.

[0072] In this embodiment, the embedding features of each frame image are obtained, including:

[0073] For video sequences Each frame of the image is divided into... Each image block is converted into a... dimensional vector;

[0074] Add a category feature vector to each frame of the image to capture the overall features of the entire image, and... Image patches Concatenate 3D vectors to obtain each frame of the image. Embedding features of dimensions;

[0075] in, This indicates the number of image frames in a video sequence. Indicates the number of image channels. and These represent the height and width of the image, respectively. This represents the dimension of a block in ViT.

[0076] In this embodiment, the embedding features of each frame image are input into the first visual encoder for spatiotemporal information extraction and coupling, and the spatiotemporal coupling features of each frame image are output, including:

[0077] The embedded features of the image are input through multiple concatenated ViT layers, and the output target visual features include:

[0078] The embedded features of the image are input into the first ViT layer, and the self-attention mechanism and feedforward network are used to capture global context dependencies and perform nonlinear transformations to output the visual features of the image; the visual features include a global image feature and multiple local feature vectors.

[0079] The visual features output by the current ViT layer are used as the input to the next ViT layer, and the visual features output by the last ViT layer are obtained as the target visual features.

[0080] The target visual features are input into the adaptive spatiotemporal information coupling module, spatiotemporal convolution is performed, and the spatiotemporal coupling features of the image are output, including:

[0081] Divide multiple local feature vectors in the target visual features into Each patch is reconstructed according to the original spatial layout of the image and arranged into... The vector matrix, Indicates the size of the patch;

[0082] Divide the vector matrix into Each region, obtain Each initial region feature is then subjected to a 3D spatiotemporal convolution, and the resulting features are added to the initial region features to obtain the optimized region features for each region.

[0083] By stitching together the optimized regional features of each region, the spatiotemporal coupling features of the image are obtained.

[0084] In this embodiment, in the visual encoder, the input features of the visual encoder are compressed after being output through multiple ViT layers, and then input to the adaptive spatiotemporal information coupling module of the visual encoder.

[0085] The visual encoder of this invention comprises multiple ViT layers and an adaptive spatiotemporal information coupling module. This adaptive spatiotemporal information coupling module is designed as a module with invariant input and output dimensions, thus it can be flexibly inserted into different ViT layers as a plug-and-play unit without disrupting the original architecture. During the mid-stage of feature extraction, the adaptive spatiotemporal information coupling module directly and synchronously aggregates spatiotemporal information in local regions through 3D convolution, and fuses it with the original features through residual connections. This enables the model to simultaneously capture spatial appearance features and short-term motion changes, achieving deep fusion and collaborative learning of spatiotemporal information. The resulting features contain richer dynamic discriminative information, thereby significantly improving the accuracy of person re-identification.

[0086] Based on the above embodiments, in this embodiment of the invention, the pedestrian re-identification method based on spatiotemporal affine subspace provided by the present invention is used to perform pedestrian re-identification in RGB pedestrian video frame sequences, referring to... Figure 2 The diagram shown illustrates the structure of a pedestrian re-identification model. The specific re-identification process includes:

[0087] S201: Preprocess the RGB pedestrian video frame sequence data;

[0088] Each frame of the RGB pedestrian video frame sequence data is acquired, and data augmentation is performed using random horizontal flipping and random erasure.

[0089] Each frame of the augmented image is converted into tensor data, and the pixel values ​​of the image are scaled to the range [0,1]. Then, the mean or variance of ImageNet is used for normalization to obtain the corresponding RGB image of each frame.

[0090] S202: Input the preprocessed pedestrian video frame sequence into the re-identification model and output the affine subspace representation of the pedestrian video sequence;

[0091] The re-identification model includes a feature embedding layer, multiple visual encoders, and an affine subspace modeling module, which are sequentially connected along the forward propagation direction; wherein, the spatiotemporal information extraction coupling module includes a series of multi-layer visual Transformer layers and an adaptive spatiotemporal information coupling unit.

[0092] S202-1: Input the preprocessed pedestrian video frame sequence into the feature embedding layer, perform block convolution, and obtain the images of each frame. Image patch features;

[0093] For a given number Video sequence of individual pedestrians Each frame of the image is divided into... Each image block is converted into a... A dimensional vector; and an additional category feature (cls) capturing the overall features of the entire image is added to each frame image. The feature representation of the dimension is used to obtain the video sequence, which is represented as: ;

[0094] in, This indicates the number of image frames in a video sequence. Indicates the number of image channels. and These represent the height and width of the image in the video sequence, respectively. Indicates the number of blocks in the image. Indicates the dimension of a block in ViT;

[0095] In this embodiment, the feature embedding layer divides the entire image (256 high, 128 wide) into multiple 16x16 patches and turns them into a token. That is, an image is divided into 128 patches and turned into 128 token embeddings through convolution.

[0096] S202-2: Video sequence Input multiple cascaded visual encoders to obtain the target frame features of each frame of the image;

[0097] Each visual encoder includes A layered vision Transformer (ViT) layer coupled with an adaptive spatiotemporal information coupling module;

[0098] ① Layered visual Transformer layer;

[0099] Each frame of the image The feature representation of the dimension is input into the first ViT layer to perform global context modeling and obtain the global features of each frame of the image;

[0100] The output of the current ViT layer is used as the input of the next ViT layer to perform global context modeling and obtain the target global features after passing through N ViT layers.

[0101] Specifically, for the first ViT layer, its input is the output of the feature embedding layer; for the subsequent N-1 ViT layers, its input is the output of the ViT layer preceding it.

[0102] In this process, after every N layers of ViT, the number of channels is changed from... Compress to This makes the features more compact, resulting in higher computational efficiency. The compressed video sequence is represented as: , ;

[0103] ② Adaptive spatiotemporal information coupling module;

[0104] For each frame of image, extract separately Image patches 3D vector (excluding categorical features) ); and then rearrange all frame images according to the spatial distribution of the original features, as follows: The shape, This represents the width and height of each patch when the image is divided into patches; the rearranged video sequence is represented as: , ;

[0105] For each frame of the image, it is divided into z regions, where each region The dimension is Subsequently, a 3D spatiotemporal convolution is applied to each region to aggregate local spatiotemporal information. Then, all regions are concatenated together, restored to their original dimensions, and added to the features of regions that have not undergone spatiotemporal convolution to obtain optimized region features. , This represents the spatiotemporal convolution module;

[0106] The adaptive spatiotemporal information coupling unit provided in this embodiment will eventually restore the original dimension value of the visual coding layer, that is, its output dimension is consistent with the input dimension; since the dimension before and after has not changed, this module can be inserted between any layers.

[0107] The adaptive spatiotemporal information coupling module considers both temporal and spatial locality, mitigating the discrepancies between static image and dynamic video understanding. This module gradually integrates spatiotemporal features by coupling local spatiotemporal information from different stages of the network; it also collaborates with the ViT model to obtain more robust feature representations. This progressive fusion mechanism enhances temporal consistency, transforming frame-level representations into video-level semantic representations with only a 7.76% increase in parameters compared to the base network.

[0108] In summary, the visual encoder in this embodiment uses The final target frame features obtained are represented as follows:

[0109] ;

[0110] in, Indicates the first In the video sequence, the th Feature vector representation of a frame image ;

[0111] S202-3: Input the target frame features of each frame in the video sequence into the affine subspace modeling module to obtain the affine subspace of the video sequence;

[0112] After obtaining the target frame features of each frame in the video sequence, temporal average pooling is used to obtain the first frame. The origin of the affine subspace of a video sequence is denoted as: ;

[0113] in, The origin is represented by the affine subspace. In the affine subspace, the origin can be regarded as a bridge between the affine structure and the linear structure. In the video person re-identification task, the origin reflects the average value of the static representation of the video sequence, while the orthogonal basis conveys the discriminative dynamic features of the video sequence.

[0114] Perform singular value decomposition on all features:

[0115] , ;

[0116] Indicates the feature The right singular value vector obtained by performing singular value decomposition. Indicates the feature ;

[0117] Subsequently, take the corresponding to the previous Large eigenvalues ​​corresponding to Center front The affine subspace of a video sequence, represented by an orthogonal basis, is as follows: ;

[0118] S203: After obtaining the affine subspace of the video sequence, calculate the principal angle metric of the affine subspace of the video sequence and each video sequence in the candidate video sequence set:

[0119] To more accurately measure the differences between subspaces, this embodiment introduces a principal angle metric as the basis for difference assessment; specifically, it assumes there are two subspaces. and The distance between two subspaces can be calculated in the following way:

[0120] ; ;

[0121] in, This represents the set of cosine values ​​between the principal angles of corresponding origins in two subspaces. . and The first K columns represent the principal vectors corresponding to the K principal angles, respectively.

[0122] In this embodiment, the two video sequences used for principal angle measurement may be video sequences from two different cameras with the same ID, or they may be video sequences from different IDs.

[0123] This invention designs a spatiotemporal affine subspace network for video person re-identification and proposes a method to directly preserve the spatiotemporal structure of the video in the affine subspace. The invention designs an adaptive spatiotemporal information coupling module to enhance the feature extraction capability of the visual backbone network and simultaneously strengthen the temporal interaction during video feature extraction. Then, the extracted spatiotemporal features are projected into the affine subspace for video similarity measurement. In the subspace, each video segment is represented by a low-dimensional affine subspace spanned by its feature trajectories, where temporal dynamic features and spatial feature representations are jointly encoded into geometric attribute representations, namely, the subspace mean vector and feature vector. Furthermore, this invention introduces a principal angle measurement method to more accurately obtain the differences between subspaces. This invention shifts the measurement of video features from comparisons between vectors to measurements between subspaces, achieving more refined alignment in the spatiotemporal mode and effectively alleviating the problems of multi-view variations and dynamic occlusion.

[0124] Based on the above embodiments, in this embodiment of the invention, the video pedestrian re-identification method STARS based on spatiotemporal affine subspace provided in this embodiment is compared with several other video pedestrian re-identification methods, including M3D, AP3D, STRF, BiCnet-TKS, STMN, SANet, MFA, RFC, CAVIT, MSTAT, PiT, FIDN, SDCL, DAM, HASI, TCVIT, and MS-STI; as shown in Table 1, which is a brief overview of the comparison methods in this embodiment;

[0125] Table 1. Brief overview of the comparison method in this embodiment.

[0126]

[0127] Training and testing in this embodiment were both performed on an NVIDIA RTX 3090 GPU. During the training phase, STARS was optimized using Adam, employing label smoothing cross-entropy loss, ID loss, and triplet loss. This embodiment was set to a total of 300 epochs, starting with 10 warm-up epochs and an initial learning rate of 0.000005, decreasing by a factor of 10 at the 100th and 200th epochs, respectively. The test results of this embodiment's method compared to the contrasting methods are shown in Table 2.

[0128] Table 2 Comparison of test results between the method of this embodiment and the comparative method

[0129]

[0130] Where C indicates that the backbone in this method is CNN, and T indicates that the backbone in this method is ViT; based on Table 2, it can be seen that the video person re-identification method proposed in this embodiment outperforms most state-of-the-art methods in most scenarios. Based on baseline methods and with controlled parameter counts, the method of this invention surpasses most existing state-of-the-art methods on the three public datasets MARS, iLIDS-VID, and PRID. In particular, on the iLIDS-VID dataset, the method of this invention achieves 100% on the Rank-5 metric.

[0131] The video person re-identification method based on spatiotemporal affine subspaces described in this invention abandons the approach of compressing the entire video sequence into a single feature vector, and instead uses affine subspaces as the final feature representation of the video. An affine subspace consists of an origin representing the average static appearance and a set of orthonormal bases representing the main dynamic change patterns. This allows for a richer and more structured description of the various poses and viewpoint changes exhibited by pedestrians throughout the video than a single vector, making the model more robust to interference factors such as illumination and viewpoint changes, effectively reducing re-identification errors. Furthermore, this invention uses a principal angle metric to calculate the distance between two affine subspaces to evaluate the similarity between two video sequences. The principal angle metric can geometrically measure the similarity of the structures of two subspaces, ensuring theoretical consistency and efficiency throughout the entire process from feature representation to similarity comparison, further improving the accuracy and reliability of person re-identification. The visual encoder of this invention comprises multiple ViT layers and an adaptive spatiotemporal information coupling module. This adaptive spatiotemporal information coupling module is designed as a module with invariant input and output dimensions, thus it can be flexibly inserted into different ViT layers as a plug-and-play unit without disrupting the original architecture. During the mid-stage of feature extraction, the adaptive spatiotemporal information coupling module directly and synchronously aggregates spatiotemporal information in local regions through 3D convolution, and fuses it with the original features through residual connections. This enables the model to simultaneously capture spatial appearance features and short-term motion changes, achieving deep fusion and collaborative learning of spatiotemporal information. The resulting features contain richer dynamic discriminative information, thereby significantly improving the accuracy of person re-identification.

[0132] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0133] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0134] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0135] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0136] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A pedestrian re-identification method based on spatiotemporal affine subspace, characterized in that, include: Obtain the set of video sequences to be matched and the set of candidate video sequences, and calculate the affine subspace of all video sequences; Based on the principal angle metric, the distance between the video sequence to be matched and the affine subspace of each video sequence in the candidate video sequence set is calculated, including: Obtain the set of orthonormal bases for the affine subspace of the video sequence to be matched. ; Obtain the set of orthonormal bases for the affine subspace of any video sequence in the candidate video sequence set. ; Singular value decomposition of the orthonormal basis set of the affine subspaces of the two video sequences. , represented as: ; Obtain the diagonal matrix after singular value decomposition , with diagonal matrix diagonal elements, for The cosine value of each principal angle; based on Calculate the cosine values ​​of the principal angles. The sum of the squares of the principal angles is used to obtain the distance between the affine subspace of the video sequence to be matched and any video sequence in the candidate video sequence set, expressed as: ； in, Indicates the first The cosine value of each principal angle, , Indicates the total number of orthonormal bases; The video sequences in the candidate video sequence set are sorted in ascending order of distance to obtain the target video sequence set in descending order of similarity to the video sequence to be matched. The video sequence is input into the re-identification model, passing sequentially through a cascaded feature embedding layer, multiple visual encoders, and an affine subspace modeling module. The output affine subspace of the video sequence includes: The video sequence is input into the feature embedding layer. Block convolution is performed on each frame of the video sequence to obtain multiple block feature vectors for each frame. A class feature vector with the same dimension as the block feature vector is assigned to each frame, and this vector is concatenated with the multiple block feature vectors to form the embedding features for each frame. This includes: for the video sequence... Each frame of the image is divided into... Each image block is converted into a... A 3D vector; a category feature vector is added to each frame of the image to capture the overall features of the entire image, and... Image patches Concatenate 3D vectors to obtain each frame of the image. Embedded features of dimension; where, This indicates the number of image frames in a video sequence. Indicates the number of image channels. and These represent the height and width of the image, respectively. Indicates the dimension of a block in ViT; The embedding features of each frame image are input into the first visual encoder to extract and couple spatiotemporal information, and output the spatiotemporal coupling features of each frame image. This includes: inputting the embedding features of the image through multiple cascaded ViT layers to output the target visual features; inputting the target visual features into the adaptive spatiotemporal information coupling module to perform spatiotemporal convolution and output the spatiotemporal coupling features of the image. Using the output of the current visual encoder as the input of the next visual encoder, the spatiotemporal coupling features of each frame of the last visual encoder output are obtained and used as the target frame features of each frame. The target frame features of each frame image are input into the affine subspace modeling module. Temporal average pooling is performed on the target frame features of all frames to obtain the origin of the affine subspace. Singular value decomposition is performed on the matrix composed of the target frame features of all frames. The first preset number of column vectors in the left singular vector matrix after decomposition are used as the orthonormal basis. The set of the origin and the preset number of orthonormal bases is used as the affine subspace of the video sequence.

2. The pedestrian re-identification method based on spatiotemporal affine subspace according to claim 1, characterized in that, The embedded features of the image are input through multiple concatenated ViT layers, and the output target visual features include: The embedded features of the image are input into the first ViT layer, and the self-attention mechanism and feedforward network are used to capture global context dependencies and perform nonlinear transformations to output the visual features of the image; the visual features include a global image feature and multiple local feature vectors. The visual features output from the current ViT layer are used as the input to the next ViT layer, and the visual features output from the last ViT layer are obtained as the target visual features.

3. The pedestrian re-identification method based on spatiotemporal affine subspace according to claim 2, characterized in that, The target visual features are input into the adaptive spatiotemporal information coupling module, spatiotemporal convolution is performed, and the spatiotemporal coupling features of the image are output, including: Divide multiple local feature vectors in the target visual features into Each patch is reconstructed according to the original spatial layout of the image and arranged into... The vector matrix, Indicates the size of the patch; Divide the vector matrix into Each region, obtain Each initial region feature is then subjected to a 3D spatiotemporal convolution, and the resulting features are added to the initial region features to obtain the optimized region features for each region. By stitching together the optimized regional features of each region, the spatiotemporal coupling features of the image are obtained.

4. The pedestrian re-identification method based on spatiotemporal affine subspace according to claim 1, characterized in that, In the visual encoder, the input features are processed through multiple ViT layers and then compressed into channels. The compressed channels are then input to the adaptive spatiotemporal information coupling module of the visual encoder.

5. The pedestrian re-identification method based on spatiotemporal affine subspace according to claim 1, characterized in that, The target frame features of each frame are input into the affine subspace modeling module. Temporal average pooling is performed on the target frame features of all frames to obtain the origin of the affine subspace, represented as: ， ； in, Represents the origin of the affine subspace. This indicates the number of image frames in a video sequence. This represents the set of target frame features for each frame of an image in a video sequence. Indicates the first video sequence Target frame features of the frame image .

6. The pedestrian re-identification method based on spatiotemporal affine subspace according to claim 5, characterized in that, Singular value decomposition is performed on the matrix composed of target frame features from all frames of images. The first predetermined number of column vectors in the decomposed left singular vector matrix are used as an orthonormal basis, including: A matrix composed of the target frame features of all frames in the video sequence Singular value decomposition of the transpose , represented as: ; Indicates the feature The right singular value vector obtained by performing singular value decomposition; Obtain the left singular vector matrix after decomposition Center front column vectors for A standard orthogonal basis.

7. The pedestrian re-identification method based on spatiotemporal affine subspace according to claim 1, characterized in that, Preprocessing of the video sequence includes: For each frame of the video sequence, data augmentation is performed using random horizontal flipping and random erasure; Each frame of the augmented image is converted into tensor data, and the pixel values ​​are scaled to the range [0,1]. Then, the mean or variance of ImageNet is used for normalization to obtain the corresponding RGB image of each frame, forming a preprocessed video sequence.

Images