A spatio-temporal feature erasing video pedestrian re-identification method based on action excitation

Abstract

The application discloses a spatiotemporal feature erasing video pedestrian re-identification method based on action excitation. The method comprises the following steps: acquiring pedestrian video data containing multiple actions; constructing a deep residual network model based on the pedestrian video data; training the deep residual network model by using a deep residual network based on action excitation and spatiotemporal features, and performing video pedestrian re-identification by using the trained deep residual network model. The application aims at the problem that, in current video pedestrian re-identification research, attention is paid to apparent features and fine-grained features, i.e., static features in a single frame, and dynamic characteristics specific to videos are ignored, and action features composed of multiple frames are ignored. Starting from the extraction of action features, the application proposes a spatiotemporal feature erasing video pedestrian re-identification algorithm based on action excitation.

Description

Technical Field

[0001] This invention relates to the field of video pedestrian re-identification technology, and in particular to a method for video pedestrian re-identification based on spatiotemporal feature erasure using action-stimulated methods. Background Technology

[0002] Artificial intelligence (AI) has great potential in the field of public safety, and pedestrian re-identification is a typical application of AI in this area. Pedestrian re-identification is widely used in various fields such as criminal investigation, intelligent security, intelligent monitoring, smart cities, and unmanned supermarkets, providing valuable protection for people's lives and property. In early criminal investigations, to find the movements of criminal suspects from surveillance cameras, a large number of investigators needed to manually sift through large amounts of video from multiple cameras over long periods of time. This was undoubtedly time-consuming, labor-intensive, and inefficient, and could potentially delay the investigation and hinder progress. Pedestrian re-identification technology, however, can accurately retrieve corresponding criminal suspects from massive amounts of video data, providing powerful assistance in solving cases.

[0003] Video pedestrian re-identification faces inherently complex challenges. Pedestrians with the same identity often appear differently in photos taken from different angles, exhibiting variations in posture, scale, and clarity. Conversely, pedestrians with different identities, due to similar appearances and clothing, often appear extremely similar in images taken from the same camera, making them difficult to distinguish. Accurate video pedestrian re-identification is significantly challenging in these situations.

[0004] A common assumption in video person re-identification research is the use of multiple frames in a video sequence for mutual supplementation and reference. For example, if a pedestrian is occluded in one frame, they may not be occluded in other frames; if a pedestrian is blurred due to motion in one frame, they may be correctly focused in other frames. Therefore, video person re-identification largely focuses on fusing features from multiple frames, hoping to obtain discriminative pedestrian video features by retaining the best and discarding the worst. However, current research on video person re-identification largely borrows from the ideas of image person re-identification, focusing too much on appearance features and neglecting the pedestrian's action features. If different pedestrians have similar appearances and clothing, appearance features become ineffective in distinguishing them; in this case, action features are more discriminative than appearance features. Summary of the Invention

[0005] In view of this, the purpose of this invention is to propose a spatiotemporal feature erasure video pedestrian re-identification method based on action stimulus. This method introduces action information as an stimulus for appearance information in the video pedestrian re-identification method, which is more accurate than the original model that only focuses on appearance information.

[0006] According to one aspect of the present invention, a method for re-identifying pedestrians in video with spatiotemporal feature erasure based on action stimulus is provided, comprising: acquiring pedestrian video data containing multiple actions; constructing a deep residual network model based on the pedestrian video data; training the deep residual network model using a deep residual network based on action stimulus and spatiotemporal features; and using the trained deep residual network model to perform video pedestrian re-identification.

[0007] In the aforementioned technical solutions, addressing the problem in current video pedestrian re-identification research that focuses on appearance features and fine-grained features (i.e., static features within a single frame) while neglecting the unique dynamic characteristics of video—action features composed of multiple frames—the proposed solution starts with action feature extraction and proposes a spatiotemporal feature erasure video pedestrian re-identification method based on action stimulus. This method introduces action information as an stimulus for appearance information, resulting in higher accuracy than previous models that only focused on appearance information.

[0008] In some embodiments, training the deep residual network model using a deep residual network based on action stimulus and spatiotemporal features includes: the deep residual network extracting the appearance features, action stimulus features, spatial information, and temporal information of pedestrians from video data, and using the action stimulus features as stimulus information to extract pedestrian features from the appearance features; refining the pedestrian features using spatial information and temporal information to obtain pedestrian features containing spatiotemporal relationships; and inputting the pedestrian features containing spatiotemporal relationships into the deep residual network for training to obtain the deep residual network model.

[0009] In the above technical solution, in order to retain the role of appearance features, action features are only used as stimulus information, so that the pedestrian features extracted by the network take into account both appearance and action information. Furthermore, this invention utilizes spatial and temporal information to further refine the current feature representation.

[0010] In some embodiments, the deep residual network extracts the appearance features and motion stimulus features of pedestrians from video data; and the step of using motion stimulus features as stimulus information to extract pedestrian features from appearance features specifically includes: the deep residual network extracting appearance features; performing dimensionality reduction on the appearance features to obtain dimensionality-reduced features, and then convolving the dimensionality-reduced features to obtain convolutional dimensionality-reduced features; obtaining frame difference features through dimensionality reduction features, convolutional dimensionality-reduced features, feature frame difference method, and preprocessing; multiplying the frame difference features with the appearance features to obtain motion stimulus features; and adding the motion stimulus features and appearance features to obtain pedestrian features.

[0011] In the above technical solution, a simple feature frame difference method is used instead of the traditional method of calculating optical flow, avoiding a large amount of additional computation. Motion information is generated using this simple feature frame difference method and treated as an stimulus for appearance information.

[0012] In some embodiments, the deep residual network extracts spatial and temporal information of pedestrians from video data; and refines pedestrian features using spatial and temporal information to obtain pedestrian features containing spatiotemporal relationships. Specifically, this includes: performing a 50% dimensionality reduction operation on the pedestrian features, dividing the features into temporal and spatial parts, and extracting temporal and spatial features respectively; in the temporal part, performing matrix transformation on the input features, followed by 3D convolution, batch normalization, activation function activation, matrix transformation, and averaging to obtain temporal features; in the spatial part, performing 2D convolution on the input features, followed by batch normalization, activation function activation, and averaging to obtain spatial features; and concatenating the spatial and temporal features to obtain pedestrian features containing spatiotemporal relationships.

[0013] In the above technical solution, spatial and temporal information are extracted using 2D convolution and 3D convolution respectively, further refining the current feature representation. Features are further refined from both temporal and spatial dimensions to obtain more discriminative feature representations.

[0014] In some embodiments, the step of inputting pedestrian features containing spatiotemporal relationships into a deep residual network for training to obtain a deep residual network model includes: inputting pedestrian features containing spatiotemporal relationships into a deep residual network for training; updating the parameters of the deep residual network using a loss function of cross-entropy loss, a triplet loss function of hard sample mining, and a stochastic gradient descent algorithm; and using the parameter-updated deep residual network to train and update the deep residual network model through a finite number of iterations.

[0015] In the above technical solution, the overall network loss function consists of a cross-entropy loss function (xent) and a triplet loss function (htri) based on hard samples; parameter updates employ stochastic gradient descent. Specifically, the cross-entropy loss function (xent) is used to calculate the classification loss for pedestrian IDs, while the triplet loss function (htri) based on hard sample mining reduces intra-class variance while increasing inter-class variance. This combination effectively improves the accuracy of the proposed model.

[0016] In some embodiments, the step of inputting pedestrian features containing spatiotemporal relationships into a deep residual network for training to obtain a deep residual network model includes: randomly erasing the pedestrian features containing spatiotemporal relationships from the frames and then inputting them into the deep residual network for training; updating the parameters of the deep residual network using a loss function of cross-entropy loss, a triplet loss function of hard sample mining, and a stochastic gradient descent algorithm; and using the parameter-updated deep residual network to train and update the deep residual network model through a finite number of iterations.

[0017] In the aforementioned technical solutions, while video pedestrian re-identification datasets possess richer information due to their multi-frame nature, they also present challenges in selecting effective information and avoiding redundant information across multiple video frames. In video pedestrian re-identification tasks, image sequences provide largely consistent appearance information. However, most current video pedestrian re-identification algorithms often simply use average aggregation when aggregating features across multiple frames. This can lead to key features extracted from a particular video keyframe being smoothed out during the averaging process because they only appear once, rendering them less critical and consequently reducing the discriminative power of the final generated pedestrian feature representation. Therefore, to fully utilize these potential key features during model training, and given the lack of explicit supervision information in the dataset, making it impossible to know whether key features exist in a particular training video segment or which video frames they are present in, a random frame feature erasure scheme is proposed. This scheme involves randomly sampling video frames within the module and learning the generated features.

[0018] According to another aspect of the present invention, a deep residual network based on motion excitation and spatiotemporal features is proposed and applied to a video pedestrian re-identification method based on motion excitation and spatiotemporal feature erasure in the above-mentioned technical solution. The deep residual network includes a first block, a second block, a third block, a fourth block, a spatiotemporal feature module, and a random frame feature erasure module, which are sequentially connected by information transmission. The second block contains two motion excitation modules, and the third block contains three motion excitation modules.

[0019] In the above technical solution, to better utilize a video pedestrian re-identification method based on action-incentive-based spatiotemporal feature erasure, an optimized deep residual network based on action-incentive and spatiotemporal features is specifically proposed. In this network, the action-incentive module (ME) avoids the problem of excessive computational cost associated with traditional optical flow calculations by employing feature frame difference, and uses action features as the incentive for appearance features, thus considering both action and appearance information. The spatiotemporal feature module (STB) refines the features in both temporal and spatial dimensions, further enhancing the discriminative power of pedestrian features. The random frame feature erasure module (RFE) extracts key features from video segments through random frame sampling, mitigating feature redundancy to some extent. Furthermore, to further improve accuracy, the insertion positions of the action-incentive module within the block are designed, thereby improving the overall network performance.

[0020] According to another aspect of the present invention, a video pedestrian re-identification device based on action-stimulated spatiotemporal feature erasure is proposed, which is applied to the above-mentioned method for video pedestrian re-identification based on action-stimulated spatiotemporal feature erasure. The method includes: an acquisition module, a construction module, and a training and recognition module connected sequentially; the acquisition module is used to acquire pedestrian video data containing multiple actions; the construction module is used to construct a deep residual network model based on the pedestrian video data; the training and recognition module is used to train the deep residual network model using a deep residual network based on action-stimulated spatiotemporal features, and to perform video pedestrian re-identification using the trained deep residual network model.

[0021] In the above technical solution, in order to better utilize a video pedestrian re-identification method based on action-stimulated spatiotemporal features, an optimized deep residual network based on action-stimulated and spatiotemporal features is specifically proposed.

[0022] According to another aspect of the present invention, a motion-stimulated spatiotemporal feature erasure video pedestrian re-identification device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute a motion-stimulated spatiotemporal feature erasure video pedestrian re-identification method as described above.

[0023] In the above technical solution, in order to better utilize a spatiotemporal feature erasure video pedestrian re-identification method based on action stimulus, a spatiotemporal feature erasure video pedestrian re-identification device based on action stimulus is specifically proposed.

[0024] According to a final aspect of the present invention, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements a method for re-identifying pedestrians in a video with spatiotemporal feature erasure based on motion stimulation, as described above.

[0025] In the above technical solution, in order to better utilize a spatiotemporal feature erasure video pedestrian re-identification method based on action stimulus, a computer-readable storage medium is specifically proposed. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a structural framework diagram of a deep residual network based on action stimulus and spatiotemporal features in the video pedestrian re-identification method based on action stimulus and spatiotemporal features of the present invention;

[0028] Figure 2 This is a schematic diagram of the structural framework of the motion excitation module;

[0029] Figure 3 This is a schematic diagram of the structural framework of the spatiotemporal feature module;

[0030] Figure 4 This is a schematic diagram of the structural framework of the random frame feature erasure module;

[0031] Figure 5 This is a schematic diagram showing the results of an embodiment of the spatiotemporal feature erasure video pedestrian re-identification method based on motion stimulation of the present invention. Detailed Implementation

[0032] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] This invention provides a spatiotemporal feature erasure video person re-identification method based on action stimulus. This method introduces action information as a stimulus for appearance information, resulting in higher accuracy than previous models that only focused on appearance information. The invention will be described in detail below with reference to the accompanying drawings and specific embodiments:

[0034] Acquire pedestrian video data containing multiple actions;

[0035] Construct a deep residual network model based on pedestrian video data;

[0036] In this embodiment, the deep residual network models include: ResNet-18, ResNet-34, ResNet-50, ResNet-101, and ResNet-152. ResNet-50 is selected as an example in this case.

[0037] The deep residual network model is trained using a deep residual network based on action stimulus and spatiotemporal features, as detailed below:

[0038] The deep residual network extracts the appearance features, motion stimulus features, spatial information, and temporal information of pedestrians from the video data, and uses the motion stimulus features as stimulus information to extract pedestrian features from the appearance features. In this embodiment, in addition to appearance features, motion features can also be used. This case uses appearance features as an example.

[0039] The overall framework of the spatiotemporal feature-based video pedestrian re-identification method based on action-stimulated excitation, as follows: Figure 1 As shown. The algorithm uses ResNet50 as the backbone network.

[0040] Since the ResNet50 backbone network is divided into four blocks, each with a different number of modules, the number of modules in the four blocks is (3, 4, 6, 3). Therefore, the insertion position of the key module, the action excitation module (ME, the same below in the figure), is specially designed and inserted after the modules at (3, 4, 6, 3). Figure 1 As shown, the number of insertions for the action activation module corresponding to the number of modules in each block (3, 4, 6, 3) is (0, 2, 3, 0). Besides the key module, the action activation module, the spatiotemporal feature module (STB, hereinafter the same) and the random frame feature erasure module (RFE, hereinafter the same) are sequentially inserted after Block 4 to process the features activated by the action activation module. The overall network loss function is the cross-entropy loss function xent and the triplet loss function htri based on hard samples.

[0041] Motion stimulation module such as Figure 2 As shown, this module uses a simple feature frame difference method to generate motion information and processes the motion information as an excitation of appearance information. Specifically, the video features input to the module are represented as F. in = {t1, t2, t3, t4, t5, t6, t7, t8}, where F in Represented as 8 frames t i The set of features has feature dimensions (t, c, h, w), where t is the length of the sampled video segment, c represents the number of feature channels, and h and w represent the height and width of the feature map, respectively. Each frame contains image features t. i The dimensions are (c, h, w).

[0042] The algorithm first considers the input features F in Dimensional reduction to obtain F down Features, where Conv represents a 2D convolution operation. As shown in Equation 1:

[0043] F down =Conv(F in )#(1)

[0044] Meanwhile, regarding the input feature F in Perform Conv convolution dimensionality reduction to obtain F down After obtaining the features, the frame difference feature F is then obtained. sub Before proceeding with F down Conv convolution of features to obtain F downConv feature.

[0045] F downConv =Conv(F down )#(2)

[0046] Next, the frame difference feature F is obtained by subtracting the previous frame from the next frame. sub , where the subscripts i and i+1 represent the features of the previous and next frames, respectively.

[0047]

[0048] Then, the last frame is padded with zeros using the Pad operation to restore the t dimension of the feature map to its original length. The h and w dimensions are then averaged using the Avg operation, increased in dimension using the Conv operation, and activated by the Sigmoid activation function to obtain the final frame difference feature F. sub .

[0049] F sub =Sigmoid(Conv(Avg(Pad(F)) sub ))))#(4)

[0050] Then F sub Frame difference features and input features F in Multiplication yields the action stimulus feature F me Finally, F me Features and F representing apparent features in The summation serves as the final module output feature F. out .

[0051] F me =F in *F sub F out =F in +F me #(5)

[0052] Pedestrian features are refined using spatial and temporal information to obtain pedestrian features that include spatiotemporal relationships, as follows:

[0053] Building upon the action activation module, we aim to further refine the features from both temporal and spatial dimensions to obtain more discriminative feature representations. To this end, we propose a simple and universal spatiotemporal feature module. This module performs 3D convolution and 2D convolution on the total features after activation by the action activation module, focusing on temporal relationships and spatial features respectively.

[0054] Specifically, such as Figure 3 As shown, for the input feature F in Its dimensions are (t, c, h, w), where t, c, h, and w represent the number of frames, the number of channels, the height of the feature map, and the width of the feature map, respectively. The output part's feature F... out This is the concatenation result of the Temporal and Spatial parts of the module, with dimensions (t, c), where t represents the number of frames and c equals the input feature F. in The number of channels.

[0055] In the spatiotemporal feature module, the input features are first reduced in dimensionality by half, dividing them into Temporal and Spatial parts, and then the temporal and spatial features are extracted respectively.

[0056] F down =DownChannel(F in )#(6)

[0057] In the Temporal part, in order to perform 3D convolution, a reshape operation (matrix transformation, the same below) is required first. After 3D convolution, batch normalization (BN) and ReLU activation are performed. After reshaping to the original input dimension, average averaging is performed on the h and w dimensions.

[0058] F temporal =avg(ReLU(BN(3dconv(F down ))))#(7)

[0059] In the Spatial part, no reshape operation is needed. After performing 2dconv, batch normalization (BN) and ReLU activation are performed. Finally, average is performed on both h and w dimensions.

[0060] F spatial =avg(ReLU(BN(2dconv(F down ))))#(8)

[0061] The final feature representation is the feature F of the Temporal part. temporal and the characteristics F of the Spatial part spatial Perform cat concatenation to obtain the final output feature F out .

[0062] F out =cat(F temporal F spatial )#(9)

[0063] The DownChannel operation reduces the input dimension by half, ensuring that the final concatenated output feature dimension is the same as the input feature dimension. The reshape operation adjusts the feature dimensions required for 3D convolution across (t, h, w) dimensions. The cat operation concatenates two features of the same dimension along the channel dimension.

[0064] Pedestrian features containing spatiotemporal relationships are input into a deep residual network for training to obtain a deep residual network model, as detailed below:

[0065] Video person re-identification datasets, due to their multi-frame nature, offer richer information, but also present challenges in selecting effective information and avoiding redundant information across multiple video frames. In video person re-identification tasks, image sequences provide largely consistent appearance information. However, most current video person re-identification algorithms simply use averaging aggregation when aggregating features across multiple frames. This can lead to key features extracted from a particular keyframe of a video being smoothed out during the averaging process because they only appear once, rendering them less critical and reducing the discriminative power of the final generated person feature representation. To fully utilize these potential key features during model training, and given the lack of explicit supervision in the dataset—meaning it's impossible to know whether key features exist in a particular training video segment or which video frames they appear in—a cleverly designed random frame feature erasure module is implemented. This module randomly extracts video frames and learns the generated features.

[0066] The range of random frame extraction is the total length N of the video segment; specifically, it involves randomly extracting one, two, three, or even all N frames of the video segment. The random frame feature erasure module is as follows: Figure 4 As shown.

[0067] For i in Formula 10, the range is from 1 to N, where N is the total number of frames in the sampled video segment:

[0068] f i =Tavg(Rand i (F in ))#(10)

[0069] Tavg represents the average operation performed on the extracted features along the T dimension, resulting in a vector feature representation with c channels. Rand i Operators indicate from F in In the feature, i frames are randomly selected along the dimension of frame number t.

[0070] For input features F in via Rand i The operation involves randomly extracting frames, the number of extractions being N, the length of the video segment, with the number of frames ranging from 1 to the total length N of the video segment. The acquired features are then averaged over T dimensions to obtain {f1, f2, f3, ..., f...} N During training, all samples are fed into a supervised array of cross-entropy loss xent and triplet loss htri based on hard sample mining. During testing, features f are used. N As the final representation of pedestrian characteristics.

[0071] The trained deep residual network model is used for video pedestrian re-identification, as follows:

[0072] Visualization results of experiments on a motion-stimulated spatiotemporal erasure video pedestrian re-identification algorithm, such as... Figure 5 As shown in the figure. Rows 1 and 3 are the results of the baseline algorithm, and rows 2 and 4 are the results of this invention. In the same row, the left side is a 4-frame sample of the Query video clip, and the right side is the top 5 results with the closest similarity, denoted as Rank-1 to Rank-5. Rank-1 and Rank-5 in rows 1 and 2, and Rank-5 in rows 3 and 4 indicate incorrect search results, while the remaining images indicate correct search results.

[0073] exist Figure 5 In the baseline algorithm, although the retrieval results are mostly correct except for Rank-1, Rank-1 is the most important metric for the retrieval system, and a Rank-1 error is fatal. The algorithm proposed in this chapter not only corrects the Rank-1 error but also retrieves the correct result Rank-3, which the baseline algorithm failed to find. Figure 5 In the comparison charts in the last two rows, the baseline algorithm only retrieved one correct Rank-2, while the algorithm proposed in this chapter not only correctly retrieved Rank-1, but also correctly returned the retrieval results for Rank-1 to Rank-5. This fully verifies the effectiveness of the action-stimulated spatiotemporal feature erasure video pedestrian re-identification algorithm.

[0074] This invention addresses the problem in current video person re-identification research that focuses on appearance and fine-grained features—that is, static features within a single frame—while neglecting the dynamic characteristics of videos, including action features composed of multiple frames. Starting with action feature extraction, this invention proposes a spatiotemporal feature erasure algorithm for video person re-identification based on action stimulus. This invention uses a simple feature frame difference method instead of the traditional optical flow calculation method, avoiding a large amount of additional computation. Simultaneously, to preserve the role of appearance features, action features are treated only as stimulus information, thus ensuring that the pedestrian features extracted by the network simultaneously consider both appearance and action information. Furthermore, this invention utilizes 2D convolution and 3D convolution to extract spatial and temporal information, further refining the current feature representation. Finally, this invention uses random frame sampling to reduce redundancy in video frame information while simultaneously learning features from key frames within the video segment.

[0075] This invention also provides a deep residual network based on action stimulus and spatiotemporal features, which is applied to the above-mentioned method for re-identifying pedestrians in video with spatiotemporal feature erasure based on action stimulus. The specific structure and advantages have been described in detail in the above-mentioned method embodiments, and will not be repeated here.

[0076] This invention also provides a motion-stimulated spatiotemporal feature erasure video pedestrian re-identification device, applied to the aforementioned motion-stimulated spatiotemporal feature erasure video pedestrian re-identification method. It includes: an acquisition module, a construction module, and a training and recognition module connected sequentially; the acquisition module is used to acquire pedestrian video data containing multiple actions; the construction module is used to construct a deep residual network model based on the pedestrian video data; the training and recognition module is used to train the deep residual network model using a motion-stimulated and spatiotemporal feature-based deep residual network, and to perform video pedestrian re-identification using the trained deep residual network model.

[0077] The present invention also provides a motion-stimulated spatiotemporal feature erasure video pedestrian re-identification device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described motion-stimulated spatiotemporal feature erasure video pedestrian re-identification method.

[0078] The present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the above-described method embodiments.

[0079] The above approach reveals that this invention addresses the problem in current video pedestrian re-identification research that focuses on appearance features and fine-grained features—that is, static features within a single frame—while neglecting the dynamic characteristics of videos, including action features composed of multiple frames. Starting with action feature extraction, this invention proposes a spatiotemporal feature erasure video pedestrian re-identification algorithm based on action stimulus. This invention uses a simple feature frame difference method instead of the traditional optical flow calculation method, avoiding a large amount of additional computation. Simultaneously, to preserve the role of appearance features, action features are used only as stimulus information, thus ensuring that the pedestrian features extracted by the network simultaneously consider both appearance and action information. Furthermore, this invention utilizes 2D convolution and 3D convolution to extract spatial and temporal information, further refining the current feature representation. Finally, this invention uses random frame sampling to reduce redundancy in video frame information while performing feature learning on keyframes within the video segment. Results show that this invention introduces action information as a stimulus for appearance information in the video pedestrian re-identification method, resulting in higher accuracy than previous models that only focus on appearance information. Extensive experimental results demonstrate that the method of this invention achieves excellent performance in video pedestrian re-identification.

[0080] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for pedestrian re-identification in videos with spatiotemporal feature erasure based on action-stimulated excitation, characterized in that, include: Acquire pedestrian video data containing multiple actions; A deep residual network model based on pedestrian video data is constructed. The deep residual network model includes at least a first block, a second block, a third block, a fourth block, a spatiotemporal feature module, and a random frame feature erasure module, wherein the second block contains at least two action excitation modules and the third block contains at least three action excitation modules. A deep residual network model based on action stimulus and spatiotemporal features is used to train the deep residual network model, and the trained deep residual network model is used for video pedestrian re-identification. The training process includes: a deep residual network model extracts the appearance features, action stimulus features, spatial information, and temporal information of pedestrians from video data, and uses the action stimulus features as stimulus information to extract pedestrian features from the appearance features; the pedestrian features are subjected to dimensionality reduction operation, and the features are divided into temporal and spatial parts, and the temporal features and spatial features are extracted by convolution operation and then concatenated to obtain pedestrian features containing spatiotemporal relationships; Pedestrian features containing spatiotemporal relationships are randomly framed and erased. The number of random frames extracted is equal to the total number of frames in the video segment. One frame to all frames are randomly extracted, and the features of the extracted frames are averaged over the time dimension and represented as vector features. The vector features are then input into a deep residual network model, which is trained using the cross-entropy loss function, the triplet loss function for hard sample mining, and the stochastic gradient descent algorithm.

2. The method for re-identifying pedestrians in spatiotemporally erased videos based on action-stimulated spatiotemporal features as described in claim 1, characterized in that, The process involves training a deep residual network model based on action stimulus and spatiotemporal features, including: The deep residual network model extracts the appearance features, action stimulus features, spatial information, and temporal information of pedestrians from video data, and uses the action stimulus features as stimulus information to extract pedestrian features from the appearance features. Pedestrian features are refined using spatial and temporal information to obtain pedestrian features that include spatiotemporal relationships; Pedestrian features containing spatiotemporal relationships are input into a deep residual network model for training, resulting in a trained deep residual network model.

3. The method for re-identifying pedestrians in spatiotemporally erased videos based on action-stimulated spatiotemporal features as described in claim 2, characterized in that, The deep residual network model extracts the appearance features and motion stimulus features of pedestrians from video data; and the step of using motion stimulus features as stimulus information to extract pedestrian features from appearance features specifically includes: Deep residual network models extract apparent features; Dimensionality reduction is performed on the apparent features to obtain dimensionality-reduced features, and then convolution is performed on the dimensionality-reduced features to obtain convolutional dimensionality-reduced features; Frame difference features are obtained through dimensionality reduction features, convolutional dimensionality reduction features, feature frame difference method, and preprocessing. Multiply the frame difference features by the appearance features to obtain the action stimulus features; add the action stimulus features and the appearance features to obtain the pedestrian features.

4. The method for re-identifying pedestrians in spatiotemporally erased videos based on action-stimulated methods as described in claim 2, characterized in that, The deep residual network model extracts spatial and temporal information of pedestrians from the video data, respectively. Furthermore, spatial and temporal information is used to refine pedestrian features, obtaining pedestrian features that include spatiotemporal relationships, specifically including: The pedestrian features are reduced in dimensionality by half, and the features are divided into temporal and spatial parts, and temporal features and spatial features are extracted separately. In the temporal part, the input features are subjected to matrix transformation, followed by 3D convolution, batch normalization, activation function activation, matrix transformation, and averaging to obtain temporal features. In the spatial part, spatial features are obtained by performing 2D convolution, batch normalization, activation function activation, and averaging on the input features. Spatial and temporal features are combined to obtain pedestrian features that include spatiotemporal relationships.

5. The method for re-identifying pedestrians in spatiotemporally erased videos based on action-stimulated methods as described in claim 2, characterized in that, The step of inputting pedestrian features containing spatiotemporal relationships into a deep residual network model for training, and obtaining the trained deep residual network model, includes: Pedestrian features containing spatiotemporal relationships are input into a deep residual network model for training. The parameters of the deep residual network model are updated using a loss function of cross-entropy loss, a triplet loss function for hard sample mining, and a stochastic gradient descent algorithm. The deep residual network model is then trained and updated through a finite number of iterations using the parameter-updated deep residual network model.

6. The method for re-identifying pedestrians in spatiotemporally erased videos based on action-stimulated methods as described in claim 2, characterized in that, The step of inputting pedestrian features containing spatiotemporal relationships into a deep residual network model for training, and obtaining the trained deep residual network model, includes: After random frame feature erasure of pedestrian features containing spatiotemporal relationships, the features are input into a deep residual network model for training. The parameters of the deep residual network model are updated using a loss function of cross-entropy loss, a triplet loss function of hard sample mining, and a stochastic gradient descent algorithm. The deep residual network model is then trained and updated through a finite number of iterations using the parameter-updated deep residual network model.

7. A motion-stimulated spatiotemporal feature erasure video pedestrian re-identification device, applied to the motion-stimulated spatiotemporal feature erasure video pedestrian re-identification method according to any one of claims 1-6, characterized in that, include: The acquisition module, construction module, training and recognition module are connected sequentially. The acquisition module is used to acquire pedestrian video data containing multiple actions; The construction module is used to construct a deep residual network model based on pedestrian video data; The training and recognition module is used to train a deep residual network model based on action stimulus and spatiotemporal features, and to use the trained deep residual network model to perform video pedestrian re-identification.

8. A pedestrian re-identification device based on spatiotemporal feature erasure video using motion-stimulated methods, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform a motion-stimulated spatiotemporal feature erasure video pedestrian re-identification method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for re-identifying pedestrians in video with spatiotemporal feature erasure based on motion stimulation, as described in any one of claims 1 to 6.

Images