A residual expansion fusion method for pedestrian re-identification

Abstract

The application discloses a residual expansion fusion method for pedestrian re-identification. It includes a data preprocessing module, a residual expansion fusion module, a similarity calculation and a result output module. The main network is constructed using ResNet50, and the model pre-trained on ImageNet is used to initialize the main network. The proposed residual expansion fusion module generates a copy through weight sharing by using residual units of different sizes. At the same time, the copy is bidirectionally expanded and fused, effectively reducing the loss of important features of the original layer during feature fusion. The method effectively improves the robustness of the model in various scenes, and maximally slows down the loss of pedestrian representation information in the feature transmission process, so that the pedestrians can be more accurately identified in difficult scenes in the real world.

Description

Technical Field

[0001] This invention relates to the fields of computer vision technology and image retrieval, and more specifically, to a residual augmentation fusion method for pedestrian re-identification. Background Technology

[0002] Pedestrian re-identification is a technology that identifies target pedestrians within video sequences from existing, non-overlapping camera viewpoints. It has wide applications in numerous fields, including public safety and smart city development. In this context, pedestrian re-identification serves as a key means of urban security monitoring, assisting city managers and security personnel in tracking suspicious individuals, thereby more effectively safeguarding social order and the safety of people's lives and property.

[0003] Previous pedestrian re-identification technologies mainly relied on target detection and tracking algorithms. However, these technologies often performed poorly when faced with occlusion and complex scenes, which to some extent promoted the emergence of cross-technology combining deep learning and pedestrian re-identification.

[0004] Pedestrian re-identification technology has profound implications not only for urban management and security but also for monitoring pedestrian behavior. By analyzing pedestrian behavior, abnormal behaviors, such as loitering or tailgating, can be detected promptly. Pedestrian re-identification technology can effectively prevent crime and safeguard public safety. Therefore, this technology has attracted considerable attention in the fields of computer vision and artificial intelligence research, drawing in-depth studies from numerous scholars.

[0005] With researchers successfully applying deep learning to the field of person re-identification, this field has developed rapidly. However, in real-world situations, pedestrians face challenges such as cluttered backgrounds, partial occlusion, varied poses, changing lighting, and low resolution, which greatly obscure the features of the same person. Especially when occluded, different body parts are often obscured and become invisible, significantly increasing the challenge of person re-identification.

[0006] Current common methods utilize multi-scale feature fusion strategies to extract global and local features. For example, Zhang et al. proposed a lightweight feature pyramid branch, which extracts features from different network layers and aggregates them into a bidirectional pyramid structure. However, most current methods do not consider that features extracted from lower-level networks pass through too many network layers, resulting in the loss of original information features and reducing the diversity of pedestrian features. Summary of the Invention

[0007] Purpose of the Invention: As mentioned earlier, in the process of pedestrian feature extraction, multi-layer pedestrian features are generally fused to obtain more representational information. However, during the transmission of low-level pedestrian features to high-level features, pedestrian representation information is lost, thus affecting the model's recognition performance. To alleviate this problem, this invention proposes a residual augmentation fusion method for pedestrian re-identification. The backbone network is constructed using a modified ResNet-50 to extract effective low- and high-level features. Then, the residual fusion module uses a weight-sharing mechanism to generate copies of the input feature vector for each network layer using two different sized residual units. These copies are then augmented and fused vertically and horizontally to effectively preserve the key feature information of the original layers and minimize the loss of important features during transmission.

[0008] The objective of this invention can be achieved through the following technical solutions:

[0009] A residual augmentation fusion method for pedestrian re-identification includes the following steps:

[0010] (1) The Market-1501 and CUHK03 pedestrian datasets were selected as experimental datasets, and the datasets were divided into three parts: training set, test set and query set. Data preprocessing was performed using techniques such as data normalization, random horizontal flipping, random flipping, random erasure, and random rotation.

[0011] (2) The convolutional neural network used is ResNet-50 as the backbone network. Its structure includes four stages. Each stage uses skip connections to alleviate the gradient vanishing problem, which allows for the training of deeper networks. This design enables the network to efficiently extract multi-scale feature information while maintaining high computational efficiency, providing rich feature representations for subsequent tasks.

[0012] (3) The batch of sampled pedestrians is preprocessed and input into a predefined model to extract high-order features of pedestrians; then, the low-level features extracted by the backbone network and the high-level features are input into the residual expansion and fusion module to reduce feature loss through expansion, so that the high and low-level features are fused to the same size, and input into the loss function to calculate the loss and perform backpropagation, update the model parameters, and iteratively minimize the value of the loss function to form an optimized pedestrian re-identification model;

[0013] (4) After training, the proposed network model was used as a pedestrian feature extractor. Each pedestrian was represented using pedestrian features containing identification information. The pedestrians in the dataset were sorted according to their similarity to the target pedestrian by calculating the Euclidean distance between the target pedestrian and the feature vectors of each pedestrian in the test dataset. Finally, the pedestrian images most similar to the target pedestrian were selected, and the recognition accuracy of the model was calculated.

[0014] The Market-1501 dataset selected in step (1) is specifically divided as follows:

[0015]

[0016] The dataset CUHK03 selected in step (1) is specifically divided as follows:

[0017]

[0018] In the data preprocessing of step (1):

[0019] (1-1) The first step in data preprocessing is to resize the input image;

[0020] (1-2) Perform data normalization, which scales the pixel values ​​of the image to between 0 and 1, or converts the pixel values ​​to a standard normal distribution with a mean of 0 and a variance of 1. Normalization can eliminate the order of magnitude difference between input features, avoid the adverse effects of data size on model parameter optimization, and improve the convergence speed and stability of the model.

[0021] (1-3) Perform a random horizontal flip operation, flipping the image left or right along the horizontal axis with a certain probability. This operation simulates the symmetrical feature changes in real scenes, which helps to enhance the diversity of data and improve the model's ability to recognize targets in different directions. The flip probability can be flexibly set according to task requirements;

[0022] (1-4) Perform random erasure. Erase a rectangular area at a random location in the image and fill the area with a random value. This operation forces the model to focus on features of other parts of the image during training, rather than relying on certain fixed patterns, which helps improve the model's robustness to occlusion, noise, etc.

[0023] (1-5) Perform random rotation. Rotate the input image randomly within a certain angle range, for example, between -30 degrees and 30 degrees to generate a new rotated image. Random rotation can simulate image orientation changes in real-world scenes, improving the model's adaptability to targets at different angles. The rotated image needs edge padding or cropping to match the target size;

[0024] (1-6) Perform random flipping operations. Flip the image horizontally and vertically with certain probabilities. This operation can further increase the diversity of the data and simulate the distribution of targets in different spatial directions in reality. The choice of flipping probabilities needs to be adjusted according to the data augmentation requirements of the task;

[0025] (1-7) Perform random brightness adjustment. Randomly increase or decrease the brightness value of the input image to simulate the shooting effect under different lighting conditions. Brightness adjustment can enhance the robustness of the model to changes in lighting and avoid overfitting the model to specific lighting conditions during training. The adjusted brightness range should be controlled within a reasonable range to maintain the image's recognizability.

[0026] The construction and initialization of the backbone network and branch network in step (2) are as follows:

[0027] (2-1) ResNet consists of multiple convolutional layers, including multiple residual blocks. Each residual block includes a skip connection, which directly adds the input to the output, thus preserving the original feature information. This design not only makes the network easier to optimize but also alleviates the problems of vanishing and exploding gradients, improving the network's accuracy. This method uses ResNet-50 and sets the stride of the last convolutional layer to 1 to suit the needs of pedestrian re-identification tasks. The branch network is constructed using the basic units of ResNet, each of which consists of a convolutional filter, a batch normalization layer (BN), and a linear rectifier (ReLU).

[0028] (2-2) The pre-trained model for initializing the backbone network comes from the ImageNet dataset. ResNe50 is pre-trained on ImageNet and can then be trained on other datasets for specific tasks through fine-tuning or transfer learning.

[0029] In step (3):

[0030] (3-1) After the backbone network extracts features, the low-level features and high-level features are input into the residual expansion and fusion module to reduce feature loss by expansion, so that the high- and low-level features are fused to the same size.

[0031] (3-2) The triplet loss function is used to optimize the global features, and the hard sample sampling triplet loss function and the center loss function are used to optimize the combined features. The triplet loss is shown in the following equation:

[0032]

[0033] Where, N tp This indicates the number of available triples, where δ is a hyperparameter used to represent the interval. + Represents the hinge loss function;

[0034] Center Loss learns a center vector for each class, such that all sample points within the same class are as close to this center as possible, while the centers of different classes are as far apart as possible.

[0035]

[0036] Of particular note is c i It is category y i The center of all sample features;

[0037] (3-3) In summary, the total loss of the residual augmentation fusion branch network used for pedestrian re-identification is as follows:

[0038] L = AL t +BL c

[0039] Where A and B are the weight coefficients of each loss function.

[0040] The beneficial effects of this invention are as follows: This invention constructs a residual augmentation fusion method for pedestrian re-identification. For each network layer, residual units are used to generate copies of their respective input feature vectors. The copies generated by lower-level network layers are first augmented vertically and horizontally, and then fused with the copies generated by adjacent higher-level network layers at the same size. This reduces the loss of important features from the original layers during feature fusion, enabling more accurate pedestrian matching and identification. Attached Figure Description

[0041] Figure 1 This is a flowchart of the pedestrian recognition process of the present invention;

[0042] Figure 2 This is the overall network model diagram of the present invention;

[0043] Figure 3 This is a residual expansion fusion structure diagram of the present invention; Detailed Implementation

[0044] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0045] The residual augmentation fusion method for pedestrian re-identification described in this invention has the following feature extraction and recognition process: Figure 1 As shown, the detailed structure of residual augmentation fusion for pedestrian re-identification is as follows: Figure 2 As shown, it includes the following steps:

[0046] (1) Dataset Selection: The CUHK 03 dataset selected for the experiment from the pedestrian re-identification dataset is divided into two subsets: CUHK 03-Labeled and CUHK 03-Detected. The CUHK 03-Labeled dataset contains a training set of 767 pedestrians and 7368 images, and a test set of another 700 pedestrians, 1400 query images, and 5328 library images. The CUHK 03-Detected dataset contains a training set of 767 pedestrians and 7365 images, and a test set of another 700 pedestrians, including 1400 query images and 5332 library images. Market-1501 is a pedestrian re-identification dataset and is one of the largest and most realistic pedestrian re-identification datasets available on mobile phones within the Tsinghua University campus. The dataset contains 32,668 images depicting 1,501 pedestrians and is divided into a training set and a test set, consisting of 12,936 and 19,732 images depicting 751 and 750 people, respectively.

[0047] (2) Training data preprocessing: In this invention, the training data is scaled to 288x288, and then the images are normalized, randomly horizontally flipped, randomly erased, randomly rotated, randomly flipped, and randomly brightened. Finally, the mean and standard deviation of ImageNet are used for normalization.

[0048] (3) Model loading: The backbone network is initialized using a ResNet-50 pre-trained model trained from the ImageNet dataset.

[0049] (4) Constructing a residual augmentation and fusion module:

[0050] Specifically, firstly, a residual module is designed in the lateral connection direction. This module consists of a CBR3x3 unit, a CBR1x1 unit, and residual operations. The CBR unit is equivalent to a network layer, composed of a convolutional filter, a batch normalization (BN) layer, and a ReLU linear rectifier. The feature vector Xi input to each layer is processed by this module to generate a copy Yi that retains the discriminative features learned by that network layer. The weights in the two CBR units are shared, which helps the copy capture important regions in the image, as shown in the following equation.

[0051] CBR = ReLU(BN(C i (X i )))

[0052] Y i =RS(CBR) 3x3 (X i ),CBR 1x1 (Xi ))

[0053] Then, in the upsampling direction, the feature map characteristics extracted by ResNet50 are utilized: the resolution of the low-level feature map is four times that of the neighboring high-level feature map. The high-level feature map is bidirectionally augmented, so that the high-level features and the neighboring low-level features are fused from top to bottom. The augmentation method is shown in the following formula.

[0054] BE=(1-α)(1-β)I(x1,y1)+α(1-β)I(x2,y1)+(1-α)βI(x1,y2)+αβI(x2,y2)

[0055]

[0056] Where I(x,y) is the target pixel point to be expanded, α and β are the ratios of the horizontal and vertical distances of position I(x,y) to the four discrete points, and (x1,y1), (x1,y2) and (x2,y1), (x2,y2) represent the four discrete points in the horizontal and vertical directions.

[0057] Finally, the high-level feature maps are fused with the neighboring low-level feature maps at the same size. The fusion method is shown in the following equation.

[0058]

[0059] in The fusion operation is represented by g(·), and global average pooling is represented by g(·). and These are the learnable parameters in the two fully connected layers, followed by the sigmoid activation function σ(·). The BEF module sequentially fuses the feature maps from the first four layers to obtain the fused feature F. f .

[0060] (5) Construction of the loss function.

[0061] (5-1) This invention selects Triplet Loss and Center Loss to train the model.

[0062] Triplet Loss: Assume I, I p I n There are three samples, sample I, I p For the same pedestrian identity, sample I, I n For different group identities. Triple loss is used to find a new feature space: from sample I to I p The distance is much smaller than that between sample I and I n The distance is shown in the following formula:

[0063]

[0064] Where, N tp This indicates the number of available triples, where δ is a hyperparameter used to represent the interval. + This represents the hinge loss function.

[0065] Center Loss: Learn a center vector for each class, such that all sample points within the same class are as close to this center as possible, while the centers of different classes are as far apart as possible. It is typically expressed as follows:

[0066]

[0067] Of particular note is c i It is category y i The center of all sample features.

[0068] (5-2) Training the model

[0069] Calculate the total loss and perform backpropagation to minimize this loss function, i.e., obtain the model sample with the smallest distance to positive samples and the largest distance to negative samples, thus obtaining the optimized person re-identification model. The trained model is obtained after 120 iterations in the following environment.

[0070]

[0071] (6) Test Model

[0072] (6-1) After enlarging the images in the test set from the original size of 64×128 to 128×384, the images were normalized to ensure the consistency of data distribution and the stability of the model. Then, the trained person re-identification model was loaded into a predefined network architecture, and by fusing global and local features, discriminative person feature vectors were extracted. Finally, the model performance was evaluated using currently accepted objective evaluation metrics—mean accuracy (mAP) and rank-1 accuracy—to measure the model's effectiveness in the person re-identification task.

[0073] (6-2) The Cumulative Match Curve (CMC) is a commonly used metric in retrieval tasks, primarily used to calculate the hit probability of Rank-k. Specifically, for a probe image to be retrieved in the query set of the test set, it returns a series of sorted results from the gallery set of the search library in the test set, ordered by similarity. The Rank-k accuracy A can be expressed as:

[0074]

[0075] Where p is probe, q is query, and N q It represents the number of images, f. CMC As shown in the following formula:

[0076]

[0077] In practical applications, representative Rank-1, Rank-5, and Rank-10 accuracies are usually selected to replace the CMC curve, with Rank-1 accuracy being the most important indicator. The CMC curve is plotted with k on the x-axis and Rank-k accuracy A on the y-axis, while Rank-1 accuracy directly reflects the probability that the target object is the first result in the search results, making it a key indicator for measuring the performance of the retrieval system.

[0078] (6-3) Mean Average Precision (mAP) is another important evaluation metric. The CMC curve usually only focuses on the ranking of the top positive samples in the search database, while mAP is determined by the ranking of all positive samples in the gallery. Therefore, it can usually reflect the performance of the model more robustly. Calculating AP requires the following three steps:

[0079] (6-3-1) Precision: For a specific probe image q in the query... j The query returned a series of sorted results for the gallery. Considering the first n query results, let's assume that the number of pedestrian IDs in the first n results that are the same as the probe image is c(n,q). j ):

[0080]

[0081] (6-3-2) Average Precision: For the query probe image q j Record the set {i1,i2,....,i} of the sorting results for all M positive samples in the sorting results. m}, calculate their average precision as shown in the following formula:

[0082]

[0083] (6-3-3) Mean Average Precision (mAP): For all N... q The mean Average Precision of the probe images, i.e.:

[0084]

[0085] The mAP (middle accuracy) metric is a key parameter for measuring the performance of a retrieval system; a higher value indicates better system efficiency. The mAP metric reaches its maximum value of 1 when the retrieval system performs optimally—that is, it accurately retrieves all positive samples and these samples rank first in the sorting. As the number of samples in the gallery image database increases, the mAP metric better reflects the stability and reliability of the retrieval system when dealing with complex data, i.e., its robustness. However, if the gallery image database has a small sample size, the reference value of the mAP metric is significantly reduced. Therefore, to more comprehensively evaluate the accuracy of pedestrian re-identification models, both the CMC (Constant Mid-Range) curve and the mAP metric are generally considered.

[0086] Finally, the network model was optimized using the above method, and the trained model was tested using the aforementioned evaluation metrics. The experiments were conducted on the Market-1501 dataset. The experiments demonstrate that a residual augmentation fusion method for person re-identification effectively improves the model's recognition results, achieving significant progress in both mAP and Rank-1 evaluation metrics.

[0087] The above embodiments provide a detailed description of the specific implementation of the residual augmentation fusion method for pedestrian re-identification proposed in this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the proposed method and core ideas of this invention; based on the ideas of this invention, there may be some differences in specific implementations. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A residual augmentation fusion method for pedestrian re-identification, characterized in that, Including the following steps: (1) Data preprocessing: Select the dataset required for the experiment from the pedestrian re-identification dataset, divide the data and perform preprocessing operations; (2) Constructing the main network structure: The backbone network is constructed based on ResNet-50, and the backbone network is initialized using a pre-trained model; Each network layer generates a copy of its own feature vector using residual units, and the copies generated by the lower network layers are expanded in the vertical and horizontal directions, and then fused with the copies generated by the adjacent higher network layers at the same size to reduce the loss of important features of the original layers during feature fusion. (3) Model training: First, the batch of sampled pedestrian samples are preprocessed and input into a predefined model to extract high-order features of pedestrians; then, the low-level features extracted by the backbone network and the high-level features are input into the residual expansion and fusion module. By expanding, the feature loss is reduced, so that the high and low-level features are fused at the same size and input into the loss function to calculate the loss and perform backpropagation, update the model parameters, and iteratively minimize the value of the loss function to form an optimized pedestrian re-identification model. (4) Model testing: After training, the proposed pedestrian re-identification model is used as a pedestrian feature extractor, using pedestrian features containing identity information to represent each pedestrian; By calculating the Euclidean distance between the target pedestrian and the feature vectors of each pedestrian in the test dataset, and sorting the pedestrians in the dataset according to their similarity to the target pedestrian, the pedestrian image most similar to the target pedestrian is finally selected, and the recognition accuracy of the model is calculated.

2. The residual augmentation fusion method for pedestrian re-identification according to claim 1, characterized in that: Step (1) preprocessing data augmentation is applied to the original pedestrian image dataset, including data normalization, random horizontal flipping, random erasure, random rotation, random flipping, and random brightness adjustment techniques.

3. The residual augmentation fusion method for pedestrian re-identification according to claim 1, characterized in that: (a) In step (2), the defined convolutional neural network uses ResNet-50 as the backbone network. Its structure includes four gradient mitigation stages. Each stage alleviates the gradient vanishing problem through skip connections, thereby enabling the training of deeper networks. This design allows the network to effectively extract multi-level feature information while maintaining high computational efficiency, providing rich feature representations for subsequent tasks. (b) The residual augmentation fusion module in step (2) is designed to alleviate the problem of information loss of low-level features during the transmission process during feature fusion. It uses a residual unit composed of two CBR units of different sizes to generate a copy of the input feature vector for each network layer through weight sharing. By retaining the copy, the loss of important features of the original layer is minimized. A CBR unit is a network layer consisting of a convolutional filter, a batch normalization layer, and a linear rectifier.

4. The residual augmentation fusion method for pedestrian re-identification according to claim 1, characterized in that: The loss function L constructed in step (3) is as follows: ； Among them, the loss function For Triplet Loss, the loss function is... For Center Loss, Indicates the number of available triples. It is a hyperparameter used to represent intervals. Represents the hinge loss function. It is a category The center of all sample features, and A and B are the weights of each loss function.

Images