A pedestrian re-identification method and system based on dynamic and static regular mixed sampling

Abstract

The present invention discloses a pedestrian re-identification method and system with dynamic and static regular mixed sampling, and relates to the technical field of public safety intelligent video surveillance. In practice, sampling is often independent of the training of the pedestrian re-identification model, resulting in uncontrolled information loss during the training process, which restricts the accuracy of identification. To this end, the present invention designs a hybrid method of regular dynamic linear sampling and static linear sampling to achieve joint optimization of sampling effects during the training of the pedestrian re-identification model, wherein the former learns dynamic pixel combination weights in a data-driven dynamic manner through normalized learnable parameters; the latter uses bilinear transformation to determine static pixel combination weights to achieve a data-independent sampling strategy. The present invention further designs a dynamic and static regularization term to constrain the difference between the dynamic pixel combination weights and the static pixel combination weights, control the degree of freedom of the learnable parameters, better combine dynamic and static sampling, and reduce information loss in the traditional single static sampling process.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a pedestrian re-identification method and system based on dynamic and static regular mixed sampling. Background Art

[0002] Pedestrian re-identification aims to retrieve pedestrian images with the same identity as the target pedestrian image from a large-scale set of pedestrian images collected by a cross-domain camera network, which is of great significance to the improvement of public systems. The resolution of pedestrian images has a huge impact on the recognition accuracy of the pedestrian re-identification model. For example, pedestrian images with too low resolution are generally difficult to identify; and the recognition of pedestrian images with too high resolution requires a lot of computation. In practice, the resolution of pedestrian images is generally pre-scaled to a reasonable range using the traditional static bilinear sampling method based on nearest neighbor relationship and non-learning, and then the pedestrian re-identification model is trained. Therefore, the sampling step does not actually participate in the training of the pedestrian re-identification model, that is, sampling is independent of model training, resulting in uncontrolled loss of sampled information during the training process, which limits the accuracy of pedestrian re-identification. To this end, the present invention proposes a pedestrian re-identification method and system based on dynamic and static regularized mixed sampling. On the basis of traditional static bilinear sampling, data-driven dynamic sampling is added, and regularization technology is introduced to combine dynamic and static sampling, so as to reduce information loss in the traditional single static sampling process, realize joint optimization of sampling effects during the training process of pedestrian re-identification model, and improve the accuracy of pedestrian re-identification. Summary of the invention

[0003] The purpose of the present invention is to solve the problems in the prior art.

[0004] The technical solution adopted by the present invention to solve the technical problem is: to provide a pedestrian re-identification method and system based on dynamic and static regular mixed sampling, including the following steps:

[0005] Collect pedestrian images to build a dataset, where each image is labeled with an identity tag;

[0006] A pedestrian re-identification model consisting of a static-dynamic regular mixed sampling module and a deep learning network is constructed. The static-dynamic regular mixed sampling module samples the input image. Sampling is performed and the sampled image is output ; Deep learning network based on output image Perform pedestrian re-identification;

[0007] Construct a loss function and use the data set to train a pedestrian re-identification model. The parameters of the loss function include static pixel combination weights. and dynamic pixel combination weights ;

[0008] Use the trained person re-identification model to perform person re-identification on pedestrian images;

[0009] The dynamic and static regular mixed sampling module uses a bilinear transformation algorithm to determine the sampling grid and static pixel combination weights , h and w represent the height and width of the output image after sampling respectively; the sampling grid Records the four domain positions of each pixel in the sampled output image, and each domain position records the x-coordinate and y-coordinate; static pixel combination weight Records the weights of the original pixels corresponding to the four domain positions of each pixel in the sampled output image; uses learnable parameters as dynamic pixel combination weights And perform mean initialization; use Softmax to weight dynamic pixel combinations Normalize and combine weights with static pixels Perform weighted mixing to obtain the pixel combination weights of dynamic and static mixing The dynamic and static regular mixed sampling module uses the pixel combination weight of dynamic and static mixing and the sampling grid For input image Sampling is performed and the sampled image is output .

[0010] Preferably, the learnable parameters are used as dynamic pixel combination weights. And perform mean initialization, expressed as:

[0011] ;

[0012] in, The generated shape is A full volume of Represents transformation into learnable parameter operations.

[0013] Preferably, the pixel combination weight of the dynamic and static mixture is , expressed as:

[0014] ;

[0015] in, Indicates that the Softmax operation is performed along the third dimension, that is, normalization is performed along the corresponding dimension of 4 to ensure that the output image is sampled. Each pixel is a linear combination of the original image pixels with a set of normalized weights.

[0016] Preferably, the sampled output image , expressed as:

[0017] ;

[0018] in, Represents an input image with RGB three channels, H and W represent the height and width of the input image respectively; Represents an output image with RGB three channels, where h and w represent its height and width respectively; Represents the input image According to the sampling network Sampling is performed according to The 4 fields position of each pixel recorded in the read Pixels; Pixel combination weights representing static and dynamic mixing According to the sampling network Sampling is performed according to The 4 fields position of each pixel recorded in the read Elements of It means to perform a summation operation along the fourth dimension, and to sum along the dimension corresponding to 4; Represents dot multiplication operation; Represents a sum operation.

[0019] Preferably, the step of constructing a loss function and using a data set to train a person re-identification model comprises the following steps:

[0020] Combined with static pixel weights and dynamic pixel combination weights Construct dynamic and static regularization terms;

[0021] Construct an overall loss function that includes dynamic and static regularization terms;

[0022] Based on the chain rule of derivation, the mature gradient descent method in deep learning is used to Optimization is performed to achieve joint optimization of the pedestrian re-identification model and sampling effect, thereby improving the pedestrian re-identification performance.

[0023] Preferably, the pixel combination weights combined with the static and dynamic pixel combination weights Construct the dynamic and static regularization terms, expressed as:

[0024] ;

[0025] in, represents the dynamic and static regularization term based on the Frobenius norm.

[0026] Preferably, the overall loss function is expressed as:

[0027] ;

[0028] in, Represents the overall loss function; Net represents the deep learning network used to output images after sampling Extract features, Represents the network parameters of Net; express The category label of Represents the cross entropy loss function, which is used to supervise the training of Net; Expressed as a hyperparameter, it is used to control the contribution of dynamic and static regularization terms.

[0029] The present invention also provides a pedestrian re-identification system with dynamic and static regular mixed sampling, comprising:

[0030] The data collection module collects pedestrian images to build a data set, where each image is equipped with an identity label as a label;

[0031] Model building module, which builds a pedestrian re-identification model including a dynamic and static regular mixed sampling module and a deep learning network. The dynamic and static regular mixed sampling module is used to input the image. Sampling is performed and the sampled image is output ; The deep learning network outputs the image based on sampling Perform pedestrian re-identification;

[0032] Model training module, constructs a loss function and uses the data set to train the pedestrian re-identification model. The parameters of the loss function include static pixel combination weights and dynamic pixel combination weights ;

[0033] The pedestrian re-identification module uses the trained pedestrian re-identification model to perform pedestrian re-identification on pedestrian images;

[0034] The dynamic and static regular mixed sampling module uses a bilinear transformation algorithm to determine the sampling grid and static pixel combination weights , h and w represent the height and width of the output image after sampling respectively; the sampling grid Records the four domain positions of each pixel in the sampled output image, and each domain position records the x-coordinate and y-coordinate; static pixel combination weight Records the weights of the original pixels corresponding to the four domain positions of each pixel in the sampled output image; uses learnable parameters as dynamic pixel combination weights And perform mean initialization; use Softmax to weight dynamic pixel combinations Normalize and combine weights with static pixels Perform weighted mixing to obtain the pixel combination weights of dynamic and static mixing The dynamic and static regular mixed sampling module uses the pixel combination weight of dynamic and static mixing and the sampling grid For input image Sampling is performed and the sampled image is output .

[0035] The present invention has the following beneficial effects: On the one hand, in order to solve the problem of information loss in image sampling during the training process in the pedestrian re-identification task, the present invention designs a dynamic and static mixed sampling, in which the dynamic branch learns the sampling combination weights in a data-driven dynamic manner through normalized learnable parameters; the static branch uses spatial position information to determine the combination weights of static sampling, and implements a data-independent sampling strategy; this dynamic and static mixed sampling helps to more effectively compensate for the information loss of traditional sampling, improve the accuracy of pedestrian re-identification, and has important applications in public security intelligent video surveillance systems, etc. On the other hand, the present invention further constructs a dynamic and static regularization term, and optimizes the sampling results in combination with the loss function of the pedestrian re-identification method, balances the dynamic and static sampling strategies, and improves the accuracy of pedestrian re-identification.

[0036] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A method step diagram of an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of a pedestrian re-identification model according to an embodiment of the present invention;

[0039] Figure 3 4 is a system structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0040] See also Figure 1 FIG. 1 is a diagram showing steps of a method according to an embodiment of the present invention, comprising the following steps:

[0041] S101, collect pedestrian images to build a dataset, where each image is equipped with an identity label as a label;

[0042] S102, constructing a pedestrian re-identification model including a dynamic and static regular mixed sampling module and a deep learning network, wherein the dynamic and static regular mixed sampling module performs a Sampling is performed and the sampled image is output ; Deep learning network based on output image Perform pedestrian re-identification;

[0043] S103, constructing a loss function and using the data set to train a pedestrian re-identification model, wherein the parameters of the loss function include static pixel combination weights and dynamic pixel combination weights ;

[0044] S104, performing pedestrian re-identification on the pedestrian image using the trained pedestrian re-identification model;

[0045] The pedestrian re-identification model of the embodiment of the present invention can be found in Figure 2 As shown in the figure, it includes a dynamic and static regular mixed sampling (DSRMS) module and a ResNet50 network. The ResNet50 network includes a network stem, four residual groups (Layer1, Layer2, Layer3, Layer4), a generalized mean pooling (GeP) layer, a batch normalization layer (Batch Normalization, BN) and a fully connected layer (Fully Connected, FC). The dynamic and static regular mixed sampling DSRMS module adds data-driven dynamic sampling on the basis of traditional static bilinear sampling, and introduces regularization technology to combine dynamic and static sampling, so as to reduce the information loss in the traditional single static sampling process and realize the joint optimization sampling effect in the training process of the pedestrian re-identification model. It specifically includes the following steps:

[0046] S201, sampling grid and static pixel combination weight generation step: using bilinear transformation algorithm to determine the sampling grid and static pixel combination weights , h and w represent the height and width of the output image after sampling, respectively, and the sampling grid It records the four domain positions of each pixel in the sampled output image, and each domain position records the x-coordinate and y-coordinate, and the static pixel combination weight It records the weights of the original pixels corresponding to the four domain positions of each pixel in the sampled output image. represents the field of real numbers;

[0047] S202, dynamic pixel combination weight generation step: using learnable parameters as dynamic pixel combination weights And perform mean initialization. The processing process of the dynamic pixel combination weight generation step is as follows:

[0048]

[0049] in, The generated shape is A full tensor of , where h and w represent the height and width of the output image after sampling respectively; Represents the operation of converting a tensor into a learnable parameter;

[0050] S203, dynamic and static pixel combination weight mixing step: Softmax normalizes the dynamic pixel combination weight D and combines it with the static pixel combination weight Perform weighted mixing to obtain the pixel combination weights of dynamic and static mixing The processing process of the weighted mixing step of the dynamic and static pixel combination is as follows:

[0051]

[0052] in, The pixel combination weight representing the dynamic and static mixture; Indicates a Softmax operation along the third dimension;

[0053] S204, dynamic and static mixed sampling step: pixel combination weight K and sampling grid based on dynamic and static mixing For input image Sampling is performed and the sampled image is output The processing of the dynamic and static mixed sampling step is as follows:

[0054]

[0055] in, Represents an input image with RGB three channels, H and W represent the height and width of the input image respectively; Represents an output image with RGB three channels, where h and w represent its height and width respectively; Pixel combination weighted sampling gridding to represent dynamic and static mixing; represents a summation operation along the fourth dimension; Represents dot multiplication operation; Represents the sum operation;

[0056] S205, joint optimization step: using the mature gradient descent method in deep learning to optimize the deep learning network. The processing process of the joint optimization step is as follows:

[0057] ;

[0058] in, represents the overall person re-identification loss function; Net represents a deep learning network, which is used to extract features from the sampled output image Y. Represents the network parameters of Net; L represents the category label of Y; Represents the cross entropy loss function, which is used to supervise the training of Net; represents the dynamic and static regularization term based on the Frobenius norm, which controls the static pixel combination weights and dynamic pixel combination weights The difference controls the dynamic pixel combination weight The degree of freedom can better achieve the mixed effect of dynamic and static sampling; It is expressed as a hyperparameter, which is used to control the contribution of the dynamic and static regularization terms. In this embodiment of the present invention, Based on the chain rule of derivation, when the mature gradient descent method in deep learning is used to optimize the Loss, the dynamic pixel combination weights of Y will be adjusted. , and the dynamic pixel combination weights will be adjusted for optimization of the dynamic and static regularization terms , thereby achieving the joint optimization of the pedestrian re-identification model and sampling effect, thereby improving the pedestrian re-identification performance.

[0059] The embodiment of the present invention runs on the Ubuntu 18.04.5 LTS system, equipped with an Intel Xeon E5-2698 v4 CPU, 128GB RAM, and a GeForce RTX 3090 GPU with 24GB video memory. Model training is based on PyTorch 1.13 and CUDA 11.7, while NumPy is used for data processing and Matplotlib is used for result visualization. Git is used for code version management, and ResNet50 pre-trained on the ImageNet dataset is used as the network backbone. The dataset uses Market-1501, one of the most commonly used public datasets in the field of person re-identification (Re-ID), for model training; the training is carried out for a total of 120 epochs, and the initial learning rate is set to 3.5×10 −4 , and decayed 10 times at the 40th and 90th Epochs respectively. The optimizer uses the Adam algorithm, and the weight decay rate is 5×10 −4, the momentum is set to 0.9. At the same time, the batch size is 64, each batch contains 16 IDs, and each ID corresponds to 4 images. In the inference stage, the re-identification of pedestrians is ranked by combining the Euclidean distance of the normalized features and combining the characteristics of the BN layer and the GeP layer. After training, Rank-1 and mAP are used as evaluation indicators, which conform to the dataset protocol: Rank-1 (Rank-1 Accuracy) indicates the proportion of correct matches of the test samples that appear in the first retrieval result for the first time, reflecting the recognition ability of the model; mAP (mean Average Precision) is used to evaluate the average accuracy of the matching targets in all retrieval results, comprehensively measuring the overall retrieval performance of the model.

[0060] For details, see Figure 3 FIG. 1 is a system structure diagram of an embodiment of the present invention, including:

[0061] The data collection module 301 collects pedestrian images to construct a data set, wherein each image is equipped with an identity identifier as a label;

[0062] Model building module 302, building a pedestrian re-identification model including a dynamic and static regular mixed sampling module and a deep learning network, the dynamic and static regular mixed sampling module is used to input the image Sampling is performed and the sampled image is output ; Deep learning network based on output image Perform pedestrian re-identification;

[0063] Model training module 303, constructs a loss function and uses the data set to train a pedestrian re-identification model, wherein the parameters of the loss function include static pixel combination weights and dynamic pixel combination weights ;

[0064] A pedestrian re-identification module 304 performs pedestrian re-identification on pedestrian images using a trained pedestrian re-identification model;

[0065] The key of the present invention is to design a pedestrian re-identification method and system with dynamic and static regular mixed sampling, solve the problem that sampling is independent of model training, and realize the joint optimization sampling effect in the process of pedestrian re-identification model training, wherein the regular dynamic linear sampling learns the dynamic pixel combination weight in a data-driven dynamic manner through normalized learnable parameters; the static linear sampling hybrid method uses bilinear transformation to determine the static pixel combination weight to realize a data-independent sampling strategy. In addition, the present invention further designs a dynamic and static regularization term to constrain the difference between the dynamic pixel combination weight and the static pixel combination weight, control the degree of freedom of the learnable parameters, better combine dynamic and static sampling, and reduce information loss in the traditional single static sampling process. Therefore, the present invention can be widely used in public security intelligent video surveillance systems.

[0066] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A pedestrian re-identification method based on dynamic and static regular mixed sampling, characterized in that: The steps include: Collect pedestrian images to build a dataset, where each image is labeled with an identity tag; A pedestrian re-identification model is constructed, which includes a dynamic and static regular mixed sampling module and a deep learning network. The dynamic and static regular mixed sampling module samples the input image X and obtains the sampled output image Y. The deep learning network performs pedestrian re-identification based on the output image Y; Constructing a loss function and using the data set to train a pedestrian re-identification model, wherein the parameters of the loss function include a static pixel combination weight S and a dynamic pixel combination weight D; Use the trained person re-identification model to perform person re-identification on pedestrian images; The dynamic and static regular mixed sampling module uses a bilinear transformation algorithm to determine the sampling grid and static pixel combination weights h and w represent the height and width of the sampled output image respectively; the sampling grid G ​​records the four domain positions of each pixel in the sampled output image, and each domain position records the x-coordinate and y-coordinate; the static pixel combination weight S records the weight of the original pixel corresponding to the four domain positions of each pixel in the sampled output image; the learnable parameters are used as the dynamic pixel combination weight D and initialized with the mean; the dynamic pixel combination weight D is normalized by Softmax and weighted mixed with the static pixel combination weight S to obtain the dynamic and static mixed pixel combination weight The dynamic and static regular mixed sampling module samples the input image X using the dynamic and static mixed pixel combination weight K and the sampling grid G ​​to obtain the sampled output image Y; The pixel combination weight of the dynamic and static mixture It is expressed as: K = S + Softmax (D, dim = 2); Among them, dim=2 means to perform Softmax operation along the third dimension, ensuring that each pixel of the output image after sampling is a linear combination of a set of normalized weights on the original image pixels; The method of constructing a loss function and using a data set to train a person re-identification model includes the following steps: Combine the static pixel combination weight S and the dynamic pixel combination weight D to construct the dynamic and static regularization term; Construct an overall loss function that includes dynamic and static regularization terms; Based on the chain rule of derivation, the mature gradient descent method in deep learning is used to optimize the loss, thereby achieving joint optimization of the pedestrian re-identification model and sampling effect, thereby improving the pedestrian re-identification performance; The static pixel combination weight S and the dynamic pixel combination weight D are combined to construct the static and dynamic regularization term, which is expressed as: <h2 style=";text-align:left;direction:ltr">||S-Softmax(D,dim=2)||<h2 style=";text-align:left;direction:ltr"> F <h2 style=";text-align:left;direction:ltr"> ; Among them, |||| F represents the dynamic and static regularization term based on the Frobenius norm; The overall loss function is expressed as: Loss=CE(Net(Y,θ),L)+λ||S-Softmax(D,dim=2)|| F ; Among them, Loss represents the overall loss function; Net represents the deep learning network, which is used to extract features from the sampled output image Y, θ represents the network parameters of Net; L represents the category label of D; CE represents the cross entropy loss function, which is used to supervise the training of Net; λ>0 represents a hyperparameter, which is used to control the contribution of dynamic and static regularization terms.

2. The pedestrian re-identification method based on dynamic and static regular mixed sampling according to claim 1 is characterized in that: The learnable parameters are used as dynamic pixel combination weights D and the mean is initialized, which is expressed as: Among them, Ones(h,w,4) means generating a full tensor with shape (h,w,4); Parameter means converting to a learnable parameter operation.

3. The pedestrian re-identification method based on dynamic and static regular mixed sampling according to claim 1 is characterized in that: The sampled output image It is expressed as: in, Represents an input image with RGB three channels, H and W represent the height and width of the input image respectively; Represents an output image with RGB three channels, where h and w represent its height and width respectively; It means that the input image X is sampled according to the sampling network G, and the pixels of X are read according to the four domain positions of each pixel recorded in G; The pixel combination weight K representing the dynamic and static mixture is sampled according to the sampling network G, and the elements of K are read according to the 4 domain positions of each pixel recorded in G; dim=3 indicates a summation operation along the fourth dimension; ⊙ indicates a dot multiplication operation; SUM indicates a summation operation.

4. A pedestrian re-identification system based on dynamic and static regular mixed sampling, characterized in that: include: The data collection module collects pedestrian images to build a data set, where each image is equipped with an identity label as a label; A model building module is used to build a pedestrian re-identification model including a dynamic and static regular mixed sampling module and a deep learning network. The dynamic and static regular mixed sampling module samples the input image X and obtains the sampled output image Y. The deep learning network performs pedestrian re-identification based on the output image Y; A model training module constructs a loss function and uses the data set to train a pedestrian re-identification model, wherein the parameters of the loss function include a static pixel combination weight S and a dynamic pixel combination weight D; The pedestrian re-identification module uses the trained pedestrian re-identification model to perform pedestrian re-identification on pedestrian images; wherein the dynamic and static regular mixed sampling module uses a bilinear transformation algorithm to determine the sampling grid and static pixel combination weights h and w represent the height and width of the sampled output image respectively; the sampling grid G ​​records the four domain positions of each pixel in the sampled output image, and each domain position records the x-coordinate and y-coordinate; the static pixel combination weight S records the weight of the original pixel corresponding to the four domain positions of each pixel in the sampled output image; the learnable parameters are used as the dynamic pixel combination weight D and initialized with the mean; the dynamic pixel combination weight D is normalized by Softmax and weighted mixed with the static pixel combination weight S to obtain the dynamic and static mixed pixel combination weight The dynamic and static regular mixed sampling module samples the input image X using the dynamic and static mixed pixel combination weight K and the sampling grid G ​​to obtain the sampled output image Y; The pixel combination weight of the dynamic and static mixture It is expressed as: K = S + Softmax (D, dim = 2); Among them, dim=2 means to perform Softmax operation along the third dimension, ensuring that each pixel of the output image after sampling is a linear combination of a set of normalized weights on the original image pixels; The method of constructing a loss function and using a data set to train a person re-identification model includes the following steps: Combine the static pixel combination weight S and the dynamic pixel combination weight D to construct the dynamic and static regularization term; Construct an overall loss function that includes dynamic and static regularization terms; Based on the chain rule of derivation, the mature gradient descent method in deep learning is used to optimize the loss, thereby achieving joint optimization of the pedestrian re-identification model and sampling effect, thereby improving the pedestrian re-identification performance; The static pixel combination weight S and the dynamic pixel combination weight D are combined to construct the static and dynamic regularization term, which is expressed as: <h2 style=";text-align:left;direction:ltr">||S-Softmax(D,dim=2)||<h2 style=";text-align:left;direction:ltr"> F <h2 style=";text-align:left;direction:ltr"> ; Among them, || || F represents the dynamic and static regularization term based on the Frobenius norm; The overall loss function is expressed as: Loss=CE(Net(Y,θ),L)+λ||S-Softmax(D,dim=2)|| F ; Wherein, Loss represents the overall loss function; Net represents the deep learning network, which is used to extract features from the sampled output image Y, θ represents the network parameters of Net; L represents the category label of D; CE represents the cross entropy loss function, which is used to supervise the training of Net; λ>0 represents a hyperparameter, which is used to control the contribution of dynamic and static regularization terms.

Images