Pedestrian re-identification method, system, electronic device and storage medium
By constructing the auxiliary and target models of convolutional neural networks and using loss function training, the accuracy of pedestrian recognition technology is improved without increasing the parameter quantity and calculation quantity, and the parameter quantity and calculation in the existing technology are solved The problem of increasing volume is suitable for portable devices and scenarios with high real-time performance.
Patent Information
- Application Number
- CN202111344388.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-15
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-11-15
AI Technical Summary
When the existing pedestrian re-identification technology improves accuracy, the increase in parameter quantity and calculation quantity is not conducive to the storage and deployment of portable devices, and is not suitable for scenarios with high real-time requirements.
Build an auxiliary training model and target model based on convolutional neural network, train through cross-entropy loss function, feature similarity loss function and class-center loss function, and use knowledge migration to increase pedestrian weight without increasing the amount of parameters and calculations The accuracy of recognition.
Without increasing the amount of parameters and calculations, the accuracy of pedestrian re-identification is improved, and is suitable for portable devices and scenarios with high real-time requirements.
Smart Images

Figure CN114299442B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of deep learning technology, and in particular to a pedestrian re-identification method, system, electronic device and storage medium. Background Art
[0002] Deep learning technology can solve problems in computer vision fields such as image classification, image segmentation and object detection. With the continuous development of deep learning technology, pedestrian re-identification technology has also made great progress.
[0003] Person re-identification (Re-ID) is an important image recognition technology. Person re-identification searches cameras distributed in different locations to determine whether pedestrians in the field of view of different cameras are the same pedestrian. In order to further improve network performance, related technologies usually improve the accuracy of person re-identification technology by building more complex network structures. However, deeper, wider or more complex networks usually lead to a surge in the number of parameters and computational complexity. The increase in the number of parameters is not conducive to the storage and deployment of portable devices, and the increase in computational complexity is not conducive to application in scenarios with high real-time requirements.
[0004] Therefore, how to improve the accuracy of pedestrian re-identification without increasing the number of parameters and the amount of calculation is a technical problem that technicians in this field currently need to solve. Summary of the invention
[0005] The purpose of this application is to provide a pedestrian re-identification method, system, electronic device and storage medium, which can improve the accuracy of pedestrian re-identification without increasing the number of parameters and the amount of calculation.
[0006] In order to solve the above technical problems, the present application provides a pedestrian re-identification method, which includes:
[0007] Construct auxiliary training models and target models based on convolutional neural networks;
[0008] Determine the loss function of the auxiliary training model and the target model, and train the auxiliary training model and the target model using the loss function;
[0009] After the auxiliary training model is trained, the knowledge of the auxiliary training model is transferred to the target model to obtain a pedestrian re-identification model;
[0010] Inputting a pedestrian image into the pedestrian re-identification model to obtain an embedding layer feature of the pedestrian image;
[0011] The embedding layer features of the pedestrian image are compared with the embedding layer of the query image for similarity, and a pedestrian re-identification result is output according to the similarity comparison result.
[0012] Optionally, build auxiliary training models and target models based on convolutional neural networks, including:
[0013] Constructing the auxiliary training model including at least two convolutional neural networks, and constructing the target model including at least two convolutional neural networks;
[0014] Or, the auxiliary training model is constructed using a convolutional neural network including at least two head networks, and the target model is constructed using a convolutional neural network including at least two head networks;
[0015] The head network includes a pooling layer, an embedding layer, a fully connected layer, an output layer and a softmax layer.
[0016] Optionally, determining the loss function of the auxiliary training model and the target model includes:
[0017] Calculating the cross entropy loss function of each of the convolutional neural networks;
[0018] Performing similarity comparison on the embedding layer features of the convolutional neural network to obtain a feature similarity loss function;
[0019] Determining a class center loss function according to the class center of the embedding layer features of the convolutional neural network;
[0020] Constraining the position of the class center of the embedding layer feature according to a preset rule to obtain a loss function constraining the class center distance;
[0021] The loss functions of the auxiliary training model and the target model are determined according to the cross entropy loss function, the feature similarity loss function, the class center loss function, and the loss function for constraining the class center distance.
[0022] Optionally, also include:
[0023] The most recently determined class center and the currently output embedding layer features are weighted to obtain the updated class center.
[0024] Optionally, before weighting the most recently determined class center and the currently output embedding layer features, the following is also included:
[0025] Determine whether the feature classification corresponding to the embedding layer feature currently output is correct;
[0026] If so, the step of weighted calculation of the most recently determined class center and the currently output embedding layer features is entered.
[0027] Optionally, constraining the position of the class center of the embedding layer feature according to a preset rule to obtain a loss function constraining the class center distance includes:
[0028] Calculate the minimum inter-class difference of all class centers by a difficult sample mining method to obtain the position information of the class center of the embedding layer feature;
[0029] The loss function for constraining the class center distance is generated according to the position information.
[0030] Optionally, the constructing of an auxiliary training model and a target model based on a convolutional neural network includes:
[0031] The auxiliary training model and the target model based on the convolutional neural network are constructed according to preset rules; wherein the preset rule is that the model complexity of the auxiliary training model is higher than the model complexity of the target model.
[0032] The present application also provides a pedestrian re-identification system, the system comprising:
[0033] Model building module, used to build auxiliary training models and target models based on convolutional neural networks;
[0034] A model training module, used to determine the loss function of the auxiliary training model and the target model, and train the auxiliary training model and the target model using the loss function;
[0035] A knowledge transfer module, used to transfer the knowledge of the auxiliary training model to the target model after the auxiliary training model is trained, so as to obtain a pedestrian re-identification model;
[0036] A feature extraction module, used for inputting a pedestrian image into the pedestrian re-identification model to obtain an embedding layer feature of the pedestrian image;
[0037] The pedestrian re-identification module is used to perform similarity comparison between the embedding layer features of the pedestrian image and the embedding layer of the query image, and output the pedestrian re-identification result according to the similarity comparison result.
[0038] The present application also provides a storage medium on which a computer program is stored, and when the computer program is executed, the steps of the above-mentioned pedestrian re-identification method are implemented.
[0039] The present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned pedestrian re-identification method when calling the computer program in the memory.
[0040] The present application provides a pedestrian re-identification method, comprising: constructing an auxiliary training model and a target model based on a convolutional neural network; determining the loss functions of the auxiliary training model and the target model, and using the loss functions to train the auxiliary training model and the target model; after the auxiliary training model is trained, migrating the knowledge of the auxiliary training model to the target model to obtain a pedestrian re-identification model; inputting a pedestrian image into the pedestrian re-identification model to obtain an embedding layer feature of the pedestrian image; performing a similarity comparison between the embedding layer feature of the pedestrian image and the embedding layer of the image to be queried, and outputting a pedestrian re-identification result based on the similarity comparison result.
[0041] The present application constructs an auxiliary training model and a target model based on a convolutional neural network, determines the loss functions of the auxiliary training model and the target model, and then uses the loss function to train the auxiliary training model and the target model. After the auxiliary training model is trained, the present application transfers the knowledge learned in the auxiliary training model to the target model through knowledge transfer to obtain a pedestrian re-identification model. Since the pedestrian re-identification model includes the knowledge learned by the auxiliary training model and the target model, the accuracy of the pedestrian re-identification model can be improved without the need for additional reasoning costs. Therefore, the present application can improve the accuracy of pedestrian re-identification without increasing the number of parameters and the amount of calculation. The present application also provides a pedestrian re-identification system, an electronic device and a storage medium, which have the above-mentioned beneficial effects and are not repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0043] Figure 1 A flowchart of a pedestrian re-identification method provided in an embodiment of the present application;
[0044] Figure 2 A schematic diagram of the first model provided in the embodiment of the present application;
[0045] Figure 3 A schematic diagram of a second model provided in an embodiment of the present application;
[0046] Figure 4 A schematic diagram of a model retention result provided in an embodiment of the present application;
[0047] Figure 5 A schematic diagram of a pedestrian re-identification application provided in an embodiment of the present application;
[0048] Figure 6 A schematic diagram of the structure of a pedestrian re-identification system provided in an embodiment of the present application. DETAILED DESCRIPTION
[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0050] See below Figure 1 , Figure 1 A flowchart of a pedestrian re-identification method provided in an embodiment of the present application.
[0051] Specific steps may include:
[0052] S101: Construct auxiliary training model and target model based on convolutional neural network;
[0053] Among them, this embodiment can establish an auxiliary training model and a target model including one or more convolutional neural networks based on the idea of knowledge supervised learning. The above-mentioned convolutional neural networks can realize knowledge transfer and thus improve the generalization ability of each model through mutual supervised learning.
[0054] As a feasible implementation mode, the present application can construct an auxiliary training model and a target model based on a convolutional neural network in the following manner: construct the auxiliary training model including at least two convolutional neural networks, and construct the target model including at least two convolutional neural networks; or, construct the auxiliary training model using a convolutional neural network including at least two head networks, and construct the target model using a convolutional neural network including at least two head networks; wherein the head network includes a pooling layer, an embedding layer, a fully connected layer, an output layer and a softmax layer.
[0055] See also Figure 2 , Figure 2 The first model schematic diagram provided in the embodiment of the present application provides an implementation method for establishing an auxiliary training model and a target model including two convolutional neural networks. The auxiliary training model and / or the target model can be Figure 2 The model shown. Figure 2As shown, two convolutional neural networks Net1 and Net2 are established. These two convolutional neural networks can be isomorphic or heterogeneous. The output of the network can reduce the dimension of the feature map (Batchsize×Channel×H×W) into a vector through the Pooling layer. Here, e1 and e2 are used to represent the embedding layer features, and the dimensions of e1 and e2 are Batchsize×Channel. The above model includes a backbone model and a head model. The backbone network is used to extract features, and the head network is used to implement classification and loss function calculation. The head network includes a pooling layer pool, an embedding layer, a fully connected layer fc, an output layer, and a softmax layer. The head network can use the triplet loss function Triplet loss and the cross-entropy loss function cross-entropy loss for parameter adjustment.
[0056] See also Figure 3 , Figure 3 The second model schematic diagram provided in the embodiment of the present application shows an implementation method for establishing an auxiliary training model and a target model including a convolutional neural network. The model is a multi-head convolutional neural network (i.e., there are multiple head networks).
[0057] The auxiliary training model in this embodiment can be Figure 2 and Figure 3 For any of the models shown, the target training model can also be Figure 2 and Figure 3 Any of the models shown. The complexity of the auxiliary training model is higher than that of the target model, and the complexity of the model can be measured by the number of parameters and the amount of calculation of the model. Specifically, the present embodiment can construct the auxiliary training model and the target model based on the convolutional neural network according to a preset rule; wherein the preset rule is that the model complexity of the auxiliary training model is higher than the model complexity of the target model.
[0058] S102: Determine the loss function of the auxiliary training model and the target model, and train the auxiliary training model and the target model using the loss function;
[0059] Among them, in this embodiment, the loss functions of the auxiliary training model and the target model can be the same, and the loss functions of the two can be determined specifically in the following ways: calculating the cross entropy loss function of each of the convolutional neural networks; performing similarity comparison on the embedding layer features of the convolutional neural network to obtain a feature similarity loss function; determining the class center loss function according to the class center of the embedding layer features of the convolutional neural network; constraining the position of the class center of the embedding layer features according to preset rules to obtain a loss function that constrains the class center distance; determining the loss functions of the auxiliary training model and the target model according to the cross entropy loss function, the feature similarity loss function, the class center loss function, and the loss function that constrains the class center distance.
[0060] Furthermore, this embodiment can obtain a loss function that constrains the class center distance in the following manner: calculate the minimum inter-class difference of all class centers through the difficult sample mining method to obtain the position information of the class center of the embedding layer feature; and generate a loss function that constrains the class center distance based on the position information.
[0061] Before this step, there may be a step of obtaining training data for pedestrian re-identification, and then using the training data to train the auxiliary training model and the target model respectively. This embodiment improves the accuracy of neural network training and reasoning without increasing the number of parameters and calculations in the network during reasoning.
[0062] S103: After the auxiliary training model is trained, the knowledge of the auxiliary training model is transferred to the target model to obtain a pedestrian re-identification model;
[0063] After the auxiliary training model is trained, the auxiliary training model has learned knowledge information about person re-identification. This embodiment can transfer the above knowledge information to the target model through knowledge transfer. This embodiment uses the target model that has been trained and has transferred the knowledge of the auxiliary training model as the person re-identification model. The above knowledge refers to the features in the network. This embodiment provides additional regularization information for multiple views of the same data, thereby improving the accuracy of the network.
[0064] S104: inputting a pedestrian image into a pedestrian re-identification model to obtain an embedding layer feature of the pedestrian image;
[0065] After obtaining the pedestrian re-identification model, if a pedestrian re-identification task is received, a pedestrian image is input into the pedestrian re-identification model to obtain the embedding layer features of each pedestrian image.
[0066] S105: performing a similarity comparison between the embedding layer features of the pedestrian image and the embedding layer of the query image, and outputting a pedestrian re-identification result according to the similarity comparison result.
[0067] Among them, this embodiment can perform a similarity comparison between the embedding layer features of the pedestrian image and the embedding layer of the query image, and determine the pedestrian image with the highest similarity based on the similarity comparison result, so as to use the pedestrian image with the highest similarity as the pedestrian re-identification result.
[0068] This embodiment constructs an auxiliary training model and a target model based on a convolutional neural network, determines the loss functions of the auxiliary training model and the target model, and then uses the loss function to train the auxiliary training model and the target model. After the auxiliary training model is trained, this embodiment transfers the knowledge learned in the auxiliary training model to the target model by means of knowledge transfer to obtain a pedestrian re-identification model. Since the pedestrian re-identification model includes the knowledge learned by the auxiliary training model and the target model, the accuracy of the pedestrian re-identification model can be improved without the need for additional reasoning costs. Therefore, this embodiment can improve the accuracy of pedestrian re-identification without increasing the number of parameters and the amount of calculation.
[0069] As for Figure 1 Further introduction to the corresponding embodiment, this embodiment can perform weighted calculation on the most recently determined class center and the currently output embedding layer feature to obtain an updated class center. Before performing weighted calculation on the most recently determined class center and the currently output embedding layer feature, it can also be determined whether the feature classification corresponding to the currently output embedding layer feature is correct; if so, enter the step of performing weighted calculation on the most recently determined class center and the currently output embedding layer feature; if not, discard the currently output embedding layer feature.
[0070] The following describes the process of training the model in the above embodiment. After a convolutional neural network is established, it needs to be trained to converge, and after convergence, the trained network weights are obtained. During the inference process, the weight coefficients trained by the network are pre-loaded to perform the final classification of the input data.
[0071] The model training ideas of this embodiment are as follows: (1) According to different network structures, multiple network models for training are constructed, and usually a larger model (i.e., auxiliary training model) and a smaller model (i.e., target model) are selected to realize knowledge transfer. Cross entropy loss, mutual learning loss, class center loss, and class center optimization loss are calculated for all network models. Among them, the cross entropy loss is calculated by the cross entropy loss function, the mutual learning loss is calculated by the feature similarity loss function, the class center loss is calculated by the class center loss function, and the class center optimization loss is calculated by the loss function that constrains the class center distance. The network is trained according to the above loss function to make it converge.
[0072] The training process of convolutional neural network is as follows: The training process of convolutional neural network is divided into two stages. The first stage is the stage of data propagation from low level to high level, that is, the forward propagation stage. The other stage is when the result obtained by forward propagation does not match the expectation, the error is propagated from high level to low level for training, that is, the back propagation stage. The training process includes the following steps:
[0073] Step 1: Initialize the network layer weights, usually using random initialization;
[0074] Step 2: The input image data is forward propagated through the convolutional layer, downsampling layer, fully connected layer, etc. to obtain the output value;
[0075] Step 3: Find the error between the network output value and the target value (label):
[0076] Step 4: Propagate the error back to the network and obtain the back propagation error of each layer of the network in turn: fully connected layer, convolutional layer, etc.
[0077] Step 5: Each layer of the network adjusts all weight coefficients in the network according to the back propagation error of each layer, that is, updates the weights.
[0078] Step 6: Randomly select new image data again, and then proceed to the second step to obtain the output value through network forward propagation.
[0079] Step 7: Infinitely iterate back and forth. When the error between the network output value and the target value (label) is less than a certain threshold, or the number of iterations exceeds a certain threshold, the training ends.
[0080] Step 8: Save the trained network parameters of all layers and store the trained weights.
[0081] See also Figure 4 , Figure 4 A schematic diagram of a model retention result provided in an embodiment of the present application. The reasoning process provided in this embodiment is as follows: remove all auxiliary training models, retain only one network model (i.e., the target model), load pre-trained weights, and classify images or extract image features.
[0082] The process described in the above embodiment is explained below through an embodiment in actual application.
[0083] With the continuous development of deep learning, deep learning networks have achieved remarkable performance in various fields. In order to further improve network performance, scholars usually continue to improve its performance by building more complex network structures. However, improving network performance in this way has the following disadvantages: (1) Deeper, wider or more complex networks usually lead to a surge in the number of parameters, and the increase in the number of parameters is not conducive to the storage and deployment of portable devices. For example, to implement real-time pedestrian detection and recognition program deployment in network cameras, the network needs to have a smaller number of parameters (easy to store) and a higher recognition accuracy. (2) Deeper, wider or more complex networks usually lead to an increase in computational complexity, which is not conducive to application scenarios with high real-time requirements. Large computational delays will cause the entire system to miss the best opportunity and have a negative impact on system functions. Therefore, how to improve network performance without increasing the number of parameters and computational complexity has become a problem that needs to be solved.
[0084] To solve the above problems, this embodiment proposes a method for constructing, training, and reasoning a convolutional neural network based on knowledge supervision, which can achieve knowledge transfer (migrating knowledge from a large model to a small model) without increasing the number of parameters and the amount of calculation, maximize the potential of the network, and improve network performance. In this embodiment, multiple results for the same image can assist each other, thereby using the knowledge learned by the group to obtain more accurate results. Among them, multiple results include both final results and intermediate results.
[0085] This embodiment is based on the idea of knowledge supervised learning. First, one or more networks are established. The networks learn from each other through mutual supervision to achieve knowledge transfer and improve the generalization ability of each model.
[0086] In the task of person re-identification, the embedding layer features are ultimately used for feature matching retrieval, so constrained optimization is of great significance for the task of person re-identification.
[0087] This embodiment designs a new loss function, and the implementation steps of the function are as follows:
[0088] Loss function (1) Calculation process: Add a fully connected layer (fc) after the embedding layer to obtain the fully connected layer features, perform softmax normalization on the fully connected layer features, and finally calculate the loss using the cross entropy loss function. The superscript 1 represents the first branch.
[0089] Loss function (2) calculation process: The embedded features e1 and e2 of the convolutional neural network Net1 and the convolutional neural network Net2 should be similar, because they both support the same pedestrian classification task during training, and e1 and e2 will be used for similarity comparison during reasoning. Therefore, the embedded features e1 and e2 of Net1 and Net2 are functionally the same, and similar features should be learned.
[0090] Net1 and Net2 have different network structures and initialization weight coefficients. Their e1 and e2 are diverse, but their commonality is that they have excellent expression ability for pedestrians. In order to give full play to their commonality advantages and suppress noise. This embodiment provides a mutual learning mechanism, that is, e1 learns from e2, and e2 learns from e1, and obtains the feature similarity loss function L m .
[0091] Its Loss function is:
[0092] Where n represents the nth input sample, u and v represent the uth network and the vth network. The formula can be summarized as follows:
[0093] Traverse all samples in each batch. As mentioned above, assuming that each batch contains N samples, traverse N times. Pass the samples through each network in turn and obtain the output results of the embedding layer of each network. For example, for sample x n , assuming there are 2 networks, there are 2 embedding layers outputting the results and Similarly, if there are 3 networks, there are 3 embedding layer outputs.
[0094] For the embedding layer output results of all networks, perform a pairwise traversal. For example, in this embodiment, there are two networks 1 and 2. Use the above formula to calculate the feature similarity loss L between the two networks e Similarly, assuming there are 3 networks, there are 3 combinations of non-repeated traversals: (1, 2) (1, 3) (2, 3), and the feature similarity loss function L for each combination is calculated: e (u,v).
[0095] Loss function (3) calculation process: e1 and e2 features are similar, and have the following defects in the mutual learning process. For example: At the beginning of the training process, the network model prediction is very inaccurate, and its network features e1 and e2 have large deviations and noise. The mutual learning between e1 and e2 may be inaccurate features learning from inaccurate features, which may not have a good effect. In order to suppress the noise, this embodiment proposes an embedded feature optimization method, which uses a feature similarity loss function to effectively reduce the noise of embedded features. The specific implementation method is as follows:
[0096] The core idea of constructing the feature similarity loss function is that the embedding layer features of each image learn from their own class centers. Because the class centers of each type of image sample are relatively stable, the deviation caused by the embedding layer features learning from other branches can be effectively suppressed. The learning method is to obtain the class centers of all samples, and N Input each network in turn to obtain the embedding features of all samples and The superscripts 1 and 2 represent different branches, and the subscript N represents a total of N samples. For each network output, find the class center of the sample. Assume that all samples x N There are C categories in total (i.e., C pedestrians), and the class center is calculated using the following formula: in, Represents the class center of the cth class of the embedding layer features of the first network. There are C class centers in total, Similarly, for multiple networks, the class center of each network embedding layer is obtained respectively. Represents the embedding feature of the nth sample of the first network, and its category is the cth category.
[0097] For each sample, we go to the embedding layer class center corresponding to its sample category for learning, and the final feature similarity loss function is: in, Representative samples The corresponding class center. Traverse the features of each network in turn and calculate the class center loss function of each network and
[0098] Since the network is continuously iteratively optimized, the class centers of its samples relative to each network are also constantly changing. The class centers of each class of samples are dynamically updated in the following ways: The superscript u represents the uth network. The update method of the class center can be a first-in-first-out stack method, and the class center of the n-step sample closest to the current step is taken as the true class center.
[0099] This embodiment can also select the class center according to the classification probability corresponding to the embedding feature, that is, first determine whether the classification using the feature is correct, and only include it in the class center calculation if it is correct.
[0100] Loss function (4) Calculation process: For each network, the class center position can be further constrained so that the centers of each class can be separated as much as possible, which is conducive to distinguishing different pedestrian features and improving the identifiability of the network. That is, each pedestrian feature can be better separated. This embodiment can construct a loss function that constrains the class center distance:
[0101] represents the class center of the i-th class of the u-th network, Representative distance The nearest class center. In this embodiment, the method of hard sample mining (hard sample mining) can be used to achieve class center loss optimization. Hard sample mining does not seek the average value of the inter-class difference of all classes, but seeks the minimum inter-class difference of all classes.
[0102] In this embodiment, the loss functions (1) to (4) are combined to obtain the total loss function L loss :
[0103]
[0104] is the cross entropy loss function of the first convolutional neural network, is the cross entropy loss function of the second convolutional neural network, L m is the feature similarity loss function of the first convolutional neural network and the second convolutional neural network, is the class center loss function of the first convolutional neural network, is the cross entropy loss function of the second convolutional neural network, is the loss function of the first convolutional neural network that constrains the class center distance, The loss function for the second convolutional neural network constrains the class center distance.
[0105] This embodiment provides a network structure for collaborative training of multi-model knowledge, which protects the combined mutual learning loss, class center loss, and class center optimization loss functions for supervised learning training. The multi-model knowledge supervised training method mines the features in the embedding layer of the network to improve the discriminative power of the features in the embedding layer, and deletes redundant models during reasoning, so no additional reasoning cost is required to improve the accuracy. This method has broad application prospects in the field of image classification.
[0106] The following provides a method for training a model using the knowledge collaborative network training method in the above embodiment and applying it to the field of pedestrian re-identification. The training process has been described in detail above, and the specific method of reasoning application is explained below:
[0107] During inference: remove the remaining models (auxiliary training models) and keep only the main model (target model). Figure 5 , Figure 5 A schematic diagram of a pedestrian re-identification application provided in an embodiment of the present application. Figure 5 Here, conv stands for convolutional layer, and bottleneck stands for bottleneck layer. The bottleneck layer represents a specific network structure of ResNet. In the person re-identification application, input images 1, 2, 3 and the query image are input into the network to obtain the embedding layer features in the network. Images 1, 2, and 3 constitute the query data set for the person re-identification task. The query image is also input into the network to obtain the embedding layer features of the query image. The embedding layer features of the query image are compared with all the features in the query data set. The comparison method is to find the distance between the embedding layer features of the query image and all the features in the query data set, that is, the vector distance. The query data sample with the smallest distance is the same person as the query image.
[0108] For the task of pedestrian re-identification, the identifiability of the embedding features directly affects the highest performance of the model. Therefore, it is extremely important to mine the features of the model's embedding layer so that the samples can be correctly classified and discriminated. Therefore, the present invention proposes a new embedding feature mining method and a multi-model collaborative training method, which establishes a foundation for feature mining by establishing multiple neural network models. The embedding mining between branches is achieved by mutual learning of the embedding features between two models and constructing a new loss function. At the same time, the loss function of learning from each classification center using the embedding features within the branch is combined into a new loss function to train the entire network.
[0109] The training method proposed in this embodiment does not increase the number of parameters and the amount of calculation during network reasoning. By optimizing the training process, the potential of the network is tapped to enable it to achieve optimal performance, thereby showing better results in the reasoning process. This embodiment is aimed at the task of pedestrian re-identification. The present invention proposes a method for embedding feature mining with multi-model knowledge supervision and collaborative training, which can improve the accuracy of pedestrian re-identification without increasing the number of parameters and the amount of calculation.
[0110] See also Figure 6 , Figure 6A schematic diagram of the structure of a pedestrian re-identification system provided in an embodiment of the present application, the system may include:
[0111] A model building module 601 is used to build an auxiliary training model and a target model based on a convolutional neural network;
[0112] A model training module 602 is used to determine the loss function of the auxiliary training model and the target model, and train the auxiliary training model and the target model using the loss function;
[0113] A knowledge transfer module 603 is used to transfer the knowledge of the auxiliary training model to the target model after the auxiliary training model is trained to obtain a pedestrian re-identification model;
[0114] A feature extraction module 604 is used to input a pedestrian image into the pedestrian re-identification model to obtain an embedding layer feature of the pedestrian image;
[0115] The pedestrian re-identification module 605 is used to perform a similarity comparison between the embedding layer features of the pedestrian image and the embedding layer of the query image, and output a pedestrian re-identification result according to the similarity comparison result.
[0116] This embodiment constructs an auxiliary training model and a target model based on a convolutional neural network, determines the loss functions of the auxiliary training model and the target model, and then uses the loss function to train the auxiliary training model and the target model. After the auxiliary training model is trained, this embodiment transfers the knowledge learned in the auxiliary training model to the target model by means of knowledge transfer to obtain a pedestrian re-identification model. Since the pedestrian re-identification model includes the knowledge learned by the auxiliary training model and the target model, the accuracy of the pedestrian re-identification model can be improved without the need for additional reasoning costs. Therefore, this embodiment can improve the accuracy of pedestrian re-identification without increasing the number of parameters and the amount of calculation.
[0117] Furthermore, the model construction module 601 is used to construct the auxiliary training model including at least two convolutional neural networks, and to construct the target model including at least two convolutional neural networks; or, to construct the auxiliary training model using a convolutional neural network including at least two head networks, and to construct the target model using a convolutional neural network including at least two head networks; wherein the head network includes a pooling layer, an embedding layer, a fully connected layer, an output layer and a softmax layer.
[0118] Furthermore, the model training module 602 is used to calculate the cross entropy loss function of each of the convolutional neural networks; it is also used to perform similarity comparison on the embedding layer features of the convolutional neural network to obtain a feature similarity loss function; it is also used to determine the class center loss function based on the class center of the embedding layer features of the convolutional neural network; it is also used to constrain the position of the class center of the embedding layer features according to preset rules to obtain a loss function that constrains the class center distance; it is also used to determine the loss function of the auxiliary training model and the target model based on the cross entropy loss function, the feature similarity loss function, the class center loss function, and the loss function that constrains the class center distance.
[0119] Furthermore, it also includes:
[0120] The class center update module is used to perform weighted calculation on the most recently determined class center and the currently output embedding layer features to obtain the updated class center.
[0121] Furthermore, it also includes:
[0122] The judgment module is used to judge whether the feature classification corresponding to the currently output embedding layer feature is correct before performing weighted calculation on the most recently determined class center and the currently output embedding layer feature; if so, enter the step of performing weighted calculation on the most recently determined class center and the currently output embedding layer feature.
[0123] Furthermore, the model training module 602 constrains the position of the class center of the embedding layer feature according to preset rules, and the process of obtaining the loss function for constraining the class center distance includes: calculating the minimum inter-class difference of all class centers through the difficult sample mining method to obtain the position information of the class center of the embedding layer feature; generating a loss function for constraining the class center distance based on the position information.
[0124] Furthermore, the model construction module 601 is used to construct the auxiliary training model and the target model based on the convolutional neural network according to preset rules; wherein the preset rule is that the model complexity of the auxiliary training model is higher than the model complexity of the target model.
[0125] Since the embodiments of the system part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the system part, which will not be repeated here.
[0126] The present application also provides a storage medium on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.
[0127] The present application also provides an electronic device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps provided in the above embodiment may be implemented. Of course, the electronic device may also include various network interfaces, power supplies and other components.
[0128] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of this application.
[0129] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
Claims
1. A pedestrian re-identification method, characterized in that: include: Construct auxiliary training models and target models based on convolutional neural networks; Determine the loss function of the auxiliary training model and the target model, and train the auxiliary training model and the target model using the loss function; After the auxiliary training model is trained, the knowledge of the auxiliary training model is transferred to the target model to obtain a pedestrian re-identification model; Inputting a pedestrian image into the pedestrian re-identification model to obtain an embedding layer feature of the pedestrian image; Performing a similarity comparison between the embedding layer features of the pedestrian image and the embedding layer of the query image, and outputting a pedestrian re-identification result according to the similarity comparison result; Construct auxiliary training models and target models based on convolutional neural networks, including: Constructing the auxiliary training model including at least two convolutional neural networks, and constructing the target model including at least two convolutional neural networks; the two convolutional neural networks are isomorphic neural networks, or the two convolutional neural networks are heterogeneous neural networks; Or, the auxiliary training model is constructed using a convolutional neural network including at least two head networks, and the target model is constructed using a convolutional neural network including at least two head networks; The head network includes a pooling layer, an embedding layer, a fully connected layer, an output layer and a softmax layer; Determining the loss function of the auxiliary training model and the target model includes: Calculating the cross entropy loss function of each of the convolutional neural networks; Performing similarity comparison on the embedding layer features of the convolutional neural network to obtain a feature similarity loss function; Determining a class center loss function according to the class center of the embedding layer features of the convolutional neural network; Constraining the position of the class center of the embedding layer feature according to a preset rule to obtain a loss function constraining the class center distance; Determining the loss functions of the auxiliary training model and the target model according to the cross entropy loss function, the feature similarity loss function, the class center loss function, and the loss function constraining the class center distance; the loss functions of the auxiliary training model and the target model are the same; The method of constraining the position of the class center of the embedding layer feature according to a preset rule to obtain a loss function constraining the class center distance includes: Calculate the minimum inter-class difference of all class centers by a difficult sample mining method to obtain the position information of the class center of the embedding layer feature; Generating the loss function constraining the class center distance according to the position information; The construction of the auxiliary training model and the target model based on the convolutional neural network includes: Constructing the auxiliary training model and the target model based on the convolutional neural network according to a preset rule; wherein the preset rule is that the model complexity of the auxiliary training model is higher than the model complexity of the target model; Perform weighted calculation on the most recently determined class center and the currently output embedding layer features to obtain the updated class center; The determining the loss function of the auxiliary training model and the target model, and using the loss function to train the auxiliary training model and the target model includes: Use random initialization operation to initialize the network layer weights; The input image data is forward propagated through the convolution layer, the downsampling layer, and the fully connected layer to obtain an output value; the forward propagation is the propagation of the image data from a low level to a high level; Calculate the error between the network's output value and the target value; Back-propagating the error into the network, and calculating the back-propagation error of each layer in the network in turn; the network includes a fully connected layer and a convolutional layer; Adjust and update all weight coefficients in the network based on the back-propagation error of each layer; Randomly select new image data, and re-enter the step of forward propagating the input image data through the convolutional layer, the downsampling layer, and the fully connected layer to obtain the output value, so as to obtain the output value through network forward propagation; When the error between the output value of the network and the target value is less than a first preset threshold, or the number of iterations exceeds a second preset threshold, the training is terminated; Save the trained network parameters of all layers and store the trained weights so that the target model can load the pre-trained weights and perform image classification or image feature extraction operations; A fully connected layer is added after the embedding layer to obtain the fully connected layer features, softmax normalization is performed based on the fully connected layer features, and the loss is calculated using the cross entropy loss function. The loss The superscript 1 represents the first branch; Obtaining an embedded feature e1 corresponding to a convolutional neural network Net1 and an embedded feature e2 corresponding to a convolutional neural network Net2; wherein the embedded feature e1 and the embedded feature e2 have similarity and diversity and are functionally identical; and the Net1 and the Net2 have different network structures and initialization weight coefficients; The embedded feature e1 is used to learn from the embedded feature e2, and the embedded feature e2 is used to learn from the embedded feature e1 to obtain a feature similarity loss function L m : ; Where n is the nth input sample, u and v are the uth network and the vth network respectively; Traverse the embedding layer output results of all networks in pairs to obtain corresponding non-repeated traversal combinations, and calculate the feature similarity loss function Le(u,v) for each traversal combination; Find the class center of all the samples, and Input each network in turn to obtain the embedding features of all samples and ;in and The superscripts 1 and 2 represent different branches, and the subscript N represents a total of N samples; For each network output, the class center of the sample is calculated separately. When there are C categories, the class center is calculated using the formula: ; in, Characterize the class center of the cth class of the embedding layer features of the first network; use The representation has a total of C class centers; For multiple networks, calculate the class center of each network embedding layer separately; Characterize the embedding features of the nth sample of the first network, the category is the cth category; Each sample learns from the embedding layer class center corresponding to its sample category, and obtains the feature similarity loss function: ; in, Characterization samples The corresponding class center; Traverse the features of each network in turn and calculate the class center loss function of each network and ; By formula Dynamically update the class center of each sample class; The superscript u of represents the u-th network; the updating method of the class center includes a first-in-first-out stack method, and the class center of the n-step sample closest to the current step is determined as the updated class center; Or, select the class center according to the classification probability corresponding to the embedding feature, so as to enter the class center calculation when the feature classification is correct; Construct a loss function that constrains the class center distance: ; in, Characterizes the class center of the i-th class of the u-th network, Characterization distance nearest class center; The class center loss is optimized by using the difficult sample mining method, wherein the difficult sample mining is to find the minimum inter-class difference of all classes.
2. The pedestrian re-identification method according to claim 1, characterized in that: Before weighting the most recently determined class center and the currently output embedding layer features, it also includes: Determine whether the feature classification corresponding to the embedding layer feature currently output is correct; If so, the step of weighted calculation of the most recently determined class center and the currently output embedding layer features is entered.
3. A pedestrian re-identification system, characterized in that: include: Model building module, used to build auxiliary training models and target models based on convolutional neural networks; A model training module, used to determine the loss function of the auxiliary training model and the target model, and train the auxiliary training model and the target model using the loss function; A knowledge transfer module, used to transfer the knowledge of the auxiliary training model to the target model after the auxiliary training model is trained, so as to obtain a pedestrian re-identification model; A feature extraction module, used for inputting a pedestrian image into the pedestrian re-identification model to obtain an embedding layer feature of the pedestrian image; A pedestrian re-identification module is used to perform a similarity comparison between the embedding layer features of the pedestrian image and the embedding layer of the query image, and output a pedestrian re-identification result according to the similarity comparison result; The system is also used to: construct the auxiliary training model including at least two convolutional neural networks, and construct the target model including at least two convolutional neural networks; the two convolutional neural networks are isomorphic neural networks, or the two convolutional neural networks are heterogeneous neural networks; or, construct the auxiliary training model using a convolutional neural network including at least two head networks, and construct the target model using a convolutional neural network including at least two head networks; wherein the head network includes a pooling layer, an embedding layer, a fully connected layer, an output layer and a softmax layer; calculate the cross entropy loss function of each convolutional neural network; perform similarity comparison on the embedding layer features of the convolutional neural network, and obtain to feature similarity loss function; determine the class center loss function according to the class center of the embedding layer feature of the convolutional neural network; constrain the position of the class center of the embedding layer feature according to a preset rule to obtain a loss function that constrains the class center distance; determine the loss function of the auxiliary training model and the target model according to the cross entropy loss function, the feature similarity loss function, the class center loss function, and the loss function that constrains the class center distance; the loss function of the auxiliary training model and the target model is the same; calculate the minimum inter-class difference of all class centers by a difficult sample mining method to obtain the position information of the class center of the embedding layer feature; generate the target model according to the position information The invention relates to a loss function constrained by the class center distance; the auxiliary training model and the target model based on the convolutional neural network are constructed according to the preset rules; wherein the preset rule is that the model complexity of the auxiliary training model is higher than the model complexity of the target model; the class center determined most recently and the embedding layer features of the current output are weighted to obtain the updated class center; the network layer weights are initialized by random initialization operation; the input image data is forward propagated through the convolution layer, the downsampling layer, and the fully connected layer to obtain the output value; the forward propagation is the propagation of the image data from the low level to the high level; the error between the output value of the network and the target value is calculated; the error is back-propagated to the network, and the network is sequentially Calculate the back propagation error of each layer in the network; the network includes a fully connected layer and a convolutional layer; adjust and update all weight coefficients in the network based on the back propagation error of each layer; randomly select new image data, and re-enter the step of forward propagating the input image data through the convolutional layer, the downsampling layer, and the fully connected layer to obtain the output value, so as to obtain the output value through network forward propagation; when the error between the output value of the network and the target value is less than a first preset threshold, or the number of iterations exceeds a second preset threshold, end the training; save the network parameters of all trained layers, and store the trained weights, so that the target model can load the pre-trained weights, and classify the image or extract image features; A fully connected layer is added after the embedding layer to obtain the fully connected layer features, softmax normalization is performed based on the fully connected layer features, and the loss is calculated using the cross entropy loss function. The loss The superscript 1 represents the first branch; Obtaining an embedded feature e1 corresponding to a convolutional neural network Net1 and an embedded feature e2 corresponding to a convolutional neural network Net2; wherein the embedded feature e1 and the embedded feature e2 have similarity and diversity and are functionally identical; and the Net1 and the Net2 have different network structures and initialization weight coefficients; The embedded feature e1 is used to learn from the embedded feature e2, and the embedded feature e2 is used to learn from the embedded feature e1 to obtain a feature similarity loss function L m : ; Where n is the nth input sample, u and v are the uth network and the vth network respectively; Traverse the embedding layer output results of all networks in pairs to obtain corresponding non-repeated traversal combinations, and calculate the feature similarity loss function Le(u,v) for each traversal combination; Find the class center of all the samples, and Input each network in turn to obtain the embedding features of all samples and ;in and The superscripts 1 and 2 represent different branches, and the subscript N represents a total of N samples; For each network output, the class center of the sample is calculated separately. When there are C categories, the class center is calculated using the formula: ; in, Characterize the class center of the cth class of the embedding layer features of the first network; use The representation has a total of C class centers; For multiple networks, calculate the class center of each network embedding layer separately; Characterize the embedding features of the nth sample of the first network, the category is the cth category; Each sample learns from the embedding layer class center corresponding to its sample category, and obtains the feature similarity loss function: ; in, Characterization samples The corresponding class center; Traverse the features of each network in turn and calculate the class center loss function of each network and ; By formula Dynamically update the class center of each sample class; The superscript u of represents the u-th network; the updating method of the class center includes a first-in-first-out stack method, and the class center of the n-step sample closest to the current step is determined as the updated class center; Or, select the class center according to the classification probability corresponding to the embedding feature, so as to enter the class center calculation when the feature classification is correct; Construct a loss function that constrains the class center distance: ; in, Characterizes the class center of the i-th class of the u-th network, Characterization distance nearest class center; The class center loss is optimized by using the difficult sample mining method, wherein the difficult sample mining is to find the minimum inter-class difference of all classes.
4. An electronic device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the pedestrian re-identification method as claimed in claim 1 or 2 when calling the computer program in the memory.
5. A storage medium, characterized in that: The storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by the processor, the steps of the pedestrian re-identification method as claimed in claim 1 or 2 are implemented.
Citation Information
Patent Citations
Pedestrian re-identification method, device and equipment and storage medium
CN109711316A
Model training method and device for pedestrian re-identification, model identification method and device for pedestrian re-identification and medium
CN111027442A
Pedestrian re-identification method based on knowledge distillation
CN112560631A
Image recognition model training method and device, electronic equipment and readable storage medium
CN113111968A
Pedestrian re-identification method, apparatus and device, and readable storage medium
CN113191338A
