Pedestrian re-identification method and device based on cross-scene continuous learning based on consistency learning

Abstract

The present invention discloses a pedestrian re-identification method and device for cross-scene continuous learning based on consistency learning, and the method includes: obtaining the original features of each batch of data; inputting the original features into a pseudo-task data conversion module, respectively obtaining the old scene features and new scene features corresponding to each original feature after scene style conversion; calculating the domain and inter-domain cross-scene consistency loss function on the old scene features, and performing pseudo-task identity discrimination learning; calculating the pairwise similarity of the old scene feature samples and the new scene feature samples, and performing pseudo-task knowledge distillation; calculating the cross entropy loss function after inputting the new scene features into the classifier of the new scene, and performing identity discrimination learning; calculating the distance between the old scene features and the new scene features corresponding to each sample, and performing cross-scene consistency learning. The present invention can deploy a continuously updated and iterated model in multiple scenes that are learned successively, thereby achieving the purpose of reducing the cost of training models and manual maintenance.

Description

Technical Field

[0001] The present invention relates to the technical field of pedestrian re-identification, and in particular to a pedestrian re-identification method and device based on cross-scene continuous learning based on consistency learning. Background Art

[0002] Most of the existing pedestrian re-identification methods are based on deep convolutional neural network (CNN) models. Training CNN models usually requires pre-collecting video data from multiple different cameras in the monitoring area of ​​a specific scene in a specific camera monitoring scene, performing pedestrian detection to obtain pedestrian images and manually annotating the images across cameras. The annotated training data is used to perform supervised pre-training of the model with the stochastic gradient descent (SGD) method, and then the model is deployed to the scene. Since the shooting conditions of different video surveillance systems are usually quite different, when the model trained in a certain scene is directly migrated to a new scene for deployment, the recognition accuracy usually drops significantly. Existing solutions generally require re-collecting and annotating data in the new scene to fine-tune the model so that the model can adapt to the new scene. However, after fine-tuning the model, the recognition accuracy in old scenarios will be significantly reduced due to the use of data inconsistent with the distribution of old scenarios for incremental training. This phenomenon is usually called the "catastrophic forgetting" phenomenon of deep convolutional neural networks. This phenomenon makes it difficult for us to obtain a model with relatively high recognition accuracy in multiple scenarios at the same time, and retaining an independent model parameter for each scene requires relatively high maintenance costs.

[0003] The GwFReID algorithm proposed by Wu and Gong in 2021 is a continuous learning algorithm designed for the characteristics of pedestrian re-identification. Based on the sample playback strategy of the old data set, GwFReID improves the knowledge distillation learning in LwF and proposes classification consistency learning, distribution consistency learning and representation consistency learning, that is, the cross entropy loss function, feature contrast loss function and triple loss function are respectively used with knowledge distillation technology, which can achieve better pedestrian re-identification performance in the scenario of continuous learning.

[0004] The LwF and GwFReID algorithms can solve the problem of high data storage overhead to a certain extent. However, since they only save a small number of samples from the old data set, only a small amount of data comes from the old scene when training mixed data of the new scene. The data ratio is not balanced, which can easily lead to two phenomena: (1) The model is prone to overfitting on the training data of the old scene, resulting in poor generalization performance in the old scene; (2) The model is more likely to be biased towards the new scene, and the performance in the old scene decreases, resulting in an imbalance in the recognition accuracy of the new and old scenes. From the experimental results in the literature, the recognition accuracy of the LwF and GwFReID models is still far from the ideal multi-task joint training. Summary of the invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a pedestrian re-identification method and device based on consistent learning and continuous learning across scenarios. The pedestrian re-identification technology of the present invention can reduce data storage costs, shorten model training time, and improve the generalization ability of the model (i.e., recognition accuracy across scenarios) during continuous learning across scenarios. It can then deploy our continuously updated and iterated model in multiple scenarios that are learned successively, without the need to train a separate model for each scenario, thereby reducing the cost of training models and manual maintenance.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] On one hand, the present invention provides a pedestrian re-identification method based on cross-scene continuous learning based on consistency learning, comprising the following steps:

[0008] The old scene data and the new scene data are mixed, randomly sampled, and input into the pre-established old scene model and the new scene model in batches in parallel, and the original features of each batch of data are obtained at the same time;

[0009] The original features are input into the pseudo-task data conversion module, and the old scene features and the new scene features corresponding to each original feature after the scene style conversion are obtained respectively; the pseudo-task data conversion module transforms a pedestrian image from scene (s) to the distribution of scene (t), while the identity remains unchanged, so as to obtain the style training data of the same pedestrian in different scenes;

[0010] Calculating intra-domain and inter-domain cross-scene consistency loss functions on old scene features to perform pseudo-task identity discriminative learning; the cross-scene consistency loss function is used to minimize the inconsistency of the same input sample under different pseudo-task feature transformations;

[0011] Calculate the pairwise similarity between old scene feature samples and new scene feature samples to perform pseudo-task knowledge distillation;

[0012] After inputting the new scene features into the new scene classifier, the cross entropy loss function is calculated to perform identity discriminative learning;

[0013] Calculate the distance between the old scene features and the new scene features corresponding to each sample, and perform cross-scene consistency learning.

[0014] As a preferred technical solution, the original features include: image features extracted from data collected in old scenes through a deep convolutional neural network; and image features extracted from data collected in new scenes through a deep convolutional neural network.

[0015] As a preferred technical solution, the pseudo-task data conversion module performs scene conversion specifically as follows:

[0016] For a dataset collected in a specific scenario, each feature channel in the batch normalization layer corresponding to each convolution kernel will have two statistical statistics, the mean μ (s) and variance σ (s) , the calculation formula is as follows:

[0017]

[0018] in, represents the sample feature vector of the i-th sample extracted by CNN, N represents the number of samples in a small batch for training, and (s) represents the number of the current scene;

[0019] The batch normalization layer calculates the mean and variance by sliding average and updates the learnable scaling factor γ and offset factor β by back propagation algorithm, mapping the output of all channels of each layer of the neural network model to a Gaussian distribution. The parameters of the batch normalization layer reflect the overall distribution characteristics of samples of a data set on each feature channel.

[0020] For the sample feature f numbered i from a certain scene(s), i (s) , use the task-specific batch normalization statistics of another scene (t) to convert it to the style of scene (t) and obtain the converted feature f i (s,t) , as shown in the following formula:

[0021]

[0022] Among them, γ and β represent the scaling factor and offset coefficient of the batch normalization layer, μ (t) and σ (t)They represent the mean statistic and variance statistic of the task-specific batch normalization layer of the scene (t) respectively; when learning different scenes, different batch normalization layers use shared scaling factors γ and offset factors β to maintain continuity in cross-domain learning.

[0023] As a preferred technical solution, the cross-scenario consistency loss function is expressed as follows:

[0024]

[0025] Where D is the cosine distance metric function, f i (s) is the original sample feature numbered i from scene(s), N represents the number of samples for training a small batch, P i represents the set of images in the dataset that have the same identity label as the currently selected sample, S represents the number of scenes that have been learned, and Stop-grad represents the gradient stop return flag. The purpose of this loss function is to make all samples with the same identity label as the current sample pass through the pseudo-task data conversion module to obtain the features and the original feature f i (s) As close as possible; use the gradient stop module stop_grad to prevent the model from updating the parameters of the old batch normalization layer when learning new data, and adapt the features to the new scene by updating the parameters of the network convolution layer.

[0026] As a preferred technical solution, the similarity calculation method is as follows:

[0027] In each small batch during the training process, the high-dimensional feature vector after global pooling of the feature extraction network is used as the sample feature, and the pairwise similarity of all samples in the small batch is calculated to obtain the similarity matrix. The difference between this matrix and the matrix of the similarity output of the student model is used as the optimization target, so that the similarity matrix information extracted by the student model is close to that of the teacher model, thereby retaining the discrimination ability in the old scene.

[0028] As a preferred technical solution, the specific calculation formula of the similarity matrix M is as follows:

[0029] M=[f1,f2,…,f N ] T [f1,f2,…,f N ]∈R d×N

[0030] Among them, M is a feature similarity matrix of a small batch of samples with a sample size of N and a number of feature channels of d, f1,f2,…,f N They are sample features numbered from 1 to N, and T represents the transpose operation of the matrix;

[0031] When training with data from new scene(s), the expression for pseudo-task knowledge distillation learning is:

[0032]

[0033] Among them, M (s-1,i) The similarity matrix of the features calculated by the model trained in the first (s-1) scenes after the task data conversion module, M (s,j) The similarity matrix of the features calculated by the model trained in the first (s) scenes after the task data conversion module, S is the number of scenes that have been learned so far.

[0034] As a preferred technical solution, the identity-discriminating learning is specifically as follows:

[0035] For the traditional triplet loss function, all samples in each sample triplet are sampled from the same dataset, including anchor samples, positive samples and negative samples, which is called in-domain identity discriminative learning, as shown in the following formula:

[0036]

[0037] in, are the features of anchor samples, positive samples, and negative samples, respectively. N represents the number of samples for training a small batch, S represents the number of scenes currently learned, D is the cosine distance metric function, and m is the threshold of the triplet loss function.

[0038] The pseudo-scene data conversion module is used to improve the triplet loss function. When constructing a sample triplet, a sample is first randomly sampled in the data set A as an anchor sample, and then a positive sample and a negative sample are sampled according to the label of the anchor sample. The positive sample is a sample with the same label as the anchor sample, and the negative sample is a sample with a different label from the anchor sample. The pseudo-scene data conversion module is used to convert the positive sample and the negative sample respectively, so that the positive sample and the negative sample look more like samples with the same and different identities as the anchor sample collected from another scene. In this way, a cross-domain sample triplet is constructed, and the triplet loss function is used to train the model to improve the cross-domain generalization ability of the model. This optimization goal is called inter-domain identity discrimination learning, as shown in the following formula:

[0039]

[0040] Among them, f i (s) is the original, It is the feature obtained after the positive and negative samples pass through the data conversion module, N represents the number of samples for training a small batch, S represents the number of scenes currently learned, D is the cosine distance metric function, and m is the threshold of the triple loss function;

[0041] Pseudo-task identity discriminative learning is composed of the above-mentioned intra-domain identity discriminative learning and inter-domain identity discriminative learning. The specific formula is as follows:

[0042] L PT-ID =L Intra-PT-ID +L Inter-PT-ID .

[0043] On the other hand, the present invention provides a pedestrian re-identification system based on cross-scene continuous learning based on consistency learning, which is applied to the pedestrian re-identification method based on cross-scene continuous learning based on consistency learning, and is characterized by comprising a data mixing module, a pseudo-task data conversion module, a pseudo-task identity discriminative learning module, a pseudo-task knowledge distillation module, an identity discriminative learning module and a consistency learning module;

[0044] The data mixing module is used to mix the old scene data and the new scene data, and input the data into the pre-established old scene model and the new scene model in batches in parallel after random sampling, and obtain the original features of each batch of data at the same time;

[0045] The pseudo-task data conversion module is used to obtain the old scene features and the new scene features corresponding to each original feature after the scene style conversion; the pseudo-task data conversion module transforms a pedestrian image from scene (s) to the distribution of scene (t), while the identity remains unchanged, so as to obtain the style training data of the same pedestrian in different scenes;

[0046] The pseudo-task identity discriminability learning module is used to calculate the intra-domain and inter-domain cross-scene consistency loss function on the old scene features to perform pseudo-task identity discriminability learning; the cross-scene consistency loss function is used to minimize the inconsistency of the same input sample under different pseudo-task feature transformations;

[0047] The pseudo-task knowledge distillation module is used to calculate the pairwise similarity between the old scene feature samples and the new scene feature samples to perform pseudo-task knowledge distillation;

[0048] The identity discriminability learning module is used to input the new scene features into the classifier of the new scene and then calculate the cross entropy loss function to perform identity discriminability learning;

[0049] The consistency learning module is used to calculate the distance between the old scene features and the new scene features corresponding to each sample, and perform cross-scene consistency learning.

[0050] Another aspect of the present invention provides an electronic device, the electronic device comprising:

[0051] at least one processor; and,

[0052] a memory communicatively connected to the at least one processor; wherein,

[0053] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the pedestrian re-identification method based on consistency learning and cross-scene continuous learning.

[0054] On the other hand, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the pedestrian re-identification method based on consistency learning and cross-scene continuous learning.

[0055] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0056] 1. The present invention only needs to retain a small amount of training samples from the old data set to achieve good generalization performance in both new and old scenes. This is because the pseudo-scene data conversion and pseudo-scene knowledge distillation learning are used to convert the styles of samples from different data sets and enable the model to retain the recognition accuracy in the old scene.

[0057] 2. The present invention does not need to save the classifier parameters and identity labels of the data in the old scene during the model training stage. Knowledge distillation based on the similarity matrix can retain the recognition performance of the model in the old scene in an unsupervised learning manner.

[0058] 3. The present invention does not need to retain multiple model parameters during the model deployment stage. Only a single model can be used to achieve high recognition accuracy in multiple scenarios at the same time, significantly reducing the model maintenance cost.

[0059] 4. The present invention has better cross-scenario generalization performance. Through cross-scenario consistency learning and pseudo-task identity discriminability learning, the model trained by the method we invented can learn features that are invariant across scenarios, and the generalization performance of the model is stronger. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0061] Figure 1 This is the inference flow chart for the current person re-ID deployment;

[0062] Figure 2 It is a flowchart of a pedestrian re-identification method based on cross-scene continuous learning based on consistency learning provided by an embodiment of the present invention;

[0063] Figure 3 is a schematic diagram of a continuous person re-identification training process based on pseudo-scene consistency learning according to an embodiment of the present invention;

[0064] Figure 4 is a schematic diagram of pseudo-task knowledge distillation learning based on feature similarity matrix in an embodiment of the present invention;

[0065] FIG5( a ) is a flow chart of intra-domain identity discriminability learning based on a triplet loss function according to an embodiment of the present invention;

[0066] FIG5( b ) is a flow chart of inter-domain identity discriminability learning based on a triplet loss function according to an embodiment of the present invention;

[0067] Figure 6 A schematic diagram of the structure of a pedestrian re-identification system with cross-scenario continuous learning based on consistency learning provided by an embodiment of the present invention;

[0068] Figure 7 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to enable those skilled in the art to better understand the present application, the technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments and drawings in the present application. It should be understood that the drawings are only used for exemplary illustration and cannot be understood as limitations on this patent. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0070] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0071] Person re-identification (ReID) is a technology that uses computer vision technology to automatically identify whether a specific pedestrian exists in an image or video sequence. Unlike face recognition, fingerprint recognition and other identity recognition technologies that require active cooperation from the subject, it can detect and identify pedestrian images / videos captured by different cameras with large differences in posture, shooting angle, lighting and other conditions. It has been widely used in the fields of intelligent video surveillance and smart security. The main problem that this invention solves is the problem of catastrophic forgetting in cross-scene continuous learning in pedestrian re-identification technology, that is, the problem that the recognition accuracy of the pedestrian re-identification model in the old scene drops significantly after learning the data of the new scene.

[0072] like Figure 1 As shown in the figure, the reasoning process of the current pedestrian re-identification deployment is as follows: video data is collected by surveillance cameras, pedestrian detection algorithms such as Faster R-CNN are used for detection, the spatial and temporal positions of pedestrians in the video are determined, and the video is frame sampled and pedestrian images are captured. Then, the pedestrian image is passed through the feature extraction network to obtain the pedestrian feature vector, and the query pedestrian feature vector and the pedestrian features of known identities in the pedestrian feature database are compared for similarity and sorted. If the similarity between the query pedestrian feature and the most similar feature in the database is higher than a certain threshold, the identity of the query pedestrian is determined to be the corresponding known identity, otherwise it is determined to be an unknown identity.

[0073] like Figure 2 As shown, the pedestrian re-identification method based on cross-scene continuous learning based on consistency learning in this embodiment specifically includes the following steps:

[0074] S1. Mix the old scene data and the new scene data, randomly sample them, and input them into the pre-established old scene model and the new scene model in batches in parallel, and obtain the original features of each batch of data.

[0075] The old scene model refers to a model that is trained only using data collected in the old scene. After adding new scene data, the parameters of the old scene model will not change.

[0076] The new model scene refers to a model that is initialized using the parameters of the old scene model and then trained by adding data collected in the new scene. The original features include: (1) image features extracted from the data collected in the old scene by a deep convolutional neural network; (2) image features extracted from the data collected in the new scene by a deep convolutional neural network.

[0077] S2. Input the original features into the pseudo-task data conversion module to obtain the old scene features and new scene features corresponding to each original feature after scene style conversion; the pseudo-task data conversion module transforms a pedestrian image from scene (s) to the distribution of scene (t), while the identity remains unchanged, thereby obtaining the style training data of the same pedestrian in different scenes.

[0078] Since most of the samples in the past data sets that the model has learned are not saved in the context of incremental learning, it is difficult for the model to directly learn the distribution of the old data sets when learning new tasks. In order to provide training data that is closer to the original distribution in the old data sets, this embodiment uses a pseudo-task data conversion module to perform style conversion on a portion of the newly collected samples, so that the distribution of these samples is closer to the old tasks, thereby achieving the effect of allowing the model to learn the feature distribution of the new and old data sets at the same time.

[0079] Furthermore, when the model learns each data set, it will train a batch normalization layer (TSBN) for each task. The batch normalization layer will use the sliding average method to calculate the mean and variance and use the back-propagation algorithm to update the learnable scaling factor γ and offset factor β, mapping the output of all channels of each layer of the neural network model to a Gaussian distribution. Since the sample feature distribution in each data set (scenario) is inconsistent, the learned statistics of the batch normalization layer for each task are also different. The parameters of the batch normalization layer reflect the overall distribution characteristics of the samples of a data set on each feature channel.

[0080] Generally speaking, in the inference phase of the model, if a test sample comes from dataset a, we will use the batch normalization layer learned on dataset a to normalize the features of the test sample x. In the pseudo-task data conversion module, we use the batch normalization layer learned on datasets other than dataset a to normalize the features of this sample. The purpose of doing this is to assume that this sample comes from other datasets and to normalize the sample in this way, thereby achieving an implicit data enhancement effect.

[0081] In the above step S2, when the pseudo task data conversion module performs data conversion, the specific process is as follows:

[0082] For each dataset collected in each scene, each feature channel in the batch normalization layer corresponding to each convolution kernel will have two statistical statistics, the mean μ (s) and variance σ (s) , the calculation formula is as follows:

[0083]

[0084] For the sample feature f numbered i from a certain scene(s), i (s) , use the task-specific batch normalization statistics of another scene (t) to convert it to the style of scene (t) and obtain the converted feature f i (s,t) , as shown in the following formula:

[0085]

[0086] Among them, γ and β represent the scaling factor and offset coefficient of the batch normalization layer, μ (t) and σ (t) Represent the mean statistic and variance statistic of the task-specific batch normalization layer of scene (t). When learning different scenes, different batch normalization layers use shared scaling factors γ and offset factors β to maintain continuity in cross-domain learning. Experiments have shown that the model trained in this way is more stable than the one using task-specific scaling factors γ and offset factors β. Through pseudo-scene data conversion, a pedestrian image from scene (s) can be transformed to the distribution of scene (t), while the identity remains unchanged, thereby obtaining training data on the style of the same pedestrian in different scenes, which helps our model learn features that are generalizable across scenes.

[0087] S3. Calculate the intra-domain and inter-domain cross-scene consistency loss function on the old scene features to perform pseudo-task identity discriminability learning; the cross-scene consistency loss function is used to minimize the inconsistency of the same input sample under different pseudo-task feature transformations.

[0088] Domain consistency learning (DCL). Based on the pseudo-task data samples generated in the previous step, it is hoped that the same sample (pedestrian) will be consistent in different data domain feature transformations, thereby reducing the differences in feature extraction of the model in different scenarios. In order to solve this problem, this embodiment proposes a cross-scene consistency loss function to minimize the inconsistency of the same input sample under different pseudo-task feature transformations, so that the features learned by the model in the process of continuous learning can be more robust to the feature transformations of different specific tasks.

[0089] Furthermore, the expression of the cross-scenario consistency loss function is as follows:

[0090]

[0091] Where D is the cosine distance metric function, f i(s) is the original sample feature numbered i from scene(s), N represents the number of samples for training a small batch, P i represents the set of images in the dataset that have the same identity label as the currently selected sample, and S represents the number of scenes that have been learned. Stop-grad represents the gradient stop return flag. The purpose of this loss function is to make all samples with the same identity label as the current sample (including the current sample itself) pass through the pseudo-task data conversion module to obtain the features and the original feature f i (s) As close as possible; use the gradient stop module stop_grad to prevent the model from updating the parameters of the old batch normalization layer when learning new data, and adapt the features to the new scene by updating the parameters of the network convolution layer.

[0092] At the same time, by performing cross-scene consistency learning on samples converted from pseudo-task data, the features extracted by the network are made invariant to transformations in different scenes, thereby improving the generalization performance of the model across scenes, so that after learning new scenes, the model still has a high recognition accuracy in old scenes.

[0093] S4. Calculate the pairwise similarity between the old scene feature samples and the new scene feature samples to perform pseudo-task knowledge distillation.

[0094] Pseudo-task knowledge distillation (PT-KD) learning. In the case of continuous learning with limited storage of old samples, it is inevitable that the network model will forget the old knowledge to a certain extent due to the inherent differences in data distribution in different scenarios. In order to reduce the performance degradation of the network model in the old scene after learning the knowledge in the new scene, a similarity-based knowledge distillation method is proposed. Knowledge distillation is a teacher-student model training paradigm. In the process of continuous learning, the model trained in the old scene is used as the teacher model, and the model fine-tuned in the new scene is used as the student model. While the model learns new knowledge, it imitates the output of the teacher model to maintain the recognition ability of the old scene. Typical knowledge distillation training is based on classifiers. In the pedestrian re-identification task, since the identities of pedestrians in different scenes (i.e., the categories of the images) are completely different, and the categories corresponding to the classifiers are also inconsistent, it is difficult for us to directly perform knowledge distillation based on the output of the classifier. In order to solve this problem, in each small batch in the training process, this embodiment uses the high-dimensional feature vector after global pooling of the feature extraction network as the sample feature, calculates the pairwise similarity of all samples in the small batch, obtains the similarity matrix, and takes the difference between this matrix and the matrix of the similarity output of the student model as the optimization target, so that the similarity matrix information extracted by the student model is close to that of the teacher model, thereby retaining the discrimination ability in the old scene.

[0095] Furthermore, the specific calculation formula for calculating the similarity matrix M is as follows:

[0096] M=[f1,f2,…,f N ] T [f1,f2,…,f N ]∈R d×N

[0097] In the above formula, M is a feature similarity matrix of a small batch of samples with a sample size of N and a number of feature channels of d, f1,f2,…,f N They are sample features numbered from 1 to N, and T represents the transpose operation of the matrix;

[0098] When training with data from new scene(s), the expression for pseudo-task knowledge distillation learning is:

[0099]

[0100] Among them, M (s-1,i) The similarity matrix of the features calculated by the model trained in the first (s-1) scenes after the task data conversion module, M (s,j)The similarity matrix of the features calculated by the model trained in the first (s) scenes after passing through the task data conversion module. S is the number of scenes that have been learned so far.

[0101] Furthermore, when performing pedestrian image retrieval, it is calculated that the features learned by the model are expected to be discriminative, that is, when calculating feature similarity, the similarity of pedestrian images of the same identity is higher, and the similarity of pedestrian images of different identities is lower. A model trained with data collected and labeled in a specific scene has a strong discrimination ability in that scene, so the sample similarity information of the scene extracted using this model is what we hope to retain. This application hopes to retain the discriminative ability of the model in the old scene while updating the iterative model in the new scene, so a knowledge distillation method based on sample similarity is proposed. The old data uses common image data augmentation methods such as random inversion, random erasure, and random hue perturbation to expand the data volume, and the new data uses the batch normalization layer transformation mentioned above to achieve data transformation, such as Figure 4 shown.

[0102] S5. After inputting the new scene features into the classifier of the new scene, the cross entropy loss function is calculated to perform identity discriminative learning.

[0103] Pseudo task identity discriminative learning (PT-ID): In order to enhance the cross-domain generalization ability of the person re-identification model in different scenarios, this embodiment combines the pseudo-scene data conversion module and improves the sample triplet construction method of the triplet loss function commonly used in the training of the person re-identification model, which we call inter-domain identity discriminative learning.

[0104] Traditional triplet loss function: (in-domain identity discriminative learning). This application hopes that the feature distance of sample pairs with different identities is greater than the feature distance of sample pairs with the same identity. The optimization goal of the triplet loss function is to make the difference between the feature distance of sample pairs with different identities (i.e., anchor samples and negative samples) and the feature distance of sample pairs with the same identity (i.e., anchor samples and positive samples) greater than a threshold, so as to improve the model's ability to discriminate identities in pedestrian datasets. Usually for the traditional triplet loss function, all samples in each sample triplet (including anchor samples, positive samples, and negative samples) are sampled from the same dataset, which is called in-domain identity discriminative learning, as shown in Figure 5(a), as shown in the following formula:

[0105]

[0106] in are the features of anchor samples, positive samples, and negative samples, respectively. N represents the number of samples for training a small batch, S represents the number of scenes currently learned, D is the cosine distance metric function, and m is the threshold of the triplet loss function.

[0107] The pseudo-scene data conversion module is used to improve the triplet loss function. When constructing a sample triplet, a sample is first randomly sampled in the data set A as the anchor sample, and then a positive sample (a sample with the same label as the anchor sample) and a negative sample (a sample with a different label from the anchor sample) are sampled according to the label of the anchor sample. The pseudo-scene data conversion module is used to convert the positive sample and the negative sample respectively, so that the positive sample and the negative sample look more like samples with the same and different identities as the anchor sample collected from another scene. In this way, a cross-domain sample triplet is constructed, and the triplet loss function is used to train the model to improve the cross-domain generalization ability of the model. This optimization goal is called inter-domain identity discriminability learning, as shown in Figure 5(b). As shown in the following formula

[0108]

[0109] where f i (s) is the original, It is the feature obtained after the positive and negative samples pass through the data conversion module, N represents the sample size of a small batch for training, S represents the number of scenes currently learned, D is the cosine distance metric function, and m is the threshold of the triplet loss function.

[0110] Pseudo-task identity discriminative learning is composed of the above-mentioned intra-domain identity discriminative learning and inter-domain identity discriminative learning. The specific formula is as follows:

[0111] L PT-ID =L Intra-PT-ID +L Inter-PT-ID

[0112] S6. Calculate the distance between the old scene features and the new scene features corresponding to each sample, and perform cross-scene consistency learning.

[0113] Based on the above processing, the overall optimization goal of this embodiment consists of four parts: cross entropy loss function, cross-scene consistency loss function, pseudo-scene knowledge distillation loss function and cross-scene identity discriminability learning (triplet loss function). Their functions are: (1) Cross entropy loss function: improve the model's ability to use features for classification. (2) Cross-scene consistency loss function: improve the consistency of the model's features under the transformation of different scenes. (3) Pseudo-scene knowledge distillation: reduce the model's knowledge forgetting of discriminative features in old scenes. (4) Pseudo-task identity discriminability learning: improve the model's cross-scene generalization ability.

[0114] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0115] Based on the same idea as the pedestrian re-identification method based on consistency learning and continuous learning across scenes in the above-mentioned embodiment, the present invention also provides a pedestrian re-identification system based on consistency learning and continuous learning across scenes, which can be used to execute the above-mentioned pedestrian re-identification method based on consistency learning and continuous learning across scenes. For ease of explanation, the structural diagram of the embodiment of the pedestrian re-identification system based on consistency learning and continuous learning across scenes only shows the parts related to the embodiment of the present invention. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.

[0116] like Figure 6 As shown, in another embodiment of the present application, a pedestrian re-identification system 100 based on consistency learning and cross-scene continuous learning is provided, including a data mixing module 101, a pseudo-task data conversion module 102, a pseudo-task identity discriminability learning module 103, a pseudo-task knowledge distillation module 104, an identity discriminability learning module 105 and a consistency learning module 106.

[0117] The data mixing module 101 is used to mix the old scene data and the new scene data, and input the data into the pre-established old scene model and the new scene model in batches in parallel after random sampling, and obtain the original features of each batch of data at the same time;

[0118] The pseudo-task data conversion module 102 is used to obtain the old scene features and the new scene features corresponding to each original feature after the scene style conversion; the pseudo-task data conversion module transforms a pedestrian image from scene (s) to the distribution of scene (t), while the identity remains unchanged, so as to obtain the style training data of the same pedestrian in different scenes;

[0119] The pseudo-task identity discriminability learning module 103 is used to calculate the intra-domain and inter-domain cross-scene consistency loss function on the old scene features to perform pseudo-task identity discriminability learning; the cross-scene consistency loss function is used to minimize the inconsistency of the same input sample under different pseudo-task feature transformations;

[0120] The pseudo-task knowledge distillation module 104 is used to calculate the pairwise similarity between the old scene feature samples and the new scene feature samples to perform pseudo-task knowledge distillation;

[0121] The identity discriminability learning module 105 is used to input the new scene features into the classifier of the new scene and then calculate the cross entropy loss function to perform identity discriminability learning;

[0122] The consistency learning module 106 is used to calculate the distance between the old scene feature and the new scene feature corresponding to each sample, and perform cross-scene consistency learning.

[0123] It should be noted that the pedestrian re-identification system based on consistency learning and cross-scene continuous learning of the present invention corresponds one-to-one to the pedestrian re-identification method based on consistency learning and cross-scene continuous learning of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the pedestrian re-identification method based on consistency learning and cross-scene continuous learning are applicable to the embodiment of pedestrian re-identification based on consistency learning and cross-scene continuous learning. For specific contents, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here. This is hereby declared.

[0124] In addition, in the implementation of the pedestrian re-identification system based on consistency learning and continuous learning across scenes in the above-mentioned embodiment, the logical division of each program module is only an example. In actual applications, the above-mentioned functions can be assigned to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the pedestrian re-identification system based on consistency learning and continuous learning across scenes is divided into different program modules to complete all or part of the functions described above.

[0125] like Figure 7 As shown, in one embodiment, an electronic device is provided for implementing a pedestrian re-identification method based on consistency learning and continuous learning across scenarios. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a pedestrian re-identification program 203.

[0126] The first memory 202 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the first memory 202 may be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 may also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 200. Further, the first memory 202 may also include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can not only be used to store application software and various types of data installed in the electronic device 200, such as the code of the pedestrian re-identification program 203, but also be used to temporarily store data that has been output or is to be output.

[0127] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The first processor 201 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules (such as federated learning defense programs, etc.) stored in the first memory 202, and calls data stored in the first memory 202 to execute various functions of the electronic device 200 and process data.

[0128] Figure 7 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on the electronic device 200 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0129] The pedestrian re-identification program 203 stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions, and when executed in the first processor 201, it can achieve:

[0130] The old scene data and the new scene data are mixed, randomly sampled, and input into the pre-established old scene model and the new scene model in batches in parallel, and the original features of each batch of data are obtained at the same time;

[0131] The original features are input into the pseudo-task data conversion module, and the old scene features and the new scene features corresponding to each original feature after the scene style conversion are obtained respectively; the pseudo-task data conversion module transforms a pedestrian image from scene (s) to the distribution of scene (t), while the identity remains unchanged, so as to obtain the style training data of the same pedestrian in different scenes;

[0132] Calculating intra-domain and inter-domain cross-scene consistency loss functions on old scene features to perform pseudo-task identity discriminative learning; the cross-scene consistency loss function is used to minimize the inconsistency of the same input sample under different pseudo-task feature transformations;

[0133] Calculate the pairwise similarity between old scene feature samples and new scene feature samples to perform pseudo-task knowledge distillation;

[0134] After inputting the new scene features into the new scene classifier, the cross entropy loss function is calculated to perform identity discriminative learning;

[0135] Calculate the distance between the old scene features and the new scene features corresponding to each sample, and perform cross-scene consistency learning.

[0136] Furthermore, if the module / unit integrated in the electronic device 200 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0137] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0138] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A cross-scenario continuous learning method based on consistency learning, characterized by: The steps include: The old scene data and the new scene data are mixed, randomly sampled, and input into the pre-established old scene model and the new scene model in batches in parallel, and the original features of each batch of data are obtained at the same time; The original features are input into the pseudo-task data conversion module to obtain the old scene features and new scene features corresponding to each original feature after scene style conversion; The pseudo-task data conversion module transforms a pedestrian image from scene (s) to the distribution of scene (t), while the identity remains unchanged, so as to obtain the style training data of the same pedestrian in different scenes; Calculating intra-domain and inter-domain cross-scene consistency loss functions on old scene features to perform pseudo-task identity discriminative learning; the cross-scene consistency loss function is used to minimize the inconsistency of the same input sample under different pseudo-task feature transformations; Calculate the pairwise similarity between the old scene feature samples and the new scene feature samples to perform pseudo-task knowledge distillation; After inputting the new scene features into the new scene classifier, the cross entropy loss function is calculated to perform identity discriminative learning; Calculate the distance between the old scene features and the new scene features corresponding to each sample, and perform cross-scene consistency learning.

2. The cross-scenario continuous learning method based on consistency learning according to claim 1 is characterized in that: The original features include: image features extracted from data collected in the old scene through a deep convolutional neural network; and image features extracted from data collected in the new scene through a deep convolutional neural network.

3. The cross-scenario continuous learning method based on consistency learning according to claim 1 is characterized in that: The pseudo-task data conversion module performs scene conversion specifically as follows: For a dataset collected in a specific scenario, each feature channel in the batch normalization layer corresponding to each convolution kernel will have two statistical statistics, the mean μ (s) and variance σ (s) , the calculation formula is as follows: Among them, f i (s) represents the i-th sample feature vector extracted by CNN, N represents the number of samples in a small batch for training, and (s) represents the number of the current scene; The batch normalization layer calculates the mean and variance by sliding average and updates the learnable scaling factor γ and offset factor β by back propagation algorithm, mapping the output of all channels of each layer of the neural network model to a Gaussian distribution. The parameters of the batch normalization layer reflect the overall distribution characteristics of samples of a data set on each feature channel. For the sample feature f numbered i from a certain scene(s), i (s) , use the task-specific batch normalization statistics of another scene (t) to convert it to the style of scene (t) and obtain the converted feature f i (s,t) , as shown in the following formula: Among them, γ and β represent the scaling factor and offset coefficient of the batch normalization layer, μ (t) and σ (t) They represent the mean statistic and variance statistic of the task-specific batch normalization layer of the scene (t) respectively; when learning different scenes, different batch normalization layers use shared scaling factors γ and offset factors β to maintain continuity in cross-domain learning.

4. The cross-scenario continuous learning method based on consistency learning according to claim 1 is characterized in that: The cross-scenario consistency loss function is expressed as follows: Where D is the cosine distance metric function, f i (s) is the original sample feature numbered i from scene(s), N represents the number of samples for training a small batch, P i represents the set of images in the dataset with the same identity label as the currently selected sample, S represents the number of scenes currently learned, and Stop-grad represents the gradient stop return flag; the role of this loss function is to make all samples with the same identity label as the current sample pass through the pseudo-task data conversion module to obtain the features and the original feature f i (s) As close as possible; use the gradient stop module stop_grad to prevent the model from updating the parameters of the old batch normalization layer when learning new data, and adapt the features to the new scene by updating the parameters of the network convolution layer.

5. The cross-scenario continuous learning method based on consistency learning according to claim 1 is characterized in that: The similarity is calculated as follows: In each small batch during the training process, the high-dimensional feature vector after global pooling of the feature extraction network is used as the sample feature, and the pairwise similarity of all samples in the small batch is calculated to obtain the similarity matrix. The difference between this matrix and the matrix of the similarity output of the student model is used as the optimization target, so that the similarity matrix information extracted by the student model is close to that of the teacher model, thereby retaining the discrimination ability in the old scene.

6. The cross-scenario continuous learning method based on consistency learning according to claim 5 is characterized in that: The specific calculation formula of the similarity matrix M is as follows: M=[f1,f2,…,f N ] T [f1,f2,…,f N ]∈R d×N Among them, M is a feature similarity matrix of a small batch of samples with a sample size of N and a number of feature channels of d, f1,f2,…,f N They are sample features numbered from 1 to N, and T represents the transpose operation of the matrix; When training with data from new scene(s), the expression for pseudo-task knowledge distillation learning is: Among them, M (s-1,i) The similarity matrix of the features calculated by the model trained in the first (s-1) scenes after the task data conversion module, M (s,j) The similarity matrix of the features calculated by the model trained in the first (s) scenes after the task data conversion module, S is the number of scenes that have been learned so far.

7. The cross-scenario continuous learning method based on consistency learning according to claim 1 is characterized in that: The identity discriminative learning is specifically as follows: For the traditional triplet loss function, all samples in each sample triplet are sampled from the same dataset, including anchor samples, positive samples and negative samples, which is called in-domain identity discriminative learning, as shown in the following formula: Among them, f i (s) , are the features of anchor samples, positive samples, and negative samples, respectively. N represents the number of samples for training a small batch, S represents the number of scenes currently learned, D is the cosine distance metric function, and m is the threshold of the triplet loss function. The pseudo-scene data conversion module is used to improve the triplet loss function. When constructing a sample triplet, a sample is first randomly sampled in the data set A as an anchor sample, and then a positive sample and a negative sample are sampled according to the label of the anchor sample. The positive sample is a sample with the same label as the anchor sample, and the negative sample is a sample with a different label from the anchor sample. The pseudo-scene data conversion module is used to convert the positive sample and the negative sample respectively, so that the positive sample and the negative sample look more like samples with the same and different identities as the anchor sample collected from another scene. In this way, a cross-domain sample triplet is constructed, and the triplet loss function is used to train the model to improve the cross-domain generalization ability of the model. This optimization goal is called inter-domain identity discrimination learning, as shown in the following formula: Among them, f i (s) is the original anchor sample, It is the feature obtained after the positive and negative samples pass through the data conversion module, N represents the number of samples for training a small batch, S represents the number of scenes currently learned, D is the cosine distance metric function, and m is the threshold of the triple loss function; Pseudo-task identity discriminative learning is composed of the above-mentioned intra-domain identity discriminative learning and inter-domain identity discriminative learning. The specific formula is as follows: L PT-ID =L Intra-pT-ID +L Inter-PT-ID .

8. A cross-scenario continuous learning system based on consistency learning, characterized by: A cross-scenario continuous learning method based on consistency learning applied to any one of claims 1-7, characterized in that it includes a data mixing module, a pseudo-task data conversion module, a pseudo-task identity discriminability learning module, a pseudo-task knowledge distillation module, an identity discriminability learning module and a consistency learning module; The data mixing module is used to mix the old scene data and the new scene data, and input the data into the pre-established old scene model and the new scene model in batches in parallel after random sampling, and obtain the original features of each batch of data at the same time; The pseudo-task data conversion module is used to obtain the old scene features and the new scene features corresponding to each original feature after the scene style conversion; the pseudo-task data conversion module transforms a pedestrian image from scene (s) to the distribution of scene (t), while the identity remains unchanged, so as to obtain the style training data of the same pedestrian in different scenes; The pseudo-task identity discriminability learning module is used to calculate the intra-domain and inter-domain cross-scene consistency loss function on the old scene features to perform pseudo-task identity discriminability learning; the cross-scene consistency loss function is used to minimize the inconsistency of the same input sample under different pseudo-task feature transformations; The pseudo-task knowledge distillation module is used to calculate the pairwise similarity between the old scene feature samples and the new scene feature samples to perform pseudo-task knowledge distillation; The identity discriminability learning module is used to input the new scene features into the classifier of the new scene and then calculate the cross entropy loss function to perform identity discriminability learning; The consistency learning module is used to calculate the distance between the old scene features and the new scene features corresponding to each sample, and perform cross-scene consistency learning.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the cross-scenario continuous learning method based on consistency learning as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by the processor, the cross-scenario continuous learning method based on consistency learning described in any one of claims 1 to 7 is implemented.

Images