Mobile object re-identification, model training method, device and storage medium

Abstract

The embodiments of the present application provide a method, device and storage medium for re-identification and model training of mobile objects. When training a model for re-identifying mobile objects, there is no need to use a training set labeled with the true value of the mobile object. It is only necessary to train the neural network model end-to-end, which is more advantageous in practical applications. At the same time, the pictures in the training set are labeled with camera labels. After obtaining clustering clusters based on the characteristics of the pictures, the camera difference loss generated by the clustering operation can be calculated according to the characteristics of the pictures contained in each cluster and the camera label, and the neural network model can be optimized based on the camera difference loss. Furthermore, when training the neural network model, the impact of camera differences can be fully considered to promote the semantic consistency of the clustering clusters obtained by clustering the neural network model, optimize the recognition ability of the neural network model for the target data set, and improve the performance of the neural network model in the application scenario of re-identification of mobile objects.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device and storage medium for re-identification and model training of moving objects. Background Art

[0002] Person re-identification is a technology that uses computer vision technology to determine whether a specific pedestrian exists in an image or video sequence. Based on the person re-identification technology, when a monitored pedestrian image is given, the pedestrian image under cross-camera devices can be retrieved to make up for the visual limitations of fixed cameras.

[0003] In the prior art, there is a person re-identification method based on an unsupervised model. In this method, cluster pseudo labels of data are generated by an unsupervised clustering algorithm or a nearest neighbor association algorithm based on similarity, and the cluster pseudo labels are used as supervisory signals to train and learn the model. However, the accuracy of the cluster pseudo labels generated by the existing method is low, which is not conducive to improving the performance of the person re-identification model. Therefore, a new solution needs to be proposed. Summary of the invention

[0004] Multiple aspects of the present application provide a moving object re-identification, model training method, device and storage medium to help improve the performance of the pedestrian re-identification model.

[0005] An embodiment of the present application provides a method for re-identifying a moving object, comprising: displaying a first interface, the first interface comprising: a picture adding control; in response to a triggering operation on the picture adding control, obtaining a picture to be queried containing a moving object; inputting the picture to be queried and pictures in a picture library to be retrieved into a neural network model; the neural network model is trained based on a training set annotated with camera tags and camera difference loss generated by clustering operations on multiple pictures in the training set; obtaining at least one picture in the picture library to be retrieved output by the neural network model that matches the picture to be queried; and displaying the at least one picture in a second interface.

[0006] An embodiment of the present application provides a training method for a re-identification model of a moving object, comprising: obtaining multiple pictures containing moving objects as a training set, wherein the multiple pictures are respectively annotated with camera labels; inputting the multiple pictures into a neural network model to obtain respective features of the multiple pictures; clustering the multiple pictures according to the respective features of the multiple pictures to obtain multiple clusters; calculating a camera difference loss of the clustering operation according to the features of the pictures respectively contained in the multiple clusters and the camera labels of the pictures respectively contained in the multiple clusters; and optimizing the neural network model using the camera difference losses corresponding to the multiple clusters.

[0007] The embodiment of the present application also provides a training method for a re-identification model of a moving object, comprising: responding to a call request from a client to a specified interface, obtaining multiple pictures of the moving object contained in the interface parameters as a training set, and the multiple pictures are respectively marked with camera labels; inputting the multiple pictures into a neural network model to obtain respective features of the multiple pictures; clustering the multiple pictures according to the respective features of the multiple pictures to obtain multiple clusters; calculating the camera difference loss of the clustering operation according to the features of the pictures contained in each of the multiple clusters and the camera labels of the pictures contained in each of the multiple clusters; iteratively training the neural network model using the camera difference losses corresponding to each of the multiple clusters to obtain a re-identification model; and returning the trained re-identification model to the client.

[0008] An embodiment of the present application also provides a method for re-identifying a moving object, comprising: obtaining a to-be-queried picture containing a moving object and a to-be-retrieved picture library; inputting the to-be-queried picture and the pictures in the to-be-retrieved picture library into a neural network model to obtain at least one picture in the to-be-retrieved picture library that matches the to-be-queried picture; wherein the neural network model is trained based on a training set annotated with camera labels and a camera difference loss generated by clustering operations on multiple pictures in the training set.

[0009] An embodiment of the present application also provides an electronic device, comprising: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to: execute the steps in the method provided in the embodiment of the present application.

[0010] The present application also provides a computer-readable storage medium storing a computer program, which can implement the steps of the method provided in the present application when the computer program is executed by a processor.

[0011] In the training method of the mobile object re-identification model provided in the embodiment of the present application, there is no need to use a training set labeled with the true value of the mobile object, and only the neural network model needs to be trained end-to-end, which is more advantageous in practical applications; at the same time, the pictures in the training set are labeled with camera labels. After obtaining clustering clusters based on the characteristics of the pictures, the camera difference loss generated by the clustering operation can be calculated according to the characteristics of the pictures contained in each cluster and the camera label, and the neural network model can be optimized based on the camera difference loss. Furthermore, when training the neural network model, the impact of camera differences can be fully considered to promote the semantic consistency of the clustering clusters obtained by clustering the neural network model, optimize the recognition ability of the neural network model for the target data set, and improve the performance of the neural network model in the mobile object re-identification application scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0013] Figure 1 A schematic diagram of a flow chart of a method for training a re-identification model for a moving object provided by an exemplary embodiment of the present application;

[0014] Figure 2 A schematic diagram of camera labels, sub-cluster clusters, local pseudo labels, and indexes representing features provided by an exemplary embodiment of the present application;

[0015] Figure 3 A flowchart of a method for training a re-identification model of a moving object provided in an exemplary embodiment of the present application;

[0016] Figure 4a A schematic flow chart of a method for training a re-identification model for a moving object provided by another exemplary embodiment of the present application;

[0017] Figure 4b A schematic diagram of a flow chart of a method for re-identifying a moving object provided by an exemplary embodiment of the present application;

[0018] Figure 4c A flowchart of a method for re-identifying a moving object provided by an exemplary embodiment of the present application;

[0019] Figure 5 A flowchart of a method for re-identifying a moving object provided by another exemplary embodiment of the present application;

[0020] Figure 6 A schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0022] Person re-identification refers to the technology of matching or retrieving specific pedestrians under multiple cameras. Fully supervised person re-identification technology has been widely and fully studied in recent years, and the corresponding fully supervised model has also achieved a high accuracy rate. However, fully supervised person re-identification relies on a large amount of manual annotation operations. In order to reduce the cost of manual annotation as much as possible and realize the rapid learning and application of the re-identification model in new scenarios, the study of pure unsupervised person re-identification has important application value.

[0023] Existing unsupervised person re-identification methods can be divided into two categories: the first category is unsupervised person re-identification methods based on clustering pseudo-labels; the second category is person re-identification methods based on domain transfer learning. The following will briefly introduce each of them.

[0024] The first method generates clustering pseudo-labels for the data through an unsupervised clustering algorithm or a nearest neighbor association algorithm based on similarity, and uses the generated clustering pseudo-labels as supervisory signals to train and learn the model.

[0025] The second type of method uses a labeled source dataset to assist in training on an unlabeled target dataset; by generating images with the same style as the target dataset from the source dataset, or aligning the feature distribution of the source dataset and the target dataset, the model's recognition ability on the target dataset is improved.

[0026] Among them, the clustering-based pseudo-label generation method shows better performance than other methods. However, the existing clustering-based pseudo-label generation methods often ignore the differences in feature distribution in the same cluster. This difference in feature distribution is closely related to the difference in camera perspective. For example, the pictures of the same pedestrian under different cameras often have large differences in perspective and style, which makes it inappropriate to use a unified cluster center to represent the feature distribution of all pictures of a pedestrian. Due to the consistency of perspective under the same camera, the features of a pedestrian under the same camera are often more similar and concentrated, so that the features of the same pedestrian under different cameras show their own concentrated characteristics. That is, in the same cluster, pictures taken by different cameras show their own concentrated characteristics.

[0027] In recent years, many papers have proposed methods based on clustering to generate cluster pseudo labels to solve the problem of unsupervised person re-identification. The following will briefly introduce them.

[0028] Among them, BUC (bottom-up clustering) adopts hierarchical clustering to iteratively merge clusters and update the model, thereby gradually improving the performance of the model. This method is based on the idea of ​​hierarchical clustering. In each iteration, clusters are merged according to their similarity, and pedestrian images with consistent semantic information are merged into the same category by gradually reducing the number of merged clusters. This method has achieved good results on public pedestrian re-identification datasets such as Market-1501, DukeMTMC-reID and MARS in a purely unsupervised situation. However, the disadvantage of this method is that each iteration of merging clusters is based on the merged results of the previous cluster, which will lead to the continuous accumulation of errors, affecting the quality of the merged clusters, and thus having an adverse effect on the final model performance.

[0029] Among them, MMT (Mutual mean-teaching) adopts the method of two models learning from each other, and uses the average model to provide additional supervision information to each other's model, so that the two models promote each other and jointly improve the model performance. The main disadvantage of this method is that the training process requires the use of two models to train at the same time, which is not advantageous in terms of model complexity, and requires the use of labeled source data sets.

[0030] Among them, SpCL (Self-paced Contrastive Learning) uses the source dataset and the target dataset for joint training, and designs a unified contrastive learning framework to jointly distinguish the pedestrian categories of the source dataset, the clusters of the target dataset, and the noise samples of the target dataset. The model is trained using a unified contrast loss function. In order to improve the accuracy of clustering, SpCL also proposes a clustering reliability evaluation criterion to screen out more accurate clusters by judging the independence and compactness of clusters. This method has achieved the current best unsupervised pedestrian re-identification accuracy on the Market-1501, DukeMTMC-reID and MSMT datasets. The disadvantage is that there are many hyperparameters that need to be set in this method, which has certain limitations on the application of this method in actual scenarios.

[0031] In view of the above technical problems, a solution is provided in some embodiments of the present application. The technical solutions provided by the embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0032] Figure 1A flow chart of a method for training a re-identification model for a moving object provided by an exemplary embodiment of the present application is shown in FIG. Figure 1 As shown, the mobile object re-identification method includes:

[0033] Step 101: Acquire multiple pictures containing moving objects as a training set, wherein the multiple pictures are respectively annotated with camera labels.

[0034] Step 102: input the plurality of images into a neural network model to obtain features of each of the plurality of images.

[0035] Step 103: performing a clustering operation on the multiple images according to their respective features to obtain multiple clusters.

[0036] Step 104: Calculate the camera difference loss of the clustering operation according to the features of the pictures included in each of the plurality of clusters and the camera tags of the pictures included in each of the plurality of clusters.

[0037] Step 105: Optimize the neural network model using the camera difference losses corresponding to each of the multiple clusters.

[0038] In this embodiment, the moving object may include: pedestrians, animals, vehicles, robots and other movable objects, which are not limited in this embodiment. In the following embodiments, the moving object is implemented as a pedestrian as an example for exemplary description.

[0039] The camera tag is used to identify the camera that took the picture. It should be understood that different cameras are installed in different positions, have different shooting angles and shooting styles, and therefore the images taken of the same pedestrian have certain characteristic differences. In this embodiment, the camera tags are marked on the images of the training set, which is conducive to distinguishing the images taken by different cameras during the subsequent training process, so that the neural model can learn the differences in the shooting of the same pedestrian by different cameras.

[0040] The camera labels can be batch labeled manually or by electronic devices, or the camera can be set to automatically label the camera labels when the camera outputs the image. Labeling the camera labels on the images in the training set greatly reduces the workload of labeling and improves the efficiency of model training compared to the model training method based on full supervision.

[0041] Optionally, the neural network model in this embodiment includes a convolutional neural network (CNN) for extracting features from images in the training set to obtain the features of each image. The neural network model may be a ResNet (residual network), such as ResNet18, ResNet50 or ResNet50-IBN (where IBN stands for Intent based networking). Optionally, ResNet may be pre-trained based on ImageNet (a picture library for image / visual training).

[0042] After obtaining the features of each image in the training set, the training set can be clustered based on the features of the images. Optionally, the clustering operation can be implemented based on a variety of different clustering algorithms. For example, K-Means clustering algorithm, mean shift clustering, density-based spatial clustering algorithm (Density-Based Spatial Clustering of Applications with Noise, DBSCAN), etc. This embodiment includes but is not limited to this.

[0043] Optionally, when using DBSCAN, the distance threshold required by the clustering algorithm, the proximity range of the center point and other parameters can be set according to actual needs, and this embodiment does not limit it. In some optional embodiments, the distance threshold thresh=0.5 and the proximity range of the center point eps=2 can be set.

[0044] Based on the clustering operation, multiple images in the training set can be divided into multiple clusters. The images in the same cluster can be taken by different cameras. During the clustering operation, if a unified center is used to represent the feature distribution of all images of the same pedestrian, the clustering error caused by camera differences cannot be taken into account.

[0045] Therefore, in this embodiment, the camera difference loss of the clustering operation may be further calculated according to the features of the pictures respectively included in the multiple clusters obtained by clustering and the camera labels of the pictures respectively included in the multiple clusters.

[0046] Among them, the camera difference loss may include: the loss generated when the pictures of the same pedestrian taken by the same camera are clustered into the same clustering cluster, that is, the intra-camera loss; or, the camera difference loss may include: the loss generated when the pictures of the same pedestrian taken by different cameras are clustered into the same clustering cluster, that is, the cross-camera loss; or, the camera difference loss may include both the intra-camera loss and the cross-camera loss.

[0047] After obtaining the camera difference losses corresponding to the multiple clustering clusters, the parameters of the neural network model can be optimized based on the camera difference losses corresponding to the multiple clustering clusters, so that the neural network model can continuously learn the features of pedestrian pictures taken by different cameras and output more accurate clustering results.

[0048] In this embodiment, when training a model for re-identifying moving objects, there is no need to use a training set labeled with the true value of the moving object, and only the neural network model needs to be trained end-to-end, which is more advantageous in practical applications; at the same time, the pictures in the training set are labeled with camera labels. After obtaining clusters based on the characteristics of the pictures, the camera difference loss generated by the clustering operation can be calculated according to the characteristics of the pictures contained in each cluster and the camera label, and the neural network model can be optimized based on the camera difference loss. Furthermore, when training the neural network model, the impact of camera differences can be fully considered to promote the semantic consistency of the clusters obtained by clustering the neural network model, optimize the recognition ability of the neural network model for the target data set, and improve the performance of the neural network model in the application scenario of re-identifying moving objects.

[0049] The following embodiments will exemplarily illustrate an optional implementation of calculating the camera difference loss of the clustering operation.

[0050] In some optional embodiments, after performing a clustering operation on multiple images in a training set to obtain multiple clusters (hereinafter referred to as clusters), optionally, to facilitate subsequent queries, cluster pseudo labels can be set for the multiple clusters obtained by clustering.

[0051] For example, a cluster pseudo-label y can be assigned to each cluster, and the cluster pseudo-label y is used as the index of each cluster to facilitate subsequent calculations. Assume that, based on the clustering operation, the images in the training set are divided into Y clusters and several outliers. For the Y clusters, the values ​​of the cluster pseudo-labels y of the Y clusters can be set to 0, 1, 2, ..., Y-1 in an ascending order. Correspondingly, the images in each cluster have the same cluster pseudo-label as the cluster to which they belong. For example, if the cluster pseudo-label of the cluster to which a certain image belongs is y = 2, then the cluster pseudo-label of the image is also y = 2.

[0052] Optionally, each cluster can be further divided according to the camera tags of the pictures in the cluster. The following is an exemplary description using the first cluster among the multiple clusters obtained by clustering as an example, wherein the first cluster can be any cluster among the multiple clusters. This embodiment uses "first" to limit the cluster, which is only used for the convenience of description and distinction, and does not impose any restrictions on the order, number, etc. of the clusters.

[0053] For a first cluster among the multiple clusters, the pictures in the first cluster may be divided according to camera tags marked on the pictures in the first cluster to obtain multiple sub-clusters (hereinafter referred to as sub-clusters); wherein the multiple sub-clusters correspond to different camera tags.

[0054] That is, for a cluster, multiple pictures in the cluster are further divided according to the camera tags of each picture in the cluster to obtain multiple sub-clusters corresponding to the cluster.

[0055] The pictures in each sub-cluster have the same camera label, and each sub-cluster also has the cluster pseudo label y of the cluster to which it belongs. For example, after a cluster with a cluster pseudo label y=2 is divided into sub-cluster1, sub-cluster2, and sub-cluster3 according to the camera labels of the pictures, the pictures in sub-cluster1 have the same camera label 1, the pictures in sub-cluster2 have the same camera label 2, and the pictures in sub-cluster3 have the same camera label 3; at the same time, the cluster pseudo labels of sub-cluster1, sub-cluster2, and sub-cluster3 are all y=2.

[0056] In this embodiment, a representative feature to be learned (hereinafter referred to as proxy) is set for each sub-cluster. The representative feature proxy can be understood as the representative feature of all images in the corresponding sub-cluster, and the feature needs to be learned through continuous iteration. When the semantic consistency of the clusters obtained by the neural network model clustering is high, the pictures in each sub-cluster have a high similarity with the representative feature of the sub-cluster. Therefore, in the subsequent training process, the neural network model needs to continuously optimize the model parameters to improve the semantic consistency of the clusters.

[0057] After the first cluster is divided into multiple sub-clusters, the initial values ​​of the representative features of each of the multiple sub-clusters can be determined according to the features of the pictures contained in each of the multiple sub-clusters. The initial value of each proxy can be calculated by the average features of the pictures in the sub-cluster corresponding to the proxy. That is, the average features of the pictures in each sub-cluster are calculated, and the proxy corresponding to each sub-cluster is initialized with the calculated average features. For example, for the sub-cluster sub-cluster1 in the first cluster, the average value of the features of the pictures contained in sub-cluster1 can be calculated as the initial value of the representative features of sub-cluster1.

[0058] Optionally, in some embodiments, to facilitate subsequent queries, local pseudo labels may be set for each of the multiple sub-clusters obtained by dividing each cluster. An exemplary description will be given below.

[0059] Assume that the training set used in this embodiment contains images from C cameras in total, and assume that after further dividing the multiple clusters according to the camera labels, a total of Z c sub-cluster with camera label c, such as Figure 2 Z c sub, that is, sub1, sub2, ..., sub Zc Next, a local pseudo-label z can be assigned to each sub-cluster with camera label c. The local pseudo-label z of Zc sub-clusters can take values ​​of 1, 2, …, Z c ,like Figure 2 shown.

[0060] For each sub-cluster, each image in the sub-cluster is also assigned a local pseudo-label z of the sub-cluster. That is, each image in the sub-cluster has both a cluster pseudo-label y and a local pseudo-label z. The cluster pseudo-label y is used to distinguish the cluster to which the image belongs after feature clustering, and the local pseudo-label is used to distinguish the camera corresponding to the image in the cluster to which it belongs.

[0061] Based on the above, after setting the representative features to be learned for each sub-cluster in the training set, a query index can be set for the representative features of all sub-cluster contained in the training set according to the cluster pseudo-label and local pseudo-label corresponding to the sub-cluster. The query index can be used to quickly query the representative features corresponding to the image or sub-cluster when calculating the camera difference loss later, which will not be described here.

[0062] Alternatively, if Figure 3 As shown, in some embodiments, a proxy memory bank (representing a memory bank of features) may be constructed to store and update the features of each proxy, and the proxy memory bank may be represented by the letter K. K is a dxZ matrix, where d is the dimension of the proxy feature, and Z represents the total number of proxies, that is, the number of sub-clusters. Figure 2 As shown, each sub-cluster is associated with a proxy, and then

[0063] Among them, in the matrix dxZ, the u-th row represents the features of the u-th proxy, and each proxy has the clustering pseudo-label y and local pseudo-label z of the represented sub-cluster, that is, the query index.

[0064] Next, multiple batch training sets (hereinafter referred to as mini-batches) may be selected from the training set, each batch training set containing a first number of images.

[0065] Optionally, in some embodiments, a first number of target sub-cluster clusters may be selected from all sub-cluster clusters included in the training set, and a second number of images may be randomly sampled from each target sub-cluster cluster to obtain a batch training set.

[0066] That is, after further dividing the Y clusters according to the camera labels to obtain Z sub-clusters, S1 sub-clusters can be randomly selected from the Z sub-clusters. Next, S2 pictures are randomly selected from each selected sub-cluster to obtain S1*S2 pictures as a mini-batch.

[0067] The values ​​of S1 and S2 can be set according to actual needs. This embodiment does not limit this. In some optional embodiments, S1 can be set to 8, S2 can be set to 4, and a mini-batch containing 8*4 images is obtained.

[0068] In this embodiment, multiple batch training sets are selected from the training set, and the number of images included in the batch training set is smaller than the number of images included in the training set. Gradient descent optimization is performed based on small batches of samples. On the one hand, it can overcome the randomness of a single sample, and on the other hand, it can make up for the defect of high computational overhead of training sets with larger data volumes.

[0069] The following will take any batch training set in multiple batch training sets as an example to exemplify the optional implementation methods of calculating the cross-camera loss and the intra-camera loss.

[0070] In some embodiments, the same-camera loss may be calculated based on the features of the images in the batch training set and the representative features of the sub-cluster to which the images belong. When calculating the intra-camera loss, the intra-camera loss corresponding to each image in the batch training set may be calculated separately, and the losses calculated for each image may be accumulated to obtain the intra-camera loss corresponding to the batch training set. Based on this accumulation method, the randomness of a single image sample may be overcome and the reliability of the intra-camera loss may be improved.

[0071] Optionally, the same-camera loss can be calculated based on a contrast loss function, which can also be called the intra-camera contrast loss. The following takes any image in the training set as an example to illustrate the optional implementation method of calculating the intra-camera contrast loss. For the convenience of description, the sub-cluster to which the image belongs is described as the first sub-cluster.

[0072] Optionally, for any image in the batch training set, the representative features of the first sub-cluster to which the image belongs and the representative features of each of at least one second sub-cluster with the same camera tag as the image can be determined. When determining the representative features of the first sub-cluster, the representative features of the first cluster can be queried from the proxy query index based on the cluster pseudo-label of the cluster to which the first sub-cluster belongs and the local pseudo-label corresponding to the first sub-cluster. When determining the representative features of each of at least one second sub-cluster, the representative features of each of at least one sub-cluster with the same camera tag as the image can be queried from the proxy query index based on the camera tag of the image.

[0073] Next, the similarity between the feature of the image and the representative feature of the first sub-cluster is calculated as the first similarity; the similarity between the feature of the image and the representative feature of each of the at least one sub-cluster is calculated respectively to obtain at least one second similarity; the cumulative value of the at least one second similarity is calculated, and the intra-camera contrast loss corresponding to the image is determined according to the ratio of the first similarity to the second similarity.

[0074] Optionally, the same camera contrast loss function L Intra The calculation method of can be shown as the following formula 1:

[0075]

[0076] In formula 1, x i represents the i-th picture in the mini-batch, f(x i ) represents the image x i Features extracted by the neural network model; c i Indicates image x iThe corresponding camera labels, A represents the camera labels from the first to the (c i -1) The total number of proxies under the camera, Z c Indicates the number of proxies under the c-th camera. Indicates the c i The index range of the proxy under each camera. i Indicates image x i The local pseudo-label of the image x i The corresponding proxy index is: j = A + z i When the proxy in the proxy memory bank is represented by matrix K, K[j] represents the jth column in matrix K, i.e., image x i The proxy feature of the sub-cluster where k is located. i The proxy index under each camera, k ranges from A+1 to

[0077] Among them, N c represents the number of images taken by the cth camera, D c represents the sub-training set corresponding to the image from camera c in the dataset; τ is the scale transformation factor, and the value of τ is 0.07 during actual network training. exp(x) = e x represents the exponential function and log(x) represents the logarithmic function.

[0078] Based on the above, we can know that the numerator exp(K[j] T f(x i ) / τ), used to calculate the image x i The similarity between the feature of and its corresponding proxy feature. That is, the image x i The first similarity with the representative feature of the first sub-cluster to which it belongs. Denominator Used to calculate the image x i Features and pictures x i The accumulated value of the similarities of the representative features of the sub-clusters with the same camera tag is the accumulated value of the at least one second similarity.

[0079] In formula 1, for each image in the mini-batch, it is only compared with the proxy with the same camera label, thereby realizing the calculation of the contrast loss within the same camera.

[0080] Based on the above formula 1, after obtaining clusters based on feature clustering obtained by neural network recognition, the contrast loss of images with the same camera label and images with different camera labels in each cluster can be further calculated. In the subsequent training process, the contrast loss within the same camera can be optimized to reduce the intra-class differences caused by camera differences in the clusters obtained by clustering, thereby improving the performance of the neural network model.

[0081] In other embodiments, the cross-camera contrast loss can be calculated based on the features of the images in the batch training set and the representative features of each of the multiple sub-cluster clusters contained in the cluster cluster to which the images belong. When calculating the cross-camera loss, the cross-camera loss corresponding to each image in the batch training set can be calculated separately, and the loss calculated for each image can be accumulated to obtain the cross-camera loss corresponding to the batch training set. Based on this accumulation method, the randomness of a single image sample can be overcome and the reliability of the cross-camera loss can be improved. Optionally, the cross-camera loss can be calculated based on the contrast loss function, which can be called the cross-camera contrast loss.

[0082] The following takes any image in the training set as an example to illustrate the optional implementation method of calculating the cross-camera contrast loss.

[0083] For any image in the batch training set, representative features of each of the multiple sub-clusters contained in the cluster to which the image belongs can be determined as the first feature set. At the same time, representative features of the sub-clusters of the cluster that does not contain the image are selected from the sub-clusters contained in each of the multiple clusters as the second feature set.

[0084] Continue with picture x i For example, first, we can determine the i The representative features of all sub-clusters with the same cluster label, that is, image x i The proxy of the sub-cluster contained in the cluster obtains the first feature set P.

[0085] Optionally, the second feature set may be representative features corresponding to hard negative examples obtained by hard negative mining. The following is an exemplary description of how to obtain the representative features in the second feature set.

[0086] Optionally, the representative features of all sub-cluster clusters corresponding to the training set can be determined; from the representative features of all sub-cluster clusters, other representative features except the first feature set can be determined; the similarity between the other representative features and the features of the image can be calculated respectively; according to the similarity between the other representative features and the features of the image, the other representative features are sorted; and according to the result of the sorting, a second number of representative features are selected from the other representative features as the second feature set.

[0087] Assume that Y clusters are further divided according to camera tags, and a total of Z sub-clusters are obtained. Among the Z sub-clusters obtained by clustering, in addition to the sub-clusters other than set P, there are also (ZP) sub-clusters. Next, the proxy features corresponding to (ZP) sub-clusters and the image x are calculated respectively. i The similarity between the features of , and sort the (ZP) sub-clusters according to the calculated similarity. Based on the sorting result, M sub-clusters can be selected from the (ZP) sub-clusters, and the second feature set Q can be obtained according to the proxy corresponding to the selected M sub-clusters. This set Q can represent the image x i The corresponding negative difficulty proxy set. The value of M can be set according to the requirements, for example, the value of M can be 40, 50, 60 or other values, which is not limited in this embodiment.

[0088] After the first feature set and the second feature set are determined, the cross-camera contrast loss may be calculated.

[0089] Optionally, the similarity between the feature of the image and the first representative feature in the first feature set can be calculated as the third similarity, and the sub-cluster corresponding to the first representative feature has a different camera label from the image. The first representative feature refers to the representative feature of any sub-cluster in the first feature set. The term "first" is used here for convenience of description and distinction only, and does not impose any restriction on the order of the representative features.

[0090] Next, the similarities between the features of the image and the multiple representative features in the first feature set are calculated respectively to obtain multiple fourth similarities; the similarities between the features of the image and the multiple representative features in the second feature set are calculated respectively to obtain multiple fifth similarities. Next, the cumulative value of the multiple fourth similarities and the multiple fifth similarities is calculated, and the cross-camera contrast loss of the sub-cluster corresponding to the image and the first representative feature is determined according to the ratio of the third similarity to the cumulative value.

[0091] Similarly, the cross-camera contrast loss of the sub-cluster corresponding to each representative feature in the first feature set and the image can be calculated respectively, and the cross-camera contrast loss of the image can be determined according to the cross-camera contrast loss of the sub-cluster corresponding to each of the multiple representative features in the first feature set. For example, the cross-camera contrast loss of the sub-cluster corresponding to each of the multiple representative features in the first feature set and the image can be cumulatively calculated to obtain the cross-camera contrast loss of the image.

[0092] Optionally, the cross-camera contrast loss function L Inter The calculation formula can be shown as the following formula 2:

[0093]

[0094] In formula 2, S(p, x i )=exp(K[p] T f(x i ) / τ), represents the image x i The similarity of the proxy features corresponding to the sub-cluster with the same cluster label. That is, p∈P,S(p,x i ) is the third similarity. When p≠j, S(p, x i )=exp(K[p] T f(x i ) / τ), represents the image x i The similarity of the proxy features corresponding to the sub-clusters with the same cluster label but different camera labels, that is, the cross-camera similarity within the same cluster. Where |P| represents the size of the set P, and N′ represents the number of images in the mini-batch.

[0095] In the numerator, S(u, x i ) represents the image x i The similarity with the proxy features in the set P is the fourth similarity; when u∈P, multiple fourth similarities are obtained. u∈P S(u,x i ), indicating image x i The accumulated value of the similarity with the proxy features in the set P, that is, the accumulated value of multiple fourth similarities. S(q, x i ) represents the image x i The similarity with the proxy features in set P, i.e. the fifth similarity; q∈Q,∑ q∈Q S(q,x i ), indicating image x i The accumulated value of the similarity with the proxy features in the set Q, that is, the accumulated value of multiple fifth similarities.

[0096] Based on the above formula 2, the intra-class cross-camera contrast loss when clustering based on features can be calculated. In the subsequent training process, this intra-class cross-camera contrast loss can be optimized to reduce the intra-class differences caused by camera differences in the clusters obtained by clustering, thereby improving the performance of the neural network model.

[0097] In some other embodiments, the intra-camera contrast loss and the cross-camera contrast loss may be calculated separately, a joint loss may be constructed based on the intra-camera contrast loss and the cross-camera contrast loss, and the neural network model may be trained based on the joint loss.

[0098] In this embodiment, the overall loss function of the neural network model can be calculated by the following formula 3:

[0099] L=L Intra +λL Inter Formula 3

[0100] In Formula 3, λ is a weighting coefficient, which can be 0.5, 0.4 or other optional values ​​during actual network training.

[0101] Based on the loss function recorded in the above embodiments, after calculating the loss of the neural network model, gradient back propagation can be used to update the network parameters of the neural network model. Among them, the network parameters may include the parameters of the network used to perform feature extraction operations in the neural network model (such as ResNet-50 recorded in the above embodiments), which will not be repeated.

[0102] It is worth noting that in the above embodiments of the present application, the proxy in the proxy memory bank can be continuously updated when training is performed based on each mini-batch.

[0103] The following will continue to use the first sub-cluster as an example for exemplary explanation. Each time after determining the representative features of the first sub-cluster cluster to which the image belongs, the representative features of the first sub-cluster cluster can be updated according to the features of the image, the set update rate, and the representative features of the first sub-cluster cluster. That is, according to the image and the proxy in the cluster cluster that the image currently corresponds to, the proxy currently corresponding to the sub-cluster cluster is updated to obtain a new proxy. Optionally, the proxy update algorithm can refer to the following formula 4:

[0104] K[j]←μK[j]+(1-μ)f θ (x i ) Formula 4

[0105] In formula 4, f θ represents a neural network model with parameters θ, f θ (x i) represents the neural network model for the image x i The features obtained by feature extraction; K[j] on the right side of the arrow represents the image x i The current proxy of the corresponding sub-cluster, K[j] on the left side of the arrow represents image x i The corresponding sub-cluster is updated proxy. μ represents the update rate. In some embodiments, the value of μ may be 0.2 or other set values, which is not limited in this embodiment.

[0106] For example, the batch training set obtained from the training set includes: mini-batchA and mini-batchB. Assume that mini-batchA contains picture 1 from sub-cluster1, and mini-batchB contains picture 2 from sub-cluster1.

[0107] After training based on mini-batchA, the features of image 1 and the proxy1 currently corresponding to sub-cluster 1 can be used to update the current proxy1 of sub-cluster 1 to obtain proxy1`. When training with mini-batchB, the representative feature corresponding to image 2 queried from the proxy memory bank is proxy1`.

[0108] Similarly, after the training based on mini-batchB is completed, the features of image 2 and the proxy1` currently corresponding to sub-cluster1 can be used to update the current proxy1` of sub-cluster1 to obtain proxy1``.

[0109] For another example, suppose mini-batchA contains n images from sub-cluster1. Then, when training based on mini-batchA, the proxy corresponding to sub-cluster1 will be updated n times. That is, when training based on each image, the proxy currently corresponding to sub-cluster1 will be updated based on the features of the image and the proxy currently corresponding to sub-cluster1. I will not go into details.

[0110] It should also be noted that the training of the neural network model in the embodiment of the present application may include multiple rounds until the loss function of the neural network model converges to a specified range. One round of training includes: a feature extraction operation, a clustering operation, an operation of training the network with a mini-batch of a set size, and an operation of optimizing the model parameters based on gradient back propagation. That is to say, after one round of training is completed, when the next round of training is carried out, the training set can be re-input into the neural network model, and the neural network model can re-perform feature extraction and clustering operations based on the updated network parameters; based on the multiple clustering clusters obtained by the clustering operation, the camera difference loss can be calculated based on the method recorded in the aforementioned embodiment, and the model can be optimized based on the camera difference loss. In this iterative training method, the data set is re-clustered in each round of iteration, thereby effectively alleviating the problem of error accumulation between multiple rounds of iterations and improving the performance of the trained neural network model.

[0111] Figure 4a A flowchart of a method for training a re-identification model for a moving object provided by another exemplary embodiment of the present application is shown in FIG. Figure 4a As shown, the method includes:

[0112] Step 401a: respond to the client's call request for the designated interface, and obtain multiple pictures of the moving object included in the interface parameters as a training set, wherein the multiple pictures are respectively annotated with camera tags.

[0113] Step 402a: input the multiple images into a neural network model to obtain features of each of the multiple images.

[0114] Step 403a: performing a clustering operation on the multiple images according to their respective features to obtain multiple clusters.

[0115] Step 404a: Calculate the camera difference loss of the clustering operation according to the features of the pictures included in each of the multiple clusters and the camera tags of the pictures included in each of the multiple clusters.

[0116] Step 405a: using the camera difference losses corresponding to the multiple clusters, iteratively train the neural network model to obtain a re-identification model.

[0117] Step 406a: Return the trained re-recognition model to the client.

[0118] The execution subject of this embodiment may be a server device, such as a conventional server or a cloud server, wherein the client may be implemented as a mobile phone, computer, tablet computer or other device on the user side.

[0119] In this embodiment, the training method for the re-identification model of mobile objects provided by the aforementioned embodiments can be encapsulated as a software tool that can be used by a third party, such as a SaaS (Software-as-a-Service) tool. Among them, the SaaS tool can be implemented as a plug-in or an application. The plug-in or application can be deployed on a server device, and a specified interface can be opened to third-party users such as clients. Furthermore, third-party users such as clients can easily access and use the training method of the re-identification model provided by the server device by calling the specified interface.

[0120] For example, in some scenarios, the SaaS tool can be deployed on a cloud server, and a third-party user can call a designated interface provided by the cloud server to use the SaaS tool online. When a third-party user calls the designated interface, the input data required for training the re-identification model of mobile objects, i.e., the multiple pictures described in this embodiment, can be provided to the SaaS tool by configuring the interface parameters of the designated interface. Each picture is marked with a camera tag, which can be automatically added when the camera takes the picture, or can be added by a third-party user, or the server can automatically identify the camera corresponding to the picture and mark it according to the shooting angle, camera position and other information corresponding to the picture, which is not limited in this embodiment.

[0121] After receiving the call request for the specified interface, the SaaS tool can obtain multiple pictures provided by the client as a training set by parsing the interface parameters of the specified interface. After the SaaS tool optimizes the neural network model based on the training set and obtains the result model, it can return the trained result model to the client through the specified interface or other communication methods, so that the client can re-identify the moving object based on the result model. Among them, the optional implementation method of iteratively training the neural network model based on the training set can refer to the records of the aforementioned embodiment, which will not be repeated here.

[0122] In this embodiment, the server device can provide the client with a training service for the re-identification model of mobile objects based on the SaaS tool running thereon, and the client user can use the training service of the re-identification model provided by the server device by calling the interface provided by the SaaS tool. Based on the interaction between the client and the server device, the client can completely hand over the training operation of the re-identification model to the server device, and then, with the help of the powerful computing power and reliable attribute recognition algorithm of the server device, a low-cost and high-efficiency model training operation can be achieved.

[0123] Based on the neural network model trained in the above embodiment, the present application embodiment also provides a method for re-identifying a moving object. Figure 4b as well as Figure 4c Provide explanation.

[0124] Figure 4b A flow chart of a method for re-identifying a moving object provided by an exemplary embodiment of the present application is shown in FIG. Figure 4b As shown, the method includes:

[0125] Step 401b: Obtain the image to be queried and the image library to be retrieved that contains the moving object.

[0126] Step 402b, input the image to be queried and the images in the image library to be retrieved into a neural network model to obtain at least one image in the image library to be retrieved that matches the image to be queried; wherein the neural network model is trained based on a training set marked with camera labels and a camera difference loss generated by clustering operations on multiple images in the training set.

[0127] Among them, the optional implementation method of training the neural network model based on the training set marked with camera labels and the camera difference loss generated by clustering multiple pictures in the training set can be referred to the records of the aforementioned embodiments and will not be repeated here.

[0128] In this embodiment, after the query image and the image library to be retrieved are input into the neural network model, the neural network model can extract features from the query image and each image in the image library to be retrieved based on the pre-learned network parameters, and obtain the features of the query image and the features of each image in the image library to be retrieved, such as Figure 4c shown.

[0129] Next, the similarity between the features of the image to be queried and the features of each image in the image library to be retrieved can be calculated, wherein the similarity can be cosine similarity, Euclidean distance similarity, Manhattan distance similarity, etc., which is not limited in this embodiment.

[0130] After the similarity is calculated, the images in the image library to be searched can be sorted according to the similarity, and multiple images with the highest ranking (TopN) can be selected as search results, which will not be described in detail.

[0131] Figure 5 A flow chart of a method for re-identifying a moving object provided by another exemplary embodiment of the present application is shown in FIG. Figure 5 As shown, the method includes:

[0132] Step 501: display a first interface, wherein the first interface includes: a picture adding control.

[0133] Step 502: In response to the triggering operation of adding a control to the picture, the picture to be queried containing the moving object is obtained.

[0134] Step 503: input the to-be-queried picture and the pictures in the to-be-retrieved picture library into a neural network model; the neural network model is trained based on a training set annotated with camera labels and camera difference loss generated by clustering multiple pictures in the training set.

[0135] Step 504: Obtain at least one picture in the to-be-searched picture library output by the neural network model that matches the to-be-queried picture.

[0136] Step 505: Display the at least one picture in the second interface. This embodiment can be executed by a terminal device, which can be implemented as a user-side device such as a computer, a mobile phone, or a tablet computer. A neural network model is deployed on the terminal device, and the neural network model is used to realize the re-identification of the moving object.

[0137] Among them, the optional implementation method of training the neural network model based on the training set marked with camera labels and the camera difference loss generated by clustering multiple pictures in the training set can be referred to the records of the aforementioned embodiments and will not be repeated here.

[0138] The terminal device may provide an operation interface for users to upload pictures to be queried. For ease of description and distinction, the interface providing picture upload operation is described as a first interface. The first interface includes a picture adding control. Users may trigger the picture adding control to upload pictures to be queried.

[0139] The image to be queried may be obtained by photographing a moving object with a camera, and the moving object may be a pedestrian, a vehicle, or an animal, etc., which is not limited in this embodiment.

[0140] The image library to be searched includes images obtained by taking pictures of different moving objects by multiple cameras. The image library to be searched can be specified by the user or requested by the terminal device to the server, which is not limited in this embodiment.

[0141] After obtaining the picture to be queried, the terminal device inputs the picture to be queried and the pictures in the picture library to be retrieved into the neural network model, and obtains at least one picture that matches the picture to be queried and is retrieved by the neural network model from the picture library to be retrieved.

[0142] Next, the terminal device may display the at least one picture in a second interface, wherein the second interface may be the same interface as the first interface or may be a different interface, which is not limited in this embodiment.

[0143] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 104 can be device A; for another example, the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0144] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel, and the serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel.

[0145] It should be noted that the descriptions such as “first” and “second” in this article are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit “first” and “second” to different types.

[0146] Figure 6 is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application, such as Figure 6 As shown, the electronic device includes: a memory 601 and a processor 602.

[0147] The memory 601 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0148] Among them, the memory 602 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0149] The processor 602 is coupled to the memory 601 and is used to execute the computer program in the memory 601, so as to: obtain multiple pictures containing moving objects as training sets, wherein the multiple pictures are respectively marked with camera labels; input the multiple pictures into a neural network model to obtain respective features of the multiple pictures; perform a clustering operation on the multiple pictures according to the respective features of the multiple pictures to obtain multiple clusters; calculate a camera difference loss of the clustering operation according to the features of the pictures respectively included in the multiple clusters and the camera labels of the pictures respectively included in the multiple clusters; and optimize the neural network model using the camera difference losses corresponding to the multiple clusters.

[0150] Further optionally, when the processor 602 calculates the camera difference loss of the clustering operation according to the features of the pictures respectively included in the multiple clustering clusters and the camera labels of the pictures respectively included in the multiple clustering clusters, it is specifically used to: for a first clustering cluster among the multiple clustering clusters, according to the camera labels marked on the pictures in the first clustering cluster, divide the pictures in the first clustering cluster to obtain multiple sub-clustering clusters; the multiple sub-clustering clusters correspond to different camera labels; according to the features of the pictures respectively included in the multiple sub-clustering clusters, determine the initial values ​​of the representative features of each of the multiple sub-clustering clusters; select multiple batch training sets from the training set, each batch training set contains a first number of pictures; for any batch training set among the multiple batch training sets, based on the features of the pictures in the batch training set and the representative features of at least one sub-clustering cluster having the same camera label as the pictures, calculate the same-camera contrast loss; and / or, based on the features of the pictures in the batch training set and the representative features of each of the at least one sub-clustering clusters included in the multiple clustering clusters, calculate the cross-camera contrast loss.

[0151] Further optionally, when the processor 602 calculates the same-camera contrast loss based on the features of the pictures in the batch training set and the representative features of at least one sub-cluster cluster having the same camera label as the pictures, the processor 602 is specifically used to: determine, for any picture in the batch training set, the representative features of the first sub-cluster cluster to which the picture belongs and the representative features of each of at least one second sub-cluster cluster having the same camera label as the picture; calculate the similarity between the features of the picture and the representative features of the first sub-cluster cluster as the first similarity; respectively calculate the similarity between the features of the picture and the representative features of each of the at least one second sub-cluster clusters to obtain at least one second similarity; calculate the cumulative value of the at least one second similarity, and determine the intra-camera contrast loss corresponding to the picture according to the ratio of the first similarity to the cumulative value.

[0152] Further optionally, after determining the representative features of the first sub-cluster to which the image belongs, the processor 602 is further configured to: update the representative features of the first sub-cluster according to features of the image, a set update rate, and the representative features of the first sub-cluster.

[0153] Further optionally, when the processor 602 calculates the cross-camera contrast loss based on the features of the pictures in the batch training set and the representative features of each of at least one sub-clustering cluster contained in the multiple clustering clusters, it is specifically used to: for any picture in the batch training set, determine the representative features of each of the multiple sub-clustering clusters contained in the clustering cluster to which the picture belongs, as a first feature set; from the sub-clustering clusters contained in each of the multiple clustering clusters, select the representative features of the sub-clustering cluster of the clustering cluster that does not contain the picture as a second feature set; calculate the similarity between the features of the picture and the first representative feature in the first feature set as a third similarity, the first representative feature being the first feature set. any representative feature in the set; respectively calculate the similarities between the features of the image and each of the multiple representative features in the first feature set to obtain multiple fourth similarities; respectively calculate the similarities between the features of the image and each of the multiple representative features in the second feature set to obtain multiple fifth similarities; calculate the cumulative values ​​of the multiple fourth similarities and the multiple fifth similarities, and determine the cross-camera contrast loss of the sub-cluster cluster corresponding to the image and the first representative feature according to the ratio of the third similarity to the cumulative value; determine the cross-camera contrast loss of the image according to the cross-camera contrast loss of the sub-cluster cluster corresponding to each of the multiple representative features in the first feature set.

[0154] Further optionally, when the processor 602 selects representative features of a sub-cluster of the cluster that does not include the image from the sub-cluster clusters included in each of the multiple cluster clusters as the second feature set, it is specifically used to: determine the representative features of each of the sub-cluster clusters corresponding to the training set; determine other representative features except the first feature set from the representative features of each of the all sub-cluster clusters; respectively calculate the similarity between the other representative features and the features of the image; sort the other representative features according to the similarity between the other representative features and the features of the image; and select a second number of representative features from the other representative features as the second feature set according to the result of the sorting.

[0155] Further optionally, the processor 602 is also used to: set cluster pseudo-labels for the multiple cluster clusters respectively; set local pseudo-labels for the multiple sub-cluster clusters obtained by dividing each of the multiple cluster clusters; set a query index for the representative features of all sub-cluster clusters included in the training set according to the cluster pseudo-labels and the local pseudo-labels; wherein the query index is used to query the representative features of the sub-cluster cluster to which the pictures in the batch training set belong, and / or to query the representative features of each of the multiple sub-cluster clusters contained in the cluster cluster to which the pictures in the batch training set belong.

[0156] Further, if Figure 6 As shown, the electronic device also includes: a communication component 603, a display component 604, a power component 605, an audio component 606 and other components. Figure 6 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 6 Components shown.

[0157] Among them, the communication component 603 is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can be implemented based on near field communication (NFC) technology, radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0158] Among them, the display 604 includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0159] The power supply component 605 provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0160] In this embodiment, when training a model for re-identifying moving objects, there is no need to use a training set labeled with the true value of the moving object. It is only necessary to train the neural network model end-to-end, which is more advantageous in practical applications. At the same time, the pictures in the training set are labeled with camera labels. After obtaining clusters based on the characteristics of the pictures, the camera difference loss generated by the clustering operation can be calculated according to the characteristics of the pictures contained in each cluster and the camera label, and the neural network model can be optimized based on the camera difference loss. Furthermore, when training the neural network model, the impact of camera differences can be fully considered to promote the semantic consistency of the clusters obtained by clustering the neural network model, optimize the recognition ability of the neural network model for the target data set, and improve the performance of the neural network model in the application scenario of re-identifying moving objects.

[0161] In addition to the aforementioned embodiments, Figure 6 The electronic device shown can also execute the following training method for a re-identification model of a mobile object: the processor 602 responds to a call request from a client to a specified interface, obtains multiple pictures of the mobile object contained in the interface parameters as a training set, and the multiple pictures are respectively marked with camera labels; the multiple pictures are input into a neural network model to obtain respective features of the multiple pictures; according to the respective features of the multiple pictures, the multiple pictures are clustered to obtain multiple clusters; according to the features of the pictures contained in each of the multiple clusters and the camera labels of the pictures contained in each of the multiple clusters, the camera difference loss of the clustering operation is calculated; using the camera difference losses corresponding to each of the multiple clusters, the neural network model is iteratively trained to obtain a re-identification model; and the trained re-identification model is returned to the client.

[0162] In addition to the aforementioned embodiments, Figure 6 The electronic device shown can also execute the following method for re-identifying a moving object: obtaining a query picture containing a moving object and a library of pictures to be retrieved through the processor 602; inputting the query picture and the pictures in the library of pictures to be retrieved into a neural network model to obtain at least one picture in the library of pictures to be retrieved that matches the query picture; wherein the neural network model is trained based on a training set marked with camera labels and a camera difference loss generated by clustering operations on multiple pictures in the training set.

[0163] In addition to the aforementioned embodiments, Figure 6The electronic device shown can also execute the following method for re-identifying a moving object: displaying a first interface through a display component 604, wherein the first interface includes: a picture adding control; the processor 602 obtains a picture to be queried containing a moving object in response to a triggering operation on the picture adding control; inputs the picture to be queried and the pictures in the picture library to be retrieved into a neural network model; the neural network model is trained based on a training set marked with camera labels and camera difference loss generated by clustering operations on multiple pictures in the training set; obtains at least one picture in the picture library to be retrieved output by the neural network model that matches the picture to be queried; and displays the at least one picture in a second interface through the display component 604.

[0164] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be executed by an electronic device in the above method embodiment.

[0165] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0166] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0167] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0168] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0169] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0170] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0171] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0172] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0173] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for re-identifying a moving object, characterized in that: include: Displaying a first interface, wherein the first interface includes: a picture adding control; In response to a triggering operation of adding a control to the picture, obtaining a picture to be queried containing a moving object; Inputting the to-be-queried picture and the pictures in the to-be-retrieved picture library into a neural network model; the neural network model is trained based on a training set annotated with camera labels and camera difference loss generated by clustering multiple pictures in the training set; Obtain at least one picture in the to-be-searched picture library output by the neural network model that matches the to-be-queried picture; In the second interface, displaying the at least one picture; The camera difference loss includes: for any batch training set selected from a plurality of batch training sets from a training set, the same-camera comparison loss is calculated based on the features of the pictures in the batch training set and the representative features of at least one sub-cluster in the plurality of clusters having the same camera label as the pictures; and / or the cross-camera comparison loss is calculated based on the features of the pictures in the batch training set and the representative features of at least one sub-cluster in the plurality of clusters; each batch training set includes a first number of pictures, the plurality of clusters are obtained by clustering according to the features of the plurality of pictures, and the pictures in the cluster are divided into a plurality of sub-cluster based on the camera labels marked on the pictures in the cluster.

2. A method for re-identifying a moving object, characterized in that: include: Obtain the image to be queried and the image library to be retrieved containing the moving object; Inputting the to-be-queried picture and the pictures in the to-be-searched picture library into a neural network model, and obtaining at least one picture in the to-be-searched picture library that matches the to-be-queried picture; The neural network model is trained based on a training set labeled with camera labels and a camera difference loss generated by clustering multiple images in the training set; The camera difference loss includes: for any batch training set selected from a plurality of batch training sets from a training set, the same-camera comparison loss is calculated based on the features of the pictures in the batch training set and the representative features of at least one sub-cluster in the plurality of clusters having the same camera label as the pictures; and / or the cross-camera comparison loss is calculated based on the features of the pictures in the batch training set and the representative features of at least one sub-cluster in the plurality of clusters; each batch training set includes a first number of pictures, the plurality of clusters are obtained by clustering according to the features of the plurality of pictures, and the pictures in the cluster are divided into a plurality of sub-cluster based on the camera labels marked on the pictures in the cluster.

3. A method for training a moving object re-identification model, characterized in that: include: In response to a call request from a client to a specified interface, a plurality of pictures of a moving object included in the interface parameters are obtained as a training set, wherein the plurality of pictures are respectively annotated with camera labels; Inputting the multiple images into a neural network model to obtain features of each of the multiple images; According to the respective features of the multiple images, clustering operation is performed on the multiple images to obtain multiple clusters; Calculating a camera difference loss of the clustering operation according to features of the pictures respectively included in the plurality of clusters and camera labels of the pictures respectively included in the plurality of clusters; Iteratively training the neural network model using the camera difference losses corresponding to each of the plurality of clusters to obtain a re-identification model; Returning the trained re-identification model to the client; The camera difference loss includes: for any batch training set selected from a plurality of batch training sets, the same-camera comparison loss is calculated based on the features of the images in the batch training set and the representative features of at least one sub-cluster in the plurality of clusters having the same camera label as the images; and / or the cross-camera comparison loss is calculated based on the features of the images in the batch training set and the representative features of at least one sub-cluster in the plurality of clusters; each batch training set includes a first number of images, and the images in the cluster are divided into a plurality of sub-cluster clusters based on the camera labels marked on the images in the cluster.

4. A method for training a re-identification model for a moving object, characterized in that: include: Obtain multiple pictures containing moving objects as a training set, wherein the multiple pictures are respectively annotated with camera labels; Inputting the multiple images into a neural network model to obtain features of each of the multiple images; According to the respective features of the multiple images, clustering operation is performed on the multiple images to obtain multiple clusters; Calculating a camera difference loss of the clustering operation according to features of the pictures respectively included in the plurality of clusters and camera labels of the pictures respectively included in the plurality of clusters; Optimizing the neural network model using camera difference losses corresponding to each of the plurality of clusters; The step of calculating the camera difference loss of the clustering operation according to the features of the pictures respectively included in the multiple clusters and the camera tags of the pictures respectively included in the multiple clusters includes: Selecting a plurality of batch training sets from the training set, each batch training set comprising a first number of images; For any batch training set among the multiple batch training sets, based on features of the pictures in the batch training set and representative features of at least one sub-clustering cluster having the same camera label as the pictures, calculating the same-camera contrast loss; and / or, based on features of the pictures in the batch training set and representative features of at least one sub-clustering cluster included in the multiple clustering clusters, calculating the cross-camera contrast loss; and dividing the pictures in the clustering cluster into multiple sub-clustering clusters based on camera labels marked on the pictures in the clustering cluster.

5. The method according to claim 4, characterized in that Also includes: For a first cluster among the multiple clusters, the pictures in the first cluster are divided according to the camera tags marked on the pictures in the first cluster to obtain multiple sub-clusters; the multiple sub-clusters correspond to different camera tags; According to the features of the pictures respectively included in the plurality of sub-clusters, an initial value of a representative feature of each of the plurality of sub-clusters is determined.

6. The method according to claim 4, characterized in that Based on the features of the pictures in the batch training set and the representative features of at least one sub-cluster having the same camera label as the pictures, calculating the intra-camera contrast loss includes: For any picture in the batch training set, determine a representative feature of a first sub-cluster to which the picture belongs and a representative feature of each of at least one second sub-cluster having the same camera tag as the picture; Calculating the similarity between the feature of the image and the representative feature of the first sub-cluster as a first similarity; Respectively calculating similarities between the features of the image and the representative features of each of the at least one second sub-cluster, to obtain at least one second similarity; An accumulated value of the at least one second similarity is calculated, and an intra-camera contrast loss corresponding to the image is determined according to a ratio of the first similarity to the accumulated value.

7. The method according to claim 6, characterized in that After determining the representative feature of the first sub-cluster to which the picture belongs, the method further includes: The representative feature of the first sub-cluster is updated according to the feature of the picture, the set update rate and the representative feature of the first sub-cluster.

8. The method according to claim 4, characterized in that Calculating the cross-camera contrast loss based on the features of the images in the batch training set and the representative features of each of at least one sub-clustering clusters included in the multiple clustering clusters includes: For any picture in the batch training set, determine representative features of each of a plurality of sub-clusters contained in the cluster to which the picture belongs as a first feature set; Selecting, from the sub-clusters contained in each of the plurality of clusters, representative features of the sub-clusters of the cluster that does not contain the image as the second feature set; calculating a similarity between a feature of the image and a first representative feature in the first feature set as a third similarity, where the first representative feature is any representative feature in the first feature set; Calculating similarities between the features of the image and each of the plurality of representative features in the first feature set, respectively, to obtain a plurality of fourth similarities; respectively calculating similarities between the features of the image and a plurality of representative features in the second feature set to obtain a plurality of fifth similarities; Calculating cumulative values ​​of the plurality of fourth similarities and the plurality of fifth similarities, and determining a cross-camera contrast loss of a sub-cluster corresponding to the image and the first representative feature according to a ratio of the third similarity to the cumulative value; The cross-camera contrast loss of the image is determined according to the cross-camera contrast loss of the sub-clusters corresponding to the image and the plurality of representative features in the first feature set.

9. The method according to claim 8, characterized in that Selecting representative features of the sub-cluster of the cluster that does not include the image from the sub-cluster of each of the multiple clusters as the second feature set, including: Determine the representative features of all sub-cluster clusters corresponding to the training set; Determine other representative features except the first feature set from the representative features of all the sub-clusters; Calculating the similarity between the other representative features and the features of the image respectively; Sorting the other representative features according to the similarity between the other representative features and the features of the image; According to the sorting result, a second number of representative features are selected from the other representative features as the second feature set.

10. The method according to any one of claims 5 to 9, characterized in that: Also includes: For the multiple clusters, respectively set cluster pseudo labels; For each of the plurality of sub-clusters obtained by dividing the plurality of clusters, local pseudo labels are respectively set; According to the cluster pseudo-label and the local pseudo-label, setting a query index for representative features of all sub-cluster clusters included in the training set; The query index is used to query representative features of the sub-cluster to which the images in the batch training set belong, and / or to query representative features of each of the multiple sub-cluster contained in the cluster to which the images in the batch training set belong.

11. An electronic device, characterized in that: include: Memory and processor; The memory is used to store one or more computer instructions; The processor is configured to execute the one or more computer instructions to: perform the steps in the method according to any one of claims 1-10.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps in the method described in any one of claims 1 to 10 can be implemented.

Images