A target clustering method, device, equipment and medium

Abstract

The present application discloses a target clustering method, device, equipment and medium, which are used to implement target clustering based on the similarity matrix of multiple target images and using a clustering neural network. The target clustering method provided by the present application includes: obtaining multiple target images, and extracting the image features of each target image to obtain multiple image features; for the first target image and the second target image in every two target images: based on the image features of the first target image and the image features of the second target image, determining the similarity matrix of the first target image and the second target image; inputting the similarity matrix into a preset clustering neural network, and obtaining the classification probability corresponding to the first target image and the second target image through the clustering neural network, and the classification probability is used to determine whether the first target image and the second target image belong to the same target image; based on the classification probability corresponding to every two target images, performing target clustering.

Description

Technical Field

[0001] The present application relates to the field of image technology, and in particular to a target clustering method, device, equipment and medium. Background Art

[0002] Human clustering clusters similar images into one category by comparing the similarity of captured images. Existing human clustering technologies can be divided into traditional clustering-based methods and neural network clustering-based methods. The mainstream neural network is generally a graph neural network, such as a graph convolutional network (GCN). Traditional clustering methods include density-based spatial clustering of applications with noise (DBSCAN), agglomerative hierarchical clustering and other algorithms. Summary of the invention

[0003] The embodiments of the present application provide a target clustering method, apparatus, device and medium for implementing target clustering based on a similarity matrix of multiple target images using a clustering neural network.

[0004] An object clustering method provided in an embodiment of the present application includes:

[0005] Acquire multiple target images, and extract image features of each of the target images to obtain multiple image features;

[0006] For a first target image and a second target image in each of the two target images: based on the image features of the first target image and the image features of the second target image, determine a similarity matrix between the first target image and the second target image; input the similarity matrix into a preset clustering neural network, and obtain the classification probabilities corresponding to the first target image and the second target image through the clustering neural network, wherein the classification probabilities are used to determine whether the first target image and the second target image belong to the same target image;

[0007] Target clustering is performed based on the classification probability corresponding to every two target images.

[0008] The method of the embodiment of the present application obtains multiple target images and extracts image features of each of the target images to obtain multiple image features; thus, for the first target image and the second target image in every two of the target images: based on the image features of the first target image and the image features of the second target image, determine the similarity matrix of the first target image and the second target image; input the similarity matrix into a preset clustering neural network, and obtain the classification probabilities of the images corresponding to the first target image and the second target image through the clustering neural network for determining whether the first target image and the second target image belong to the same target; finally, target clustering can be performed based on the classification probabilities corresponding to every two target images. Therefore, the embodiment of the present application can realize target clustering based on the similarity matrices of multiple target images using a clustering neural network. For example, it can be applied to clustering applications such as human bodies, faces, and cars to achieve fast and accurate clustering of targets.

[0009] In some embodiments, the clustering neural network includes: a coding layer, a feature extraction layer, a feature fusion layer, and a classification layer;

[0010] Obtaining classification probabilities corresponding to the first target image and the second target image through the clustering neural network includes:

[0011] Encoding the similarity matrix between the first target image and the second target image through the encoding layer to obtain a similarity encoding matrix;

[0012] Performing feature extraction on the similarity encoding matrix through the feature extraction layer to obtain at least one feature, and stacking the at least one feature into a feature matrix;

[0013] The feature matrix is ​​fused through the feature fusion layer to obtain fused features;

[0014] The fused features are input into a classification layer, and the classification layer is used to obtain classification probabilities corresponding to the first target image and the second target image.

[0015] In some embodiments, the encoding layer, feature extraction layer, and feature fusion layer belong to the first branch of the clustering neural network; the fusion feature is the first fusion feature; the clustering neural network further includes an encoding layer, a feature extraction layer, and a feature fusion layer of a second branch; wherein the encoding layer of the second branch includes a first module and a second module;

[0016] The method further comprises:

[0017] Determining the timestamp information and the longitude and latitude information of the first target image, and the timestamp information and the longitude and latitude information of the second target image;

[0018] Determine a capture time interval matrix and a capture space distance matrix of the first target image and the second target image based on the timestamp information and the longitude and latitude information of the first target image and the timestamp information and the longitude and latitude information of the second target image;

[0019] The capture time interval matrix and the capture space distance matrix are stacked to obtain a space-time matrix;

[0020] and, using the attribute information of the first target image and the attribute information of the second target image, determining an attribute matrix corresponding to the first target image and the second target image;

[0021] Inputting the space-time matrix into the first module to obtain a space-time coding matrix; and inputting the attribute matrix into the second module to obtain an attribute coding matrix;

[0022] The spatiotemporal coding matrix and the attribute coding matrix are stacked to obtain a mixed feature matrix;

[0023] Inputting the mixed feature matrix into the feature extraction layer of the second branch, and obtaining a second fusion feature after being processed by the feature extraction layer and the feature fusion layer of the second branch respectively;

[0024] Concatenate the first fusion feature and the second fusion feature to obtain a final fusion feature;

[0025] The final fusion feature is input into the classification layer to obtain the classification probabilities corresponding to the first target image and the second target image.

[0026] In some embodiments, the method further comprises: performing a K-nearest neighbor search on the plurality of image features to obtain a neighbor index matrix;

[0027] Determining a similarity matrix between the first target image and the second target image based on an image feature of the first target image and an image feature of the second target image includes:

[0028] Based on the neighbor index matrix, determine a first neighbor feature matrix and a second neighbor feature matrix; wherein the first neighbor feature matrix is ​​a neighbor feature matrix of image features of the first target image; and the second neighbor feature matrix is ​​a neighbor feature matrix of image features of the second target image;

[0029] A similarity matrix between the first neighbor feature matrix and the second neighbor feature matrix is ​​calculated.

[0030] In some embodiments, determining the first neighbor feature matrix and the second neighbor feature matrix based on the neighbor index matrix includes:

[0031] From the neighbor index matrix, a preset number of neighbor features of the image features of the first target image are taken out, and stacked to obtain a first neighbor feature matrix;

[0032] And, a preset number of neighbor features of the image features of the second target image are taken out from the neighbor index matrix, and stacked to obtain a second neighbor feature matrix.

[0033] In some embodiments, clustering the targets based on the classification probabilities corresponding to every two target images includes:

[0034] Generate a probability matrix using the classification probabilities corresponding to every two target images;

[0035] For each classification probability in the probability matrix, compare the classification probability with a preset threshold, if the classification probability is greater than the preset threshold, set the reset value of the classification probability to 1, otherwise, set the reset value of the classification probability to 0;

[0036] Generate a K-nearest neighbor adjacency matrix using the reset value of each classification probability in the probability matrix;

[0037] The K-nearest neighbor adjacency matrix is ​​used to achieve target clustering.

[0038] In some embodiments, using the K-nearest neighbor adjacency matrix to achieve target clustering includes:

[0039] For each element in the K-nearest neighbor adjacency matrix, when the element is 1, it is determined that the two image features corresponding to the element belong to the image features of the same target; otherwise, it is determined that the two image features corresponding to the element do not belong to the image features of the same target.

[0040] An embodiment of the present application provides a target clustering device, comprising:

[0041] The first unit is used to acquire a plurality of target images and extract image features of each of the target images to obtain a plurality of image features;

[0042] The second unit is used for determining, for each of the first target image and the second target image in the two target images, a similarity matrix between the first target image and the second target image based on image features of the first target image and image features of the second target image; inputting the similarity matrix into a preset clustering neural network, and obtaining classification probabilities corresponding to the first target image and the second target image through the clustering neural network, wherein the classification probabilities are used to determine whether the first target image and the second target image belong to the same target image;

[0043] The third unit is used to perform target clustering based on the classification probability corresponding to every two target images.

[0044] Another embodiment of the present application provides an electronic device, which includes a memory and a processor, wherein the memory is used to store program instructions, and the processor is used to call the program instructions stored in the memory and execute any of the above methods according to the obtained program.

[0045] Another embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable the computer to execute any of the above methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A schematic diagram of the overall process of a target clustering method provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of a clustering neural network structure provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of another clustering neural network structure provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of a specific process of a target clustering method provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of a target clustering principle provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of a specific process of calculating a similarity matrix between every two target images provided in an embodiment of the present application;

[0053] Figure 7 A schematic diagram of a similarity matrix provided in an embodiment of the present application;

[0054] Figure 8 A schematic diagram of a specific process for predicting the classification probability of every two target images provided in an embodiment of the present application;

[0055] Fig. 9 A schematic diagram of a target clustering principle based on a multimodal clustering neural network provided in an embodiment of the present application;

[0056] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0057] Fig.11 A schematic diagram of the structure of a target clustering device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] The embodiments of the present application provide a target clustering method, apparatus, device and medium for implementing target clustering based on a similarity matrix of multiple target images using a clustering neural network.

[0060] Among them, the method and the device, equipment, and medium are based on the same application concept. Since the principles of solving problems by the method and the device, equipment, and medium are similar, the implementation of the device, equipment, medium, and method can refer to each other, and the repeated parts will not be repeated.

[0061] The terms "first", "second", etc. (if any) in the specification and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in a sequence other than the content illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0062] The following examples and embodiments are to be understood as illustrative examples only. Although this specification may refer to "one", "an" or "some" examples or embodiments in several places, this does not mean that each such reference relates to the same example or embodiment, nor does it mean that the feature applies only to a single example or embodiment. Individual features of different embodiments may also be combined to provide other embodiments. In addition, terms such as "including" and "comprising" should be understood as not limiting the described embodiments to consist only of those features that have been mentioned; such examples and embodiments may also include features, structures, units, modules, etc. that are not specifically mentioned.

[0063] The following is a detailed description of each embodiment of the present application in conjunction with the accompanying drawings. It should be noted that the display order of the embodiments of the present application only represents the order of the embodiments, and does not represent the advantages and disadvantages of the technical solutions provided by the embodiments.

[0064] Embodiment 1:

[0065] A target clustering method provided in the embodiment of the present application is shown in Figure 1 ,include:

[0066] S101, acquiring a plurality of target images, and extracting image features of each of the target images to obtain a plurality of image features;

[0067] The target image may be, for example, a human body image, a face image, a car image, etc., and may be determined according to actual needs.

[0068] Assume that there are N human body images captured by a camera {X1, X2, ..., X N}, and obtain N features {f1, f2, ..., f N}; Among them, a feature is extracted from each image, that is, the feature and the image correspond one to one.

[0069] S102, for a first target image and a second target image in each of the two target images: based on image features of the first target image and image features of the second target image, determine a similarity matrix between the first target image and the second target image; input the similarity matrix into a preset clustering neural network, and obtain classification probabilities corresponding to the first target image and the second target image through the clustering neural network, wherein the classification probabilities are used to determine whether the first target image and the second target image belong to the same target image;

[0070] The target clustering described in the embodiments of the present application, such as human body clustering, clusters similar images into one category by comparing the similarities of captured images.

[0071] Taking human body clustering as an example, there are two challenges in human body clustering: first, there are many human body images that are similar but not of the same person. Due to their high similarity, they are prone to form incorrect clusters; second, human body images are subject to body occlusion and long-distance blur. These images have low feature similarity with normal images and are prone to form clusters that are not recalled. These two points bring a trade-off between recall rate and error rate to human body clustering.

[0072] Therefore, the embodiment of the present application proposes to implement target clustering based on a similarity matrix of multiple target images using a preset clustering neural network. For example, it can be applied to clustering applications such as human bodies, faces, and cars to achieve fast and accurate clustering of targets.

[0073] S103: Perform target clustering based on the classification probabilities corresponding to every two target images.

[0074] In some embodiments, Figure 2 As shown, the clustering neural network includes: a coding layer, a feature extraction layer, a feature fusion layer, and a classification layer; the clustering neural network provided in the embodiment of the present application is a clustering neural network based on a convolutional neural network. Among them, the feature extraction layer may include multiple convolutional neural networks (CNNs), for example, including L=5 convolutional neural networks.

[0075] The clustering neural network provided in the embodiment of the present application can reduce the dependence of image features on the number of neighboring features selected, so it is applicable to a wider range of scenarios. In addition, since the clustering neural network provided in the embodiment of the present application does not directly use image features, even if the image feature extraction network is replaced, there is no need to retrain the clustering neural network, which greatly reduces maintenance costs.

[0076] In some embodiments, obtaining the classification probabilities corresponding to the first target image and the second target image through the clustering neural network includes:

[0077] Encoding the similarity matrix S0 of the first target image and the second target image through the encoding layer to obtain a similarity encoding matrix S1;

[0078] The feature extraction layer performs feature extraction on the similarity encoding matrix S1 to obtain at least one feature, and the at least one feature is stacked into a feature matrix; wherein, for example, the feature extraction layer includes 5 convolutional neural networks, then the similarity encoding matrix S1, after passing through the feature extraction layer, obtains 5 features, and the 5 features are stacked into a feature matrix X;

[0079] The feature matrix is ​​fused through the feature fusion layer to obtain a fusion feature , for example, the above five features are combined to obtain one feature;

[0080] The fused features are input into a classification layer, and the classification layer is used to obtain classification probabilities corresponding to the first target image and the second target image.

[0081] The classification layer, for example, includes a linear layer and a softmax operator.

[0082] In some embodiments, the encoding layer, feature extraction layer, and feature fusion layer belong to the first branch of the clustering neural network; the fusion feature is the first fusion feature; the clustering neural network further includes an encoding layer, a feature extraction layer, and a feature fusion layer of a second branch; wherein the encoding layer of the second branch includes a first module and a second module;

[0083] That is, in some embodiments, see Figure 3 The preset clustering neural network includes two branches, the first branch includes a coding layer, a feature extraction layer, and a feature fusion layer, which is used to input a similarity matrix of a first target image and a second target image, and output a first fusion feature after being processed by the coding layer, the feature extraction layer, and the feature fusion layer;

[0084] The second branch includes another encoding layer, a feature extraction layer, and a feature fusion layer, which is used to input the capture time interval matrix and the capture space distance matrix of the first target image and the second target image, and output the second fusion feature after being processed by the encoding layer, the feature extraction layer, and the feature fusion layer;

[0085] Then, the first fusion feature and the second fusion feature are stacked and input into the classification layer to predict the classification probability.

[0086] Therefore, an embodiment of the present application also provides a multimodal clustering model, which can, for example, cluster human bodies and comprehensively utilize information such as human characteristics, attributes, and time and space to improve the recall rate of human body clustering.

[0087] Therefore, in some embodiments, the method further comprises:

[0088] Determining the timestamp information and the longitude and latitude information of the first target image, and the timestamp information and the longitude and latitude information of the second target image;

[0089] Determine a capture time interval matrix and a capture space distance matrix of the first target image and the second target image based on the timestamp information and the longitude and latitude information of the first target image and the timestamp information and the longitude and latitude information of the second target image;

[0090] The capture time interval matrix and the capture space distance matrix are stacked to obtain a space-time matrix;

[0091] and, using the attribute information of the first target image and the attribute information of the second target image, determining an attribute matrix corresponding to the first target image and the second target image;

[0092] Inputting the space-time matrix into the first module to obtain a space-time coding matrix; and inputting the attribute matrix into the second module to obtain an attribute coding matrix;

[0093] The spatiotemporal coding matrix and the attribute coding matrix are stacked to obtain a mixed feature matrix;

[0094] Inputting the mixed feature matrix into the feature extraction layer of the second branch, and obtaining a second fusion feature after being processed by the feature extraction layer and the feature fusion layer of the second branch respectively;

[0095] Concatenate the first fusion feature and the second fusion feature to obtain a final fusion feature;

[0096] The final fusion feature is input into the classification layer to obtain the classification probabilities corresponding to the first target image and the second target image.

[0097] In some embodiments, the method further comprises: performing a K-nearest neighbor search on the plurality of image features to obtain a neighbor index matrix I;

[0098] Determining a similarity matrix between the first target image and the second target image based on an image feature of the first target image and an image feature of the second target image includes:

[0099] Based on the neighbor index matrix I, determine the first neighbor feature matrix F q and the second nearest neighbor feature matrix F g ; Wherein, the first neighbor feature matrix is ​​the image feature f of the first target image q The neighbor feature matrix of the second target image is f g The neighbor feature matrix of ;

[0100] A similarity matrix S0 between the first nearest neighbor feature matrix and the second nearest neighbor feature matrix is ​​calculated.

[0101] In some embodiments, determining the first neighbor feature matrix and the second neighbor feature matrix based on the neighbor index matrix includes:

[0102] From the neighbor index matrix, a preset number (for example, M=20) of neighbor features of the image features of the first target image are retrieved, and stacked to obtain a first neighbor feature matrix;

[0103] And, a preset number (eg, M=20) of neighbor features of the image features of the second target image are taken out from the neighbor index matrix, and stacked to obtain a second neighbor feature matrix.

[0104] In some embodiments, clustering the targets based on the classification probabilities corresponding to every two target images includes:

[0105] Generate a probability matrix P using the classification probabilities corresponding to each two target images;

[0106] For each classification probability p in the probability matrix, the classification probability is compared with the preset threshold p T Compare, if the classification probability is greater than the preset threshold, then reset the classification probability to 1, otherwise, reset the classification probability to 0;

[0107] A K-nearest neighbor adjacency matrix is ​​generated using the reset value of each classification probability in the probability matrix.

[0108] The K-nearest neighbor adjacency matrix is ​​used to achieve target clustering.

[0109] In some embodiments, using the K-nearest neighbor adjacency matrix to achieve target clustering includes:

[0110] For each element in the K-nearest neighbor adjacency matrix, when the element is 1, it is determined that the two image features corresponding to the element belong to the image features of the same target; otherwise, it is determined that the two image features corresponding to the element do not belong to the image features of the same target.

[0111] Embodiment 2:

[0112] For target clustering based only on image features, see Figure 4 , the clustering method provided in this embodiment includes:

[0113] S401, calculating the similarity matrix between every two target images;

[0114] For example, see Figure 5 , if there are 5 target images, 5 image features can be obtained, such as Figure 5 The nodes 1 to 5 shown represent an image feature respectively.

[0115] S402, predicting the classification probability of every two target images;

[0116] See Figure 5 ,In this step, the similarity matrix is ​​input into the clustering neural network model, and the corresponding classification probability is output;

[0117] S403: Perform target clustering based on the classification probability of every two target images.

[0118] When the number of features N is large, the time complexity of predicting the connection relationship of all N features (i.e., the adjacency matrix A) is too high, and such a scenario is difficult to have in reality. Therefore, this embodiment chooses to predict the connection relationship of the K-nearest neighbor graph (i.e., the K-nearest neighbor adjacency matrix, the value of K is related to the actual number of images of the same person, for example, K=100). Therefore, this embodiment only requires N×K prediction quantities.

[0119] Specifically, see Figure 6 , step S401 includes:

[0120] S601. Perform K-nearest neighbor search on all N features to obtain the nearest neighbor index matrix I∈R N×K ;

[0121] Among them, the nth row vector i of the neighbor index matrix I n , indicating the same characteristic f n The feature indexes of the top K in descending order of similarity.

[0122] S602: Determine the feature f of the first target image according to the index matrix I. q The neighbor feature matrix F q , and the feature f of the second target image g The neighbor feature matrix F g .

[0123] Assume that we need to judge feature f q Represents the qth node and feature f g (g∈i n ) represents whether the g-th node is connected, that is, predicting f q With f g Whether the image features belong to the same target;

[0124] According to the index matrix I, take out the feature f q The M nearest neighbor features , and stack them to get the neighbor feature matrix F q ∈R M×D ;

[0125] Similarly, we get the feature f g The neighbor feature matrix F g ∈R M×D ;

[0126] The value of M is related to the actual number of images of the same person, for example, M=20.

[0127] Each feature is a vector of length D, where D is a preset value, such as 64, 128, or 256.

[0128] S603, calculate the neighbor feature matrix F q and the neighbor feature matrix F g The similarity matrix S0 is:

[0129] .

[0130] The specific similarity matrix S0 is, for example Figure 7 As shown, the element S 23 For example, S 23 express Figure 5 The similarity between node 2 and node 3 in , that is, the possibility of node 2 and node 3 being connected.

[0131] Step S402 predicts the classification probability of each two target images, that is, performs forward inference of the neural network to obtain the classification probability of each two images, that is, the connection probability between each two image features. Figure 8 , step S402 includes:

[0132] S801, through the encoding layer, the feature f q and feature f g The corresponding similarity matrix S0 is encoded to obtain the similarity encoding matrix:

[0133] S1∈R C×M×M ;

[0134] Among them, C represents the similarity encoding feature dimension.

[0135] The encoding layer described in the embodiment of the present application is, for example, composed of a layer of convolutional neural network, whose convolution kernel size is 1×1; the activation function is GELU (other activation functions, such as ReLU, SELU, etc. may also be used).

[0136] S802. Perform feature extraction on the similarity coding matrix through a feature extraction layer to obtain L features, and stack the L features into a feature matrix.

[0137] The feature extraction layer described in the embodiment of the present application is, for example, composed of L (for example, L=5) convolutional neural networks arranged side by side, and each convolutional neural network contains a convolution layer and a global maximum pooling layer.

[0138] The convolution kernel sizes of the five convolutional neural networks of the feature extraction layer are, for example, 1×1, 5×5, 9×9, 15×15, and 19×19, respectively;

[0139] The activation function of the feature extraction layer is ReLU (other activation functions, such as SELU, GELU, etc., may also be used).

[0140] Similarity encoding matrix S1∈RC×M×M , after the feature extraction layer, L features are obtained:

[0141] ;

[0142] Here, C1 represents a new channel dimension, which is a preset value, such as 64, 128, or 256.

[0143] Then, the L features are stacked into a feature matrix:

[0144] .

[0145] S803. A feature matrix formed by stacking multiple features is fused through a feature fusion layer to obtain a fused feature.

[0146] Since the feature extraction layer uses convolution kernels of different sizes to learn feature expressions with different numbers of neighbors, in order to comprehensively learn the features under different numbers of neighbors, this embodiment fuses them into a more robust feature for subsequent classification tasks.

[0147] The feature fusion layer described in this embodiment first uses a two-dimensional convolution to map different feature spaces to a unified feature space. The convolution kernel size of the feature fusion layer is, for example, 1×1, and the activation function is, for example, LeakyReLU.

[0148] Use one-dimensional convolution on the feature matrix X to calculate the attention β∈R L , whose convolution kernel size is C1.

[0149] Using attention β, the feature matrix X is weighted and the fusion feature is obtained:

[0150]

[0151] In some embodiments, the attention-based feature fusion method may also be other methods, such as:

[0152] Input the feature matrix X into the encoder to get the updated feature matrix:

[0153] ;

[0154] Calculate the channel mean of the updated feature matrix as the fusion feature:

[0155] .

[0156] S804, input the fusion feature into the classification layer to obtain the feature f q and feature f g The classification probability of .

[0157] The classification layer described in the embodiment of the present application is, for example, composed of a linear layer and a softmax operator.

[0158] The fusion features After being sent to the classification layer, the feature f is obtained q and feature f g The classification probability p.

[0159] That is to say, through the above step S402, the classification probability corresponding to the first target image and the second target image (i.e., the connection probability between the two features) can be obtained for the first target image and the second target image in each of the two target images, which is used to determine whether the first target image and the second target image belong to the same target image.

[0160] Regarding step S403, target clustering is performed based on the classification probability of each two target images. The classification probability of each two target images is output through a clustering neural network model including a coding layer, a feature extraction layer, a feature fusion layer and a classification layer. Based on these classification probabilities, a connection relationship prediction matrix of the K-nearest neighbor graph of all N features is obtained, that is, a probability matrix P∈R N×K , where the element P nk Represents the feature f n With features The connection probability, i nk It is the element in the nth row and kth column of the neighbor index matrix I.

[0161] Preset threshold p T , p T It needs to be selected according to the model false alarm rate and actual situation, for example, p T =0.7.

[0162] If the connection probability of two features is greater than the threshold p T , then it is determined that the two features belong to the same category, and the K-nearest neighbor adjacency matrix is ​​obtained. Finally, all the nodes with connection relationships are connected to form a subgraph, and the clustering result is obtained.

[0163] For example: N=5, K=3;

[0164] There are N=5 features, and the similarity matrix of these five features is as follows:

[0165]

[0166] The neighbor index matrix I with K=3 is calculated:

[0167]

[0168] Among them, taking the first row in the neighbor index matrix I as an example, 1, 3, and 2 respectively represent the feature indices that are ranked in descending order in similarity with the feature f1, that is, referring to the first row of the similarity matrix, take the three largest similarities, namely 1, 0.9, and 0.95. The indexes of the rows of these three similarities in the similarity matrix are 1, 2, and 3, respectively. Then, if they are sorted in descending order of similarity, they are 1, 0.95, and 0.9, and the corresponding indexes are 1, 3, and 2, respectively, thereby obtaining the content of the first row in the neighbor index matrix I.

[0169] Correspondingly, the probability matrix P:

[0170]

[0171] By setting the probability threshold p T = 0.7, and we get the K-nearest neighbor adjacency matrix:

[0172]

[0173] Among them, regarding the K-nearest neighbor adjacency matrix, for example, the first row of elements in the K-nearest neighbor adjacency matrix corresponds to the first row of elements 1, 0.8, and 0.95 in the probability matrix P. The values ​​of these three elements are all greater than the probability threshold p. T , so the elements in the first row of the K-nearest neighbor adjacency matrix are all 1.

[0174] Finally, we get the clustering results {1, 2, 3}, {4, 5}, that is, we get two clusters, where the first cluster includes features 1, 2, 3, and the second cluster includes features 4, 5. In other words, the features in the first cluster belong to one target object, such as the image features of the same person; the features in the second cluster belong to another target object, such as the image features of another person.

[0175] Embodiment three:

[0176] In the field of public security, when a camera captures a human image, it will store the timestamp and the current GPS longitude and latitude information. At the same time, the captured human image will be analyzed through the human attribute network of the video structured analysis to output the human body type. It can be seen that the comprehensive use of multimodal information can enable the model to learn more complex data distribution.

[0177] See also Fig. 9 In this embodiment, a multimodal clustering neural network is preset, which includes a coding layer, a feature extraction layer, a feature fusion layer of a first branch, and a coding layer, a feature extraction layer, a feature fusion layer, and a classification layer of a second branch.

[0178] The relevant processing process of the first branch of the clustering neural network in this embodiment is the same as that of the above-mentioned embodiment 2, and will not be repeated here. The relevant content of the newly added network branch, namely, the second branch, will be specifically introduced in detail.

[0179] 1. Input the spatiotemporal information and attribute information into the second branch of the clustering neural network:

[0180] For each of the first target image and the second target image of the two target images:

[0181] Assume that the feature f of the first target image is q The M nearest neighbor features of are , the feature f of the second target image g The M nearest neighbor features of are .

[0182] First, according to the timestamps and longitude and latitude of the first target image and the second target image, the capture time interval matrix T∈R of the first target image and the second target image can be calculated: 1×M×M and the snapshot space distance matrix G∈R 1×M×M ;

[0183] The capture time interval matrix T and the capture space distance matrix G of the first target image and the second target image are stacked in the channel dimension to obtain the spatiotemporal matrix H∈R of the first target image and the second target image 2×M×M .

[0184] And, determine the attribute matrix corresponding to the first target image and the second target image, that is, construct f q and f g The corresponding attribute matrix A∈R 2×M×M , the attribute feature includes two layers of vectors, where the element a in the first layer of vectors 1,i,j , indicating f q The attribute of the i-th feature in the neighboring features; the element a in the second layer vector 2,i,j , indicating f g The attribute of the jth feature among the neighboring features of ; where i=1,2,…,M; j=1,2,…,M.

[0185] 2. Neural Network Forward Reasoning:

[0186] The encoding layer of the new network branch contains two modules: the first module and the second module. The first module processes the input spatiotemporal matrix H, and the second module processes the input attribute matrix A.

[0187] The encoding layer of the first module first encodes H to obtain the spatiotemporal encoding matrix H1∈R CH×M×M , where C HIndicates the preset spatiotemporal coding feature dimension (for example, 64, 128, 256, etc.). The coding layer consists of a convolutional neural network with a convolution kernel size of 1×1 and an activation function of GELU (other activation functions such as ReLU, SELU, etc. can also be used).

[0188] The encoding layer of the second module embeds the discrete attribute values ​​and encodes a feature vector a∈R of length E for each attribute. E , get the attribute encoding matrix A1∈R 2E×M×M .

[0189] The discrete attribute values, for example, 1 represents a pedestrian, 2 represents a cyclist, 3 represents a motorcycle rider, etc., assuming a total of U attributes.

[0190] Finally, the encoding spatiotemporal matrix H1 and the encoding attribute matrix A1 are stacked in the channel to form a mixed feature matrix W∈R (CH +2E)×2M×2M , as the input of the feature extraction layer of the new network branch.

[0191] The structures of the feature extraction layer and the feature fusion layer of the new network branch are respectively the same as the structures of the feature extraction layer and the feature fusion layer in the above-mentioned embodiment 2, so the specific processing process will not be repeated in this embodiment.

[0192] Assume that the fusion features obtained by the two branches are and , concatenate these two fusion features into one feature to obtain the fusion feature :

[0193] .

[0194] The fusion features After being sent to the classification layer, the classification probability p is obtained.

[0195] Wherein, the classification layer is composed of a linear layer and a softmax operator.

[0196] In summary, in the embodiment, human body clustering can comprehensively utilize human body image features, human body detection attributes, and time-space distance information to obtain clustering results. If only human body features are used, and human body attributes and time-space information are not fully utilized, the recall rate of human body clustering will be low. Therefore, the multimodal clustering model provided in the embodiment of the present application comprehensively utilizes human body features, attributes, time-space information, etc., to improve the recall rate of human body clustering.

[0197] The following is an introduction to the equipment or device provided in the embodiments of the present application, in which the explanations or examples of technical features that are the same or corresponding to those described in the above method will not be repeated later.

[0198] An electronic device provided in an embodiment of the present application, see Fig.10 , for example:

[0199] The processor 600 is used to read the program in the memory 620 and execute the following process:

[0200] Acquire multiple target images, and extract image features of each of the target images to obtain multiple image features;

[0201] For a first target image and a second target image in each of the two target images: based on the image features of the first target image and the image features of the second target image, determine a similarity matrix between the first target image and the second target image; input the similarity matrix into a preset clustering neural network, and obtain the classification probabilities corresponding to the first target image and the second target image through the clustering neural network, wherein the classification probabilities are used to determine whether the first target image and the second target image belong to the same target image;

[0202] Target clustering is performed based on the classification probability corresponding to every two target images.

[0203] In some embodiments, the clustering neural network includes: a coding layer, a feature extraction layer, a feature fusion layer, and a classification layer;

[0204] Obtaining classification probabilities corresponding to the first target image and the second target image through the clustering neural network includes:

[0205] Encoding the similarity matrix between the first target image and the second target image through the encoding layer to obtain a similarity encoding matrix;

[0206] Performing feature extraction on the similarity encoding matrix through the feature extraction layer to obtain at least one feature, and stacking the at least one feature into a feature matrix;

[0207] The feature matrix is ​​fused through the feature fusion layer to obtain fused features;

[0208] The fused features are input into a classification layer, and the classification layer is used to obtain classification probabilities corresponding to the first target image and the second target image.

[0209] In some embodiments, the encoding layer, feature extraction layer, and feature fusion layer belong to the first branch of the clustering neural network; the fusion feature is the first fusion feature; the clustering neural network further includes an encoding layer, a feature extraction layer, and a feature fusion layer of a second branch; wherein the encoding layer of the second branch includes a first module and a second module;

[0210] The processor 600 is further configured to read the program in the memory 620 and execute the following process:

[0211] Determining the timestamp information and the longitude and latitude information of the first target image, and the timestamp information and the longitude and latitude information of the second target image;

[0212] Determine a capture time interval matrix and a capture space distance matrix of the first target image and the second target image based on the timestamp information and the longitude and latitude information of the first target image and the timestamp information and the longitude and latitude information of the second target image;

[0213] The capture time interval matrix and the capture space distance matrix are stacked to obtain a space-time matrix;

[0214] and, using the attribute information of the first target image and the attribute information of the second target image, determining an attribute matrix corresponding to the first target image and the second target image;

[0215] Inputting the space-time matrix into the first module to obtain a space-time coding matrix; and inputting the attribute matrix into the second module to obtain an attribute coding matrix;

[0216] The spatiotemporal coding matrix and the attribute coding matrix are stacked to obtain a mixed feature matrix;

[0217] Inputting the mixed feature matrix into the feature extraction layer of the second branch, and obtaining a second fusion feature after being processed by the feature extraction layer and the feature fusion layer of the second branch respectively;

[0218] Concatenate the first fusion feature and the second fusion feature to obtain a final fusion feature;

[0219] The final fusion feature is input into the classification layer to obtain the classification probabilities corresponding to the first target image and the second target image.

[0220] In some embodiments, the processor 600 is further configured to read a program in the memory 620 and execute the following process: perform a K-nearest neighbor search on the plurality of image features to obtain a neighbor index matrix;

[0221] Determining a similarity matrix between the first target image and the second target image based on an image feature of the first target image and an image feature of the second target image includes:

[0222] Based on the neighbor index matrix, determine a first neighbor feature matrix and a second neighbor feature matrix; wherein the first neighbor feature matrix is ​​a neighbor feature matrix of image features of the first target image; and the second neighbor feature matrix is ​​a neighbor feature matrix of image features of the second target image;

[0223] A similarity matrix between the first neighbor feature matrix and the second neighbor feature matrix is ​​calculated.

[0224] In some embodiments, determining the first neighbor feature matrix and the second neighbor feature matrix based on the neighbor index matrix includes:

[0225] From the neighbor index matrix, a preset number of neighbor features of the image features of the first target image are taken out, and stacked to obtain a first neighbor feature matrix;

[0226] And, a preset number of neighbor features of the image features of the second target image are taken out from the neighbor index matrix, and stacked to obtain a second neighbor feature matrix.

[0227] In some embodiments, clustering the targets based on the classification probabilities corresponding to every two target images includes:

[0228] Generate a probability matrix using the classification probabilities corresponding to every two target images;

[0229] For each classification probability in the probability matrix, compare the classification probability with a preset threshold, if the classification probability is greater than the preset threshold, set the reset value of the classification probability to 1, otherwise, set the reset value of the classification probability to 0;

[0230] A K-nearest neighbor adjacency matrix is ​​generated using the reset value of each classification probability in the probability matrix.

[0231] The K-nearest neighbor adjacency matrix is ​​used to achieve target clustering.

[0232] In some embodiments, using the K-nearest neighbor adjacency matrix to achieve target clustering includes:

[0233] For each element in the K-nearest neighbor adjacency matrix, when the element is 1, it is determined that the two image features corresponding to the element belong to the image features of the same target; otherwise, it is determined that the two image features corresponding to the element do not belong to the image features of the same target.

[0234] The transceiver 610 is configured to receive and send data under the control of the processor 600 .

[0235] Among them, Fig.10In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically one or more processors represented by processor 600 and various circuits of memory represented by memory 620 are linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 610 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium, including wireless channels, wired channels, optical cables, and other transmission media. For different user devices, the user interface 630 may also be an interface capable of externally and internally connecting required devices, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, and the like.

[0236] The processor 600 is responsible for managing the bus architecture and general processing, and the memory 620 can store data used by the processor 600 when performing operations.

[0237] In some embodiments, the processor 600 may be a CPU (central processing unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or a CPLD (Complex Programmable Logic Device), and the processor may also adopt a multi-core architecture.

[0238] The processor calls the computer program stored in the memory to execute any of the methods provided in the embodiments of the present application according to the obtained executable instructions. The processor and the memory can also be arranged physically separately.

[0239] It should be noted here that the above-mentioned device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0240] See also Fig.11 , a target clustering device provided in an embodiment of the present application includes:

[0241] The first unit 11 is used to obtain a plurality of target images and extract image features of each of the target images to obtain a plurality of image features;

[0242] The second unit 12 is used for: for each of the first target image and the second target image in each of the two target images: determining a similarity matrix between the first target image and the second target image based on image features of the first target image and image features of the second target image; inputting the similarity matrix into a preset clustering neural network, and obtaining classification probabilities corresponding to the first target image and the second target image through the clustering neural network, wherein the classification probabilities are used to determine whether the first target image and the second target image belong to the same target image;

[0243] The third unit 13 is used to perform target clustering based on the classification probability corresponding to every two target images.

[0244] In some embodiments, the clustering neural network includes: a coding layer, a feature extraction layer, a feature fusion layer, and a classification layer;

[0245] Obtaining classification probabilities corresponding to the first target image and the second target image through the clustering neural network includes:

[0246] Encoding the similarity matrix between the first target image and the second target image through the encoding layer to obtain a similarity encoding matrix;

[0247] Performing feature extraction on the similarity encoding matrix through the feature extraction layer to obtain at least one feature, and stacking the at least one feature into a feature matrix;

[0248] The feature matrix is ​​fused through the feature fusion layer to obtain fused features;

[0249] The fused features are input into a classification layer, and the classification layer is used to obtain classification probabilities corresponding to the first target image and the second target image.

[0250] In some embodiments, the encoding layer, feature extraction layer, and feature fusion layer belong to the first branch of the clustering neural network; the fusion feature is the first fusion feature; the clustering neural network further includes an encoding layer, a feature extraction layer, and a feature fusion layer of a second branch; wherein the encoding layer of the second branch includes a first module and a second module;

[0251] The second unit 12 is also used for:

[0252] Determining the timestamp information and the longitude and latitude information of the first target image, and the timestamp information and the longitude and latitude information of the second target image;

[0253] Determine a capture time interval matrix and a capture space distance matrix of the first target image and the second target image based on the timestamp information and the longitude and latitude information of the first target image and the timestamp information and the longitude and latitude information of the second target image;

[0254] The capture time interval matrix and the capture space distance matrix are stacked to obtain a space-time matrix;

[0255] and, using the attribute information of the first target image and the attribute information of the second target image, determining an attribute matrix corresponding to the first target image and the second target image;

[0256] Inputting the space-time matrix into the first module to obtain a space-time coding matrix; and inputting the attribute matrix into the second module to obtain an attribute coding matrix;

[0257] The spatiotemporal coding matrix and the attribute coding matrix are stacked to obtain a mixed feature matrix;

[0258] Inputting the mixed feature matrix into the feature extraction layer of the second branch, and obtaining a second fusion feature after being processed by the feature extraction layer and the feature fusion layer of the second branch respectively;

[0259] Concatenate the first fusion feature and the second fusion feature to obtain a final fusion feature;

[0260] The final fusion feature is input into the classification layer to obtain the classification probabilities corresponding to the first target image and the second target image.

[0261] In some embodiments, the second unit 12 is further used to: perform a K-nearest neighbor search on the plurality of image features to obtain a neighbor index matrix;

[0262] Determining a similarity matrix between the first target image and the second target image based on an image feature of the first target image and an image feature of the second target image includes:

[0263] Based on the neighbor index matrix, determine a first neighbor feature matrix and a second neighbor feature matrix; wherein the first neighbor feature matrix is ​​a neighbor feature matrix of image features of the first target image; and the second neighbor feature matrix is ​​a neighbor feature matrix of image features of the second target image;

[0264] A similarity matrix between the first neighbor feature matrix and the second neighbor feature matrix is ​​calculated.

[0265] In some embodiments, determining the first neighbor feature matrix and the second neighbor feature matrix based on the neighbor index matrix includes:

[0266] From the neighbor index matrix, a preset number of neighbor features of the image features of the first target image are taken out, and stacked to obtain a first neighbor feature matrix;

[0267] And, a preset number of neighbor features of the image features of the second target image are taken out from the neighbor index matrix, and stacked to obtain a second neighbor feature matrix.

[0268] In some embodiments, clustering the targets based on the classification probabilities corresponding to every two target images includes:

[0269] Generate a probability matrix using the classification probabilities corresponding to every two target images;

[0270] For each classification probability in the probability matrix, compare the classification probability with a preset threshold, if the classification probability is greater than the preset threshold, set the reset value of the classification probability to 1, otherwise, set the reset value of the classification probability to 0;

[0271] A K-nearest neighbor adjacency matrix is ​​generated using the reset value of each classification probability in the probability matrix.

[0272] The K-nearest neighbor adjacency matrix is ​​used to achieve target clustering.

[0273] In some embodiments, using the K-nearest neighbor adjacency matrix to achieve target clustering includes:

[0274] For each element in the K-nearest neighbor adjacency matrix, when the element is 1, it is determined that the two image features corresponding to the element belong to the image features of the same target; otherwise, it is determined that the two image features corresponding to the element do not belong to the image features of the same target.

[0275] It should be noted that the division of units in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0276] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0277] Any of the devices or apparatuses provided in the embodiments of the present application may specifically be a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant (PDA), etc. It may include a central processing unit (CPU), a memory, an input / output device, etc. The input device may include a keyboard, a mouse, a touch screen, etc. The output device may include a display device, such as a liquid crystal display (LCD), a cathode ray tube (CRT), etc.

[0278] The memory may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor with program instructions and data stored in the memory. In the embodiment of the present application, the memory may be used to store the program of any of the methods provided in the embodiment of the present application.

[0279] The processor calls the program instructions stored in the memory, and the processor is used to execute any of the methods provided in the embodiments of the present application according to the obtained program instructions.

[0280] The present application embodiment also provides a computer program product or computer program, which includes a computer instruction, which is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction so that the computer device performs any of the methods described in the above embodiments. The program product can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0281] The embodiment of the present application provides a computer-readable storage medium for storing computer program instructions used by the apparatus provided in the embodiment of the present application, which includes a program for executing any of the methods provided in the embodiment of the present application. The computer-readable storage medium may be a non-transitory computer-readable medium.

[0282] The computer-readable storage medium can be any available medium or data storage device that can be accessed by a computer, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD)), etc.

[0283] It should be understood that:

[0284] The access technology through which entities in the communication network transmit traffic can be any suitable current or future technology, such as WLAN (Wireless Local Access Network), WiMAX (Worldwide Interoperability for Microwave Access), LTE, LTE-A, 5G, Bluetooth, infrared, etc.; in addition, the embodiments can also apply wired technology, for example, IP-based access technology, such as a wired network or a fixed line.

[0285] Embodiments suitable for being implemented as software code or a portion thereof and running using a processor or processing functionality are independent of the software code and may be specified using any known or future developed programming language, such as a high-level programming language such as objective-C, C, C++, C#, Java, Python, Javascript, other scripting languages, etc., or a low-level programming language such as machine language or assembler.

[0286] The implementation of the embodiments is hardware independent and may be implemented using any known or future developed hardware technology or any mixture thereof, such as a microprocessor or CPU (central processing unit), MOS (metal oxide semiconductor), CMOS (complementary MOS), BiMOS (bipolar MOS), BiCMOS (bipolar CMOS), ECL (emitter coupled logic) and / or TTL (transistor-transistor logic).

[0287] Embodiments may be implemented as separate devices, apparatuses, units, components, or functions, or in a distributed manner, for example, one or more processors or processing functions may be used or shared in a process, or one or more processing segments or processing portions may be used and shared in a process, wherein one physical processor or more than one physical processor may be used to implement one or more processing portions dedicated to a specific process as described.

[0288] The device may be implemented by a semiconductor chip, a chipset, or a (hardware) module including such a chip or chipset.

[0289] The embodiments may also be implemented as any combination of hardware and software, such as ASIC (Application Specific IC (Integrated Circuit)) components, FPGA (Field Programmable Gate Array) or CPLD (Complex Programmable Logic Device) components or DSP (Digital Signal Processor) components.

[0290] The embodiments may also be implemented as a computer program product including a computer usable medium having computer readable program code embodied therein, the computer readable program code being adapted to perform the processes as described in the embodiments, wherein the computer usable medium may be a non-transitory medium.

[0291] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program codes.

[0292] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0293] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0294] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0295] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A target clustering method, characterized in that: The method comprises: Acquire multiple target images, and extract image features of each of the target images to obtain multiple image features; For a first target image and a second target image in each of the two target images: based on the image features of the first target image and the image features of the second target image, determine a similarity matrix between the first target image and the second target image; input the similarity matrix into a preset clustering neural network, and obtain the classification probabilities corresponding to the first target image and the second target image through the clustering neural network, wherein the classification probabilities are used to determine whether the first target image and the second target image belong to the same target image; Target clustering is performed based on the classification probability corresponding to every two target images; Wherein, the clustering neural network comprises a first branch, a second branch and a classification layer; wherein: The first branch includes a coding layer, a feature extraction layer, and a feature fusion layer, and is used to input a similarity matrix of the first target image and the second target image, and output a first fusion feature after being processed by the coding layer, the feature extraction layer, and the feature fusion layer; The second branch includes another encoding layer, a feature extraction layer, and a feature fusion layer, and is used to input the capture time interval matrix and the capture space distance matrix of the first target image and the second target image, and output a second fusion feature after being processed by the another encoding layer, the feature extraction layer, and the feature fusion layer; After stacking the first fusion feature and the second fusion feature, a final fusion feature is obtained, and the final fusion feature is input into the classification layer, and the classification probabilities corresponding to the first target image and the second target image are obtained through the classification layer.

2. The method according to claim 1, characterized in that Outputting the first fusion feature includes: Encoding the similarity matrix between the first target image and the second target image through the encoding layer in the first branch to obtain a similarity encoding matrix; Performing feature extraction on the similarity encoding matrix through the feature extraction layer in the first branch to obtain at least one feature, and stacking the at least one feature into a feature matrix; The feature matrix is ​​fused through the feature fusion layer in the first branch to obtain the first fused feature.

3. The method according to claim 2, characterized in that The coding layer in the second branch includes a first module and a second module; Outputting the second fusion feature includes: Determining the timestamp information and the longitude and latitude information of the first target image, and the timestamp information and the longitude and latitude information of the second target image; Determine a capture time interval matrix and a capture space distance matrix of the first target image and the second target image based on the timestamp information and the longitude and latitude information of the first target image and the timestamp information and the longitude and latitude information of the second target image; The capture time interval matrix and the capture space distance matrix are stacked to obtain a spatiotemporal matrix; the attribute matrices corresponding to the first target image and the second target image are determined by using the attribute information of the first target image and the attribute information of the second target image; Input the space-time matrix into the first module to obtain a space-time coding matrix; input the attribute matrix into the second module to obtain an attribute coding matrix; The spatiotemporal coding matrix and the attribute coding matrix are stacked to obtain a mixed feature matrix; The mixed feature matrix is ​​input into the feature extraction layer of the second branch, and is processed by the feature extraction layer and the feature fusion layer of the second branch respectively to obtain a second fusion feature.

4. The method according to claim 1, characterized in that: The method further comprises: performing a K-nearest neighbor search on the plurality of image features to obtain a neighbor index matrix; Determining a similarity matrix between the first target image and the second target image based on an image feature of the first target image and an image feature of the second target image includes: Based on the neighbor index matrix, determine a first neighbor feature matrix and a second neighbor feature matrix; wherein the first neighbor feature matrix is ​​a neighbor feature matrix of image features of the first target image; and the second neighbor feature matrix is ​​a neighbor feature matrix of image features of the second target image; A similarity matrix between the first neighbor feature matrix and the second neighbor feature matrix is ​​calculated.

5. The method according to claim 4, characterized in that The determining, based on the neighbor index matrix, a first neighbor feature matrix and a second neighbor feature matrix comprises: From the neighbor index matrix, a preset number of neighbor features of the image features of the first target image are taken out, and stacked to obtain a first neighbor feature matrix; And, a preset number of neighbor features of the image features of the second target image are taken out from the neighbor index matrix, and stacked to obtain a second neighbor feature matrix.

6. The method according to claim 1, characterized in that The target clustering is performed based on the classification probability corresponding to each two target images, including: Generate a probability matrix using the classification probabilities corresponding to every two target images; For each classification probability in the probability matrix, compare the classification probability with a preset threshold, if the classification probability is greater than the preset threshold, set the reset value of the classification probability to 1, otherwise, set the reset value of the classification probability to 0; Generate a K-nearest neighbor adjacency matrix using the reset value of each classification probability in the probability matrix; The K-nearest neighbor adjacency matrix is ​​used to achieve target clustering.

7. The method according to claim 6, characterized in that Using the K-nearest neighbor adjacency matrix, target clustering is achieved, including: For each element in the K-nearest neighbor adjacency matrix, when the element is 1, it is determined that the two image features corresponding to the element belong to the image features of the same target; otherwise, it is determined that the two image features corresponding to the element do not belong to the image features of the same target.

8. A target clustering device, characterized in that: The device comprises: The first unit is used to acquire a plurality of target images and extract image features of each of the target images to obtain a plurality of image features; The second unit is used for determining, for each of the first target image and the second target image in the two target images, a similarity matrix between the first target image and the second target image based on image features of the first target image and image features of the second target image; inputting the similarity matrix into a preset clustering neural network, and obtaining classification probabilities corresponding to the first target image and the second target image through the clustering neural network, wherein the classification probabilities are used to determine whether the first target image and the second target image belong to the same target image; The third unit is used for clustering targets based on the classification probabilities corresponding to every two target images; Wherein, the clustering neural network comprises a first branch, a second branch and a classification layer; wherein: The first branch includes a coding layer, a feature extraction layer, and a feature fusion layer, and is used to input a similarity matrix of the first target image and the second target image, and output a first fusion feature after being processed by the coding layer, the feature extraction layer, and the feature fusion layer; The second branch includes another encoding layer, a feature extraction layer, and a feature fusion layer, and is used to input the capture time interval matrix and the capture space distance matrix of the first target image and the second target image, and output a second fusion feature after being processed by the another encoding layer, the feature extraction layer, and the feature fusion layer; After stacking the first fusion feature and the second fusion feature, a final fusion feature is obtained, and the final fusion feature is input into the classification layer, and the classification probabilities corresponding to the first target image and the second target image are obtained through the classification layer.

9. An electronic device, characterized in that: include: A memory for storing program instructions; A processor, configured to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 7 according to the obtained program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable the computer to execute the method according to any one of claims 1 to 7.

Images