Image clustering method, image clustering model training method and electronic device

Abstract

The present application relates to an image clustering method, an image clustering model training method and a computer device. The method comprises: obtaining a plurality of files to be clustered; the files to be clustered comprise a plurality of images to be clustered; obtaining first spatiotemporal information corresponding to the images to be clustered, and a first mapping relationship between the first spatiotemporal information and each file to be clustered; the first spatiotemporal information comprises the shooting time of the images to be clustered and the shooting location of the images to be clustered; constructing first graph structure data according to the first mapping relationship, the first spatiotemporal information, and the correlation relationship between the first spatiotemporal information; determining the first similarity between the files to be clustered according to the first graph structure data; clustering the files to be clustered according to the first similarity between the files to be clustered, and adopting the present method to improve the accuracy of clustering the files to be clustered.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an image clustering method, an image clustering model training method and an electronic device. Background Art

[0002] With the widespread popularity of intelligent video surveillance equipment, a huge amount of target object images are accumulated every day. Image clustering can be used to create archives. Archive clustering can be simply understood as archiving images involving similar target objects in one archive, which can be used for subsequent investigations.

[0003] At present, the archive clustering method provided is: obtaining the image features of the target object, clustering the images of multiple target objects according to the similarity between the image features, so as to aggregate the images of the same target object into the same archive. However, when the monitoring equipment misses to capture the image or the captured image quality is low, the accuracy of archive clustering is low. Summary of the invention

[0004] Based on this, it is necessary to provide an image clustering method, an image clustering model training method and an electronic device to address the above technical problems.

[0005] In a first aspect, the present application provides an image clustering method. The method comprises:

[0006] Acquire a plurality of files to be clustered; the files to be clustered include a plurality of images to be clustered;

[0007] Acquire first spatiotemporal information corresponding to the image to be clustered, and a first mapping relationship between the first spatiotemporal information and each of the archives to be clustered; the first spatiotemporal information includes the shooting time and the shooting location of the image to be clustered;

[0008] Constructing first graph structure data according to the first mapping relationship, the first spatiotemporal information, and the correlation relationship between the first spatiotemporal information;

[0009] Determining a first similarity between the archives to be clustered according to the first graph structure data;

[0010] The files to be clustered are clustered according to the first similarity between the files to be clustered.

[0011] In one embodiment, determining the first similarity between the to-be-clustered archives according to the first graph structure data includes:

[0012] Inputting the first graph structure data into the trained image clustering model to obtain a graph embedding vector corresponding to each of the to-be-clustered files;

[0013] According to the graph embedding vector corresponding to each of the to-be-clustered files, a first similarity between the to-be-clustered files is determined.

[0014] In one embodiment, the image clustering model is trained in the following manner:

[0015] Acquire a plurality of historical archives; the historical archives include a plurality of images to be trained;

[0016] Acquire second spatiotemporal information corresponding to the image to be trained, and a second mapping relationship between the second spatiotemporal information and each of the historical archives; the second spatiotemporal information includes the shooting time and the shooting location of the image to be trained;

[0017] Constructing second graph structure data according to the second mapping relationship, the second spatiotemporal information, and the correlation relationship between the second spatiotemporal information;

[0018] The image clustering model is trained according to the second graph structure data to obtain a trained image clustering model.

[0019] In one of the embodiments, the image clustering model includes at least two different types of graph neural network models, and the graph neural network models are used to extract different types of edge features and different types of node features in the second graph structure data respectively;

[0020] The step of training the image clustering model according to the second graph structure data to obtain the trained image clustering model includes:

[0021] Inputting the second graph structure data into the graph neural network model respectively to obtain a plurality of graph embedding vectors corresponding to the nodes in the second graph structure data;

[0022] Determining a predicted value of each edge in the second graph structure data according to a plurality of graph embedding vectors corresponding to nodes in the second graph structure data;

[0023] Obtaining the true value of each edge in the second graph structure data;

[0024] Obtaining a loss value of each edge in the second graph structure data according to the predicted value and the true value;

[0025] The image clustering model is trained according to the loss value to obtain a trained image clustering model.

[0026] In one embodiment, determining the predicted value of each edge in the second graph structure data according to a plurality of graph embedding vectors corresponding to nodes in the second graph structure data includes:

[0027] Aggregating each graph embedding vector corresponding to a node in the second graph structure data to obtain an aggregated graph embedding vector;

[0028] Determine a predicted value of each edge in the second graph structure data according to the aggregated graph embedding vector.

[0029] In one embodiment, the step of obtaining the second spatiotemporal information corresponding to the image to be trained, and the second mapping relationship between the second spatiotemporal information and each of the historical archives, further includes:

[0030] Obtaining image features of each of the images to be trained;

[0031] Determining the initial similarity between the files to be clustered according to the image features of the images to be trained;

[0032] Constructing second graph structure data according to the initial similarity, the second mapping relationship, the second spatiotemporal information, and the correlation relationship between the second spatiotemporal information;

[0033] The image clustering model is trained according to the second graph structure data to obtain a trained image clustering model.

[0034] In one embodiment, the method further comprises:

[0035] Obtaining image features of each of the images to be clustered;

[0036] Determining a second similarity between the files to be clustered according to image features of the images to be clustered;

[0037] The to-be-clustered archives are clustered according to the first similarity and the second similarity.

[0038] In one of the embodiments, the first graph structure data includes a first spatiotemporal information node, a file node to be clustered, a time edge, a space edge and a spatiotemporal edge; the time edge is obtained by connecting a plurality of the first spatiotemporal information nodes having the same shooting location but different shooting times of the images to be clustered; the space edge is obtained by connecting a plurality of the first spatiotemporal information nodes having different shooting locations but the same shooting time of the images to be clustered; the spatiotemporal edge is obtained by connecting the file node to be clustered and the first spatiotemporal information node.

[0039] In a second aspect, the present application also provides an image clustering model training method. The method comprises:

[0040] Acquire a plurality of historical archives; the historical archives include a plurality of images to be trained;

[0041] Acquire second spatiotemporal information corresponding to the image to be trained, and a second mapping relationship between the second spatiotemporal information and each of the historical archives; the second spatiotemporal information includes the shooting time and the shooting location of the image to be trained;

[0042] Constructing second graph structure data according to the second mapping relationship, the second spatiotemporal information, and the correlation relationship between the second spatiotemporal information;

[0043] An image clustering model is trained according to the second graph structure data.

[0044] In a third aspect, the present application further provides an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0045] Acquire a plurality of files to be clustered; the files to be clustered include a plurality of images to be clustered;

[0046] Acquire first spatiotemporal information corresponding to the image to be clustered, and a first mapping relationship between the first spatiotemporal information and each of the archives to be clustered; the first spatiotemporal information includes the shooting time and the shooting location of the image to be clustered;

[0047] Constructing first graph structure data according to the first mapping relationship, the first spatiotemporal information, and the correlation relationship between the first spatiotemporal information;

[0048] Determining a first similarity between the archives to be clustered according to the first graph structure data;

[0049] The files to be clustered are clustered according to the first similarity between the files to be clustered.

[0050] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0051] Acquire a plurality of files to be clustered; the files to be clustered include a plurality of images to be clustered;

[0052] Acquire first spatiotemporal information corresponding to the image to be clustered, and a first mapping relationship between the first spatiotemporal information and each of the archives to be clustered; the first spatiotemporal information includes the shooting time and the shooting location of the image to be clustered;

[0053] Constructing first graph structure data according to the first mapping relationship, the first spatiotemporal information, and the correlation relationship between the first spatiotemporal information;

[0054] Determining a first similarity between the archives to be clustered according to the first graph structure data;

[0055] The files to be clustered are clustered according to the first similarity between the files to be clustered.

[0056] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0057] Acquire a plurality of files to be clustered; the files to be clustered include a plurality of images to be clustered;

[0058] Acquire first spatiotemporal information corresponding to the image to be clustered, and a first mapping relationship between the first spatiotemporal information and each of the archives to be clustered; the first spatiotemporal information includes the shooting time and the shooting location of the image to be clustered;

[0059] Constructing first graph structure data according to the first mapping relationship, the first spatiotemporal information, and the correlation relationship between the first spatiotemporal information;

[0060] Determining a first similarity between the archives to be clustered according to the first graph structure data;

[0061] The files to be clustered are clustered according to the first similarity between the files to be clustered.

[0062] The above-mentioned image clustering method, image clustering model training method and electronic device obtain multiple archives to be clustered; the archives to be clustered include multiple images to be clustered; obtain first spatiotemporal information corresponding to the images to be clustered, and a first mapping relationship between the first spatiotemporal information and each of the archives to be clustered; the first spatiotemporal information includes the shooting time of the images to be clustered and the shooting location of the images to be clustered; construct a first graph structure data according to the first mapping relationship, the first spatiotemporal information, and the correlation relationship between the first spatiotemporal information; when the monitoring device misses to capture an image or the captured image quality is low, according to the first graph structure data, the first similarity between the archives to be clustered can be more accurately determined; thereby, according to the first similarity between the archives to be clustered, the archives to be clustered can be more accurately clustered, solving the problem of low accuracy of archive clustering. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A schematic diagram of an application scenario of an image clustering method in an embodiment;

[0064] Figure 2 is a schematic diagram of a flow chart of an image clustering method in one embodiment;

[0065] Figure 3 is a schematic diagram of first graph structure data in one embodiment;

[0066] Figure 4 is a schematic flow chart of an image clustering method in another embodiment;

[0067] Figure 5 A schematic diagram of a process of an image clustering model training method in one embodiment;

[0068] Figure 6 is a schematic diagram of second graph structure data in one embodiment;

[0069] Figure 7 is a schematic diagram of a flow chart of an image clustering method in yet another embodiment;

[0070] Figure 8 A schematic diagram of a scene for training an image clustering model in one embodiment;

[0071] Fig. 9 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0073] Figure 1 A schematic diagram of an application scenario of an image clustering method in an embodiment.

[0074] like Figure 1 As shown, the application scenario shows a plurality of archives to be clustered 110 and an electronic device 120, each archive to be clustered 110 including a plurality of images to be clustered 130. The electronic device 120 can perform clustering processing on the plurality of archives to be clustered 110 according to the first spatiotemporal information of the plurality of images to be clustered 130. The first spatiotemporal information includes the shooting time of the images to be clustered and the shooting location of the images to be clustered. The electronic device 120 is, for example, a laptop computer, a server, a network hard disk recorder, and other devices capable of clustering processing, which is not limited in the present application.

[0075] Combine the following Figure 2 The clustering method of the present application is described.

[0076] Figure 2FIG. 1 is a flow chart of an image clustering method in an embodiment. The clustering method may be executed by the electronic device 120 , for example.

[0077] like Figure 2 As shown, the image clustering method includes the following steps:

[0078] Step 201, obtaining a plurality of files to be clustered; the files to be clustered include a plurality of images to be clustered.

[0079] Among them, the images to be clustered include but are not limited to face images, vehicle images and license plate images. This example can be applied to image clustering in various scenarios. For example, face images in a certain community can be clustered and the face images of a target person can be archived in one file. License plate images in a certain area can also be clustered and images of the same license plate can be archived in one file.

[0080] Step 202, obtaining first spatiotemporal information corresponding to the images to be clustered, and a first mapping relationship between the first spatiotemporal information and each archive to be clustered; the first spatiotemporal information includes the shooting time and the shooting location of the images to be clustered.

[0081] Each file to be clustered stores a plurality of images to be clustered. Through the first mapping relationship, it can be known when and where the images to be clustered are taken in each file.

[0082] Step 203: construct first graph structure data according to the first mapping relationship, the first spatiotemporal information, and the correlation relationship between the first spatiotemporal information.

[0083] For example, Figure 3 is a schematic diagram of the first graph structure data in an embodiment, Figure 3 It can be seen that the nodes in the first graph structure data include the first spatiotemporal information node and the to-be-clustered archive node, and the edges of the first graph structure data include time edges, space edges and spatiotemporal edges; the first spatiotemporal information node represents the shooting time and shooting location of each to-be-clustered image. For the convenience of representation, each first spatiotemporal information node is represented in the form of "shooting location_shooting time". Assuming that there are 5 shooting locations, namely A, B, C, D, and E, and the shooting time is converted by hour, then A_8 means that the shooting location is A, and the corresponding shooting time is 8 o'clock in the morning;

[0084] The spatiotemporal edge is determined by the first mapping relationship. Assuming that there are three files to be clustered, the spatiotemporal edge can be used to determine that file 1 stores images to be clustered corresponding to three first spatiotemporal information nodes, and the three first spatiotemporal information nodes are: A_8, C_8, C_9; file 2 stores an image to be clustered corresponding to one first spatiotemporal information node, and the first spatiotemporal information node is B_9; file 3 stores images to be clustered corresponding to two first spatiotemporal information nodes, and the two first spatiotemporal information nodes are: C_8, E_9;

[0085] The time edge and the space edge are determined by the correlation relationship between the first time-space information.

[0086] It should be noted that Figure 3 It is only one of the first graph structure data. Figure 3 The first graph structure data obtained by deformation on the basis of the first graph structure data are all within the protection scope of this embodiment.

[0087] Step 204: Determine the first similarity between the files to be clustered according to the first graph structure data.

[0088] Specifically, the first similarity between the files to be clustered can be calculated based on the distance between each edge in the first graph structure data, or the first similarity between the files to be clustered can be obtained by extracting edge features in the first graph structure data through a graph neural network. This implementation does not specifically limit how to determine the first similarity between the files to be clustered based on the first graph structure data.

[0089] For example, assuming that there are several images to be clustered with poor quality, and the similarity between them cannot be determined based on the image features of the images to be clustered with poor quality, at this time, the images to be clustered with better quality that are close to their first spatiotemporal information can be determined based on the first spatiotemporal information of the images to be clustered, so that the similarity between the images to be clustered with better quality can be accurately determined, and further, the similarity between the files to be clustered can be determined based on the similarity between the images to be clustered.

[0090] Step 205: clustering the archives to be clustered according to the first similarity between the archives to be clustered.

[0091] In the above-mentioned image clustering method, a plurality of archives to be clustered are obtained; the archives to be clustered include a plurality of images to be clustered; the first spatiotemporal information corresponding to the images to be clustered and the first mapping relationship between the first spatiotemporal information and each archive to be clustered are obtained; the first spatiotemporal information includes the shooting time of the images to be clustered and the shooting location of the images to be clustered; the first graph structure data is constructed according to the first mapping relationship, the first spatiotemporal information, and the correlation relationship between the first spatiotemporal information, so that the first graph structure data integrates the first spatiotemporal information of the images to be clustered and the correlation relationship between the first spatiotemporal information and the archives to be clustered. Therefore, according to the first graph structure data, the first similarity between the archives to be clustered can be more accurately determined. Furthermore, according to the similarity between the above-mentioned archives to be clustered, the archives to be clustered can be clustered more accurately, which solves the problem of low accuracy of archive clustering.

[0092] Please continue reading Figure 4 , Figure 4 FIG4 is a flow chart of an image clustering method in another embodiment. As shown in FIG4 , after determining the first similarity between the files to be clustered in step 204, the method further includes:

[0093] Step 402: Obtain image features of each image to be clustered.

[0094] Step 404: Determine the second similarity between the files to be clustered according to the image features of the images to be clustered.

[0095] Step 406: cluster the files to be clustered according to the first similarity and the second similarity.

[0096] For example, the first similarity and the second similarity can be combined in a weighted average manner to obtain the similarity between the files to be clustered after dynamic adjustment. Assume that the first similarity obtained based on the first spatiotemporal information of the images to be clustered is S gnn , the second similarity obtained based on the image features of the image to be clustered is S pic , then according to the first similarity and the second similarity, the dynamically adjusted similarity S adj As shown in the following formula, where α+β=1, α can be set to 0.5 and β can be set to 0.5.

[0097] S adj =αS pic +βS gnn (1)

[0098] It can be understood that in this embodiment, the first similarity obtained based on the first spatiotemporal information of the images to be clustered and the second similarity obtained based on the image features of the images to be clustered are used to obtain the dynamically adjusted similarity between the files to be clustered. Therefore, the dynamically adjusted similarity between the files to be clustered not only takes into account the first spatiotemporal features of the images to be clustered, but also takes into account the image features of the images to be clustered. Therefore, according to the dynamically adjusted similarity between the files to be clustered, the files to be clustered can be clustered more accurately.

[0099] In one embodiment, please refer to Figure 3 The first graph structure data includes a first spatiotemporal information node, a file node to be clustered, a time edge, a space edge and a spatiotemporal edge; the time edge is obtained by connecting multiple first spatiotemporal information nodes with the same shooting location but different shooting times of the images to be clustered; the space edge is obtained by connecting multiple first spatiotemporal information nodes with different shooting locations but the same shooting time of the images to be clustered; the spatiotemporal edge is obtained by connecting the file node to be clustered and the first spatiotemporal information node.

[0100] It can be understood that the first graph structure data in this embodiment integrates the shooting location and shooting time of the images to be clustered, as well as the association between the shooting location and shooting time of the images to be clustered and the archives to be clustered. Therefore, when the quality of the images to be clustered is poor or several images to be clustered are missed, the similarity between the archives to be clustered can still be accurately determined based on the above-mentioned first graph structure data. Furthermore, the archives to be clustered can be clustered more accurately based on this similarity.

[0101] In one embodiment, step 204, determining the first similarity between the archives to be clustered according to the first graph structure data, includes:

[0102] Inputting the first graph structure data into the trained image clustering model to obtain a graph embedding vector corresponding to each file to be clustered;

[0103] According to the graph embedding vector corresponding to each file to be clustered, the first similarity between the files to be clustered is determined.

[0104] It can be understood that the first graph structure data integrates the first spatiotemporal information of the images to be clustered. Therefore, the node features and edge features in the first graph structure data are extracted through the trained image clustering model, and the graph embedding vector corresponding to each file to be clustered is also integrated with the first spatiotemporal information of the images to be clustered. Furthermore, based on the graph embedding vector integrated with the first spatiotemporal information, the first similarity between the files to be clustered can be more accurately determined.

[0105] In one embodiment, Figure 5FIG. 1 is a flow chart of an image clustering model training method in an embodiment. Figure 5 , train the image clustering model through the following steps:

[0106] Step 502, obtaining a plurality of historical archives; the historical archives include a plurality of images to be trained.

[0107] Step 504, obtaining second spatiotemporal information corresponding to the image to be trained, and a second mapping relationship between the second spatiotemporal information and each historical file; the second spatiotemporal information includes the shooting time and the shooting location of the image to be trained.

[0108] Step 506: construct second graph structure data according to the second mapping relationship, the second spatiotemporal information, and the correlation relationship between the second spatiotemporal information.

[0109] Step 508: train the image clustering model according to the second graph structure data to obtain a trained image clustering model.

[0110] It can be understood that in this embodiment, the second graph structure data is constructed according to the second spatiotemporal information of the image to be trained, the correlation relationship between the second spatiotemporal information, and the second mapping relationship between the second spatiotemporal information and each historical archive, and the image clustering model is trained according to the second graph structure data, so that the trained image clustering model can extract the graph embedding vector that integrates the first spatiotemporal information. Furthermore, based on this graph embedding vector, the similarity between the archives to be clustered can be more accurately predicted, and based on this similarity, the archives to be clustered can be clustered more accurately.

[0111] In one embodiment, the image clustering model includes at least two different types of graph neural network models, and the graph neural network models are used to extract different types of edge features and different types of node features in the second graph structure data respectively;

[0112] Step 508, training the image clustering model according to the second graph structure data to obtain a trained image clustering model, including:

[0113] Inputting the second graph structure data into the graph neural network model respectively to obtain a plurality of graph embedding vectors corresponding to the nodes in the second graph structure data;

[0114] Determining a predicted value for each edge in the second graph structure data according to a plurality of graph embedding vectors corresponding to nodes in the second graph structure data;

[0115] Obtain the true value of each edge in the second graph structure data;

[0116] According to the predicted value and the true value, the loss value of each edge in the second graph structure data is obtained;

[0117] The image clustering model is trained according to the loss value to obtain a trained image clustering model.

[0118] Among them, the graph neural network model can choose a graph neural network based on the spectral domain or a graph neural network based on the spatial domain. The graph neural network in the spectral domain mainly includes but is not limited to ChebNet and GCN, and the graph neural network in the spatial domain mainly includes but is not limited to GraphSAGE and GAT.

[0119] It should be noted that in this embodiment, the same type of graph neural network model can also be used to extract different types of edge features and different types of node features in the second graph structure data, and step 508 is executed to obtain a trained image clustering model.

[0120] In this embodiment, multiple initial prediction values ​​for each edge in the second graph structure data can be calculated based on the graph embedding vector output by each graph neural network model, and then the multiple initial prediction values ​​of each edge can be aggregated to obtain the prediction value of each edge in the second graph structure data; or the image embedding vector output by each graph neural network model can be aggregated first, and then the prediction value of each edge in the second graph structure data can be obtained based on the aggregated graph embedding vector. This implementation does not make specific limitations on how to determine the prediction value of each edge in the second graph structure data based on multiple graph embedding vectors corresponding to nodes in the second graph structure data.

[0121] In addition, in this embodiment, in order to use the graph neural network to aggregate information on multiple types of node and edge relationships, a corresponding graph neural network model is defined for each type of edge relationship in the second graph structure data. For example, please continue to refer to Figure 3 , because the GAT network can better extract archive features, and the GCN network can better extract spatiotemporal features, the GAT network can be used to extract spatiotemporal edge features in the second graph structure data, and the GCN network can be used to extract temporal edge features and spatial edge features in the second graph structure data;

[0122] For example, this embodiment determines the loss value by the following formula, where: is the predicted value of each edge in the second graph structure data, is the true value of each edge in the second graph structure data, c r To adjust the scaling factors corresponding to different dimension values ​​on the relationship edges in the second graph structure data, is the loss value of each edge in the second graph structure data.

[0123]

[0124] It can be understood that in this embodiment, different graph neural network models are defined for different edge types in the second graph structure data, so that different types of graph neural network models can better extract different edge characteristics and node characteristics in the second graph structure data, that is, the graph embedding vector output by each graph neural network model better reflects the node information in the second graph structure data, and the predicted value of each edge obtained according to this graph embedding vector can better reflect the degree of correlation between the first spatiotemporal information. Furthermore, the image clustering model is trained according to the loss value obtained according to the predicted value, and the trained image clustering model can more accurately extract different edge characteristics and node characteristics in the first graph structure data, thereby further improving the accuracy of clustering of archives to be clustered.

[0125] In one embodiment, determining a predicted value of each edge in the second graph structure data according to a plurality of graph embedding vectors corresponding to nodes in the second graph structure data includes:

[0126] Aggregate each graph embedding vector corresponding to a node in the second graph structure data to obtain an aggregated graph embedding vector;

[0127] According to the aggregated graph embedding vector, a predicted value of each edge in the second graph structure data is determined.

[0128] For example, when aggregating the outputs of the graph neural network with different nodes and edge relationships, a summing or averaging method can be used, that is, the output feature vectors of the graph neural network with different nodes and edge relationships are summed or averaged by dimension as the result of the node after the graph neural network is optimized. For example, each graph embedding vector corresponding to the node in the second graph structure data can be aggregated by the following formula, where Agg represents a method for aggregating the outputs of the graph neural network with different nodes and edge relationships, and a summing function or an averaging function can be used; represents the graph embedding feature representation of node u on the k-th layer graph, Represents the graph embedding feature representation of node u on the k+1th layer graph; The graph neural network model used to represent the k-th layer when there are edges with a relationship of r between different nodes.

[0129]

[0130] In addition, the predicted value of each edge in the second graph structure data can be calculated by the following formula, where r represents a corresponding edge relationship in the second graph structure data, R r is the feature transformation matrix with edge relationship r, h i With h j is the graph embedding feature vector of nodes i and j, h i and h jIt is calculated by formula (3). Edge prediction function Output the predicted value of each edge in the second graph structure data.

[0131]

[0132] It can be understood that in this embodiment, the graph embedding vector output by each graph neural network model can reflect the node information in the second graph structure data. The graph embedding vector is first aggregated, and then the predicted value of each edge obtained according to the aggregated graph embedding vector can better reflect the degree of correlation between the first spatiotemporal information. Furthermore, the image clustering model is trained according to the loss value obtained from the predicted value. The trained image clustering model can more accurately extract different edge characteristics and node features in the first graph structure data, thereby further improving the accuracy of clustering the archives to be clustered.

[0133] In one embodiment, the second spatiotemporal information corresponding to the image to be trained and the second mapping relationship between the second spatiotemporal information and each historical archive are obtained, and then the method further includes:

[0134] Obtain image features of each image to be trained;

[0135] Determine the initial similarity between the files to be clustered according to the image features of the images to be trained;

[0136] Constructing second graph structure data according to the initial similarity, the second mapping relationship, the second spatiotemporal information, and the correlation relationship between the second spatiotemporal information;

[0137] The image clustering model is trained according to the second graph structure data to obtain a trained image clustering model.

[0138] Among them, see Figure 6 , Figure 6 is a schematic diagram of the second graph structure data in one embodiment, such as Figure 6 As shown, the initial value of the feature edge between the files to be clustered is the size of the initial similarity.

[0139] Through the above method, the second graph structure data not only integrates the second spatiotemporal information of the image to be trained, but also integrates the image features of the image to be trained. Therefore, training the image aggregation model based on this second graph structure data further improves the accuracy of predicting the similarity between the files to be clustered, thereby further improving the accuracy of clustering the files to be clustered based on this similarity.

[0140] Please continue reading Figure 7 , Figure 7 FIG7 is a flow chart of an image clustering method in another embodiment. As shown in FIG7 , the image clustering method includes the following steps:

[0141] Step 701, constructing a spatiotemporal feature map based on the spatiotemporal information contained in the archives to be clustered and the road network information of the shooting equipment.

[0142] Among them, the spatiotemporal feature map here is the first graph structure data mentioned above, which obtains the spatiotemporal information corresponding to the captured images contained in the archives to be clustered, as well as the path distance between the shooting devices; the spatiotemporal information includes the time information and spatial information of the captured images. The time information can select the shooting time of the corresponding image, and the spatial information can select the longitude and latitude information of the shooting device; the road network information between the shooting devices can be based on the distance given by the path search of the map, or it can be the spatial distance calculated based on the longitude and latitude. The distance between different shooting devices is in meters, and the image shooting time can generally be accurate to the second level. However, in order to better construct the spatiotemporal feature map, the shooting time can generally be set to the minute level or hour level according to actual business needs.

[0143] Please continue reading Figure 6 , different shooting devices at different time periods can be abstracted into a time and space point, Figure 6 The first spatiotemporal information node in is a spatiotemporal point. The path distance between different shooting devices in the same time period defines a spatial edge, and the distance value is used as the value on the spatial edge. Similarly, there is a time edge between adjacent time points of the same shooting device, and the set time interval is used as the value on the time edge. According to the spatiotemporal point information of the images contained in different archives, a spatiotemporal edge is constructed, and the number of spatiotemporal points corresponding to the images contained in the corresponding archives is used as the value on its spatiotemporal edge. In order to construct a spatiotemporal feature graph, in addition to constructing the edges of the spatiotemporal class representing the spatiotemporal relationship, it is also necessary to determine the initial similarity based on the images contained in different archives, generate corresponding feature edges based on the initial similarity, and treat different archives to be clustered as different clusters. If the centroids of two clusters are calculated for similarity using image features, and the corresponding similarity is greater than the preset similarity threshold, then there is a feature edge between the two clusters, and the corresponding feature value is the value of the feature edge.

[0144] Step 702, using a graph neural network to obtain a graph embedding vector representation of the archive nodes to be clustered on the spatiotemporal feature graph.

[0145] Wherein, based on step 701, a spatiotemporal feature graph is constructed, an effective graph neural network based on graph attention is designed, different types of graph node features are aggregated, and then a graph embedding vector representation that integrates spatiotemporal information is generated;

[0146] The initial features of nodes on the spatiotemporal feature graph can be randomly vectors of fixed dimensions. The dimension of the vector is related to the size of the entire spatiotemporal feature graph. The larger the spatiotemporal feature graph, the more vector dimensions there are. An optional reference value for the vector latitude is 20 to 80.

[0147] Please continue to refer to Figure 6 There are two different graph nodes in the spatiotemporal feature graph, namely, the archive nodes to be clustered and the spatiotemporal nodes. There are four different edge relationships, namely, spatial edges, temporal edges, spatiotemporal edges, and feature edges. In order to use the graph neural network to aggregate information of various types of node and edge relationships, a graph neural network model can be defined for each type of edge relationship, and then the graph embedding vector representations output by the graph neural network that specifically processes different edge relationships and nodes are aggregated to achieve information fusion. For example, please refer to formula (3). When aggregating the outputs of the graph neural network with different nodes and edge relationships, the summation or averaging method can be used, that is, the output feature vectors of the graph neural network with different nodes and edge relationships are summed or averaged by dimension as the result of the node after the graph neural network optimization.

[0148] Step 703, obtaining the first similarity between the archives to be clustered according to the graph embedding vector representation of the archive nodes to be clustered.

[0149] Step 704: Obtain a second similarity between the files to be clustered according to the image features of the images to be clustered in the files to be clustered.

[0150] Step 705 , aggregating the first similarity and the second similarity to obtain aggregated similarity, and clustering the files to be clustered according to the aggregated similarity.

[0151] Through the above method, the feature similarity between archives, the spatiotemporal information associated with the archives, and the correlation between the spatiotemporal information are introduced to construct a spatiotemporal feature graph. The graph neural network is used to obtain the graph embedding vector representation of different nodes on the spatiotemporal feature graph. The similarity between the graph embedding vector representations of the archives on the spatiotemporal feature graph and the archive similarity obtained based on the image features of the images to be clustered are combined to optimize the clustering effect of the archives to be clustered. The above method fully considers the spatiotemporal relationship between images, combines the original image feature information, and further improves the clustering effect of the archives to be clustered.

[0152] The embodiment of the present invention provides an image clustering model training method. Among them, the model training method can be used in a model training device. The model training device can be integrated in an electronic device, and the electronic device can be a terminal or a server. Among them, the terminal can be a mobile phone, a tablet computer, a laptop computer, a smart TV, a wearable smart device, a personal computer (PC, Personal Computer) and other devices. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0153] See also Figure 8 , is a schematic diagram of a scene of image clustering model training in one embodiment; Figure 8 As shown, after the electronic device obtains multiple historical archives, the historical archives include multiple images to be trained, and then obtains the second spatiotemporal information corresponding to the images to be trained, and the second mapping relationship between the second spatiotemporal information and each historical archive, the second spatiotemporal information includes the shooting time of the images to be trained and the shooting location of the images to be trained, and then constructs the second graph structure data according to the second mapping relationship, the second spatiotemporal information, and the correlation relationship between the second spatiotemporal information; finally, trains the image clustering model according to the second graph structure data, thereby realizing the training process of the image clustering model.

[0154] It should be noted that Figure 8 The scene diagram of the image clustering model training shown is only an example. The model training scene described in the embodiment of the present application is to more clearly illustrate the technical solution of the present application and does not constitute a limitation on the technical solution provided by the present application. It is known to those skilled in the art that with the evolution of model training and the emergence of new business scenarios, the technical solution provided by the present application is also applicable to similar technical problems.

[0155] Based on the above implementation scenarios, detailed descriptions are given below.

[0156] The embodiment of the present application will be described from the perspective of a model training device, which can be integrated in an electronic device. The electronic device can be a terminal or a server. The following is a specific process of the image clustering model training method:

[0157] Acquire multiple historical archives; the historical archives include multiple images to be trained;

[0158] Acquire second spatiotemporal information corresponding to the image to be trained, and a second mapping relationship between the second spatiotemporal information and each historical archive; the second spatiotemporal information includes the shooting time and the shooting location of the image to be trained;

[0159] Constructing second graph structure data according to the second mapping relationship, the second spatiotemporal information, and the correlation relationship between the second spatiotemporal information;

[0160] An image clustering model is trained according to the second graph structure data.

[0161] It can be understood that in this embodiment, the second graph structure data is constructed according to the second spatiotemporal information of the image to be trained, the correlation relationship between the second spatiotemporal information, and the second mapping relationship between the second spatiotemporal information and each historical archive, and the image clustering model is trained according to the second graph structure data, so that the trained image clustering model can extract the graph embedding vector that integrates the first spatiotemporal information. Furthermore, based on this graph embedding vector, the similarity between the archives to be clustered can be more accurately predicted, and based on this similarity, the archives to be clustered can be clustered more accurately.

[0162] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0163] In one embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as follows: Fig. 9As shown. The electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image clustering method is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the electronic device housing, or an external keyboard, touchpad or mouse, etc.

[0164] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0165] In one embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0166] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0167] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0168] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0169] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

[0170] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0171] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image clustering method, characterized in that: The method comprises: Acquire a plurality of files to be clustered; the files to be clustered include a plurality of images to be clustered; Acquire first spatiotemporal information corresponding to the image to be clustered, and a first mapping relationship between the first spatiotemporal information and each of the archives to be clustered; the first spatiotemporal information includes the shooting time and the shooting location of the image to be clustered; Constructing first graph structure data according to the first mapping relationship, the first spatiotemporal information, and the correlation relationship between the first spatiotemporal information; Determining a first similarity between the archives to be clustered according to the first graph structure data; The files to be clustered are clustered according to the first similarity between the files to be clustered.

2. The method according to claim 1, characterized in that: Determining the first similarity between the to-be-clustered archives according to the first graph structure data includes: Inputting the first graph structure data into the trained image clustering model to obtain a graph embedding vector corresponding to each of the to-be-clustered files; According to the graph embedding vector corresponding to each of the to-be-clustered files, a first similarity between the to-be-clustered files is determined.

3. The method according to claim 2, characterized in that The image clustering model is trained in the following manner: Acquire a plurality of historical archives; the historical archives include a plurality of images to be trained; Acquire second spatiotemporal information corresponding to the image to be trained, and a second mapping relationship between the second spatiotemporal information and each of the historical archives; the second spatiotemporal information includes the shooting time and the shooting location of the image to be trained; Constructing second graph structure data according to the second mapping relationship, the second spatiotemporal information, and the correlation relationship between the second spatiotemporal information; The image clustering model is trained according to the second graph structure data to obtain a trained image clustering model.

4. The method according to claim 3, characterized in that The image clustering model includes at least two different types of graph neural network models, and the graph neural network models are used to extract different types of edge features and different types of node features in the second graph structure data respectively; The step of training the image clustering model according to the second graph structure data to obtain the trained image clustering model includes: Inputting the second graph structure data into the graph neural network model respectively to obtain a plurality of graph embedding vectors corresponding to the nodes in the second graph structure data; Determining a predicted value of each edge in the second graph structure data according to a plurality of graph embedding vectors corresponding to nodes in the second graph structure data; Obtaining the true value of each edge in the second graph structure data; Obtaining a loss value of each edge in the second graph structure data according to the predicted value and the true value; The image clustering model is trained according to the loss value to obtain a trained image clustering model.

5. The method according to claim 4, characterized in that The step of determining the predicted value of each edge in the second graph structure data according to a plurality of graph embedding vectors corresponding to nodes in the second graph structure data comprises: Aggregating each graph embedding vector corresponding to a node in the second graph structure data to obtain an aggregated graph embedding vector; Determine a predicted value of each edge in the second graph structure data according to the aggregated graph embedding vector.

6. The method according to claim 3, characterized in that: The step of obtaining the second spatiotemporal information corresponding to the image to be trained, and the second mapping relationship between the second spatiotemporal information and each of the historical archives, further comprises: Obtaining image features of each of the images to be trained; Determining the initial similarity between the files to be clustered according to the image features of the images to be trained; Constructing second graph structure data according to the initial similarity, the second mapping relationship, the second spatiotemporal information, and the correlation relationship between the second spatiotemporal information; The image clustering model is trained according to the second graph structure data to obtain a trained image clustering model.

7. The method according to claim 1, characterized in that The method further comprises: Obtaining image features of each of the images to be clustered; Determining a second similarity between the files to be clustered according to image features of the images to be clustered; The to-be-clustered archives are clustered according to the first similarity and the second similarity.

8. The method according to claim 1, characterized in that The first graph structure data includes a first spatiotemporal information node, a file node to be clustered, a time edge, a space edge and a spatiotemporal edge; the time edge is obtained by connecting a plurality of the first spatiotemporal information nodes having the same shooting location but different shooting times of the images to be clustered; the space edge is obtained by connecting a plurality of the first spatiotemporal information nodes having different shooting locations but the same shooting time of the images to be clustered; the spatiotemporal edge is obtained by connecting the file node to be clustered and the first spatiotemporal information node.

9. A method for training an image clustering model, characterized in that: The method comprises: Acquire a plurality of historical archives; the historical archives include a plurality of images to be trained; Acquire second spatiotemporal information corresponding to the image to be trained, and a second mapping relationship between the second spatiotemporal information and each of the historical archives; the second spatiotemporal information includes the shooting time and the shooting location of the image to be trained; Constructing second graph structure data according to the second mapping relationship, the second spatiotemporal information, and the correlation relationship between the second spatiotemporal information; An image clustering model is trained according to the second graph structure data.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images