Image clustering method and device, electronic equipment and computer readable storage medium

Abstract

Embodiments of the present application provide an image clustering method and device, electronic equipment and computer readable storage medium, and relate to the technical field of image processing. The method comprises: obtaining target regions of each target region to be clustered, and obtaining feature vectors of each target region; clustering each feature vector based on a graph convolution network to obtain a clustering result. The process of clustering the image can be processed by using an artificial intelligence model. The image clustering method provided by the embodiments of the present application does not need to determine the clustering center, avoids the deviation of the entire clustering result caused by the deviation of the clustering center, and can effectively improve the accuracy of the clustering result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular, relates to an image clustering method and device, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] A clustering algorithm is a process of aggregating individuals with the same physical or abstract attribute into the same class through computer technology, data analysis and processing technology. Each class generated by the clustering algorithm has similar abstract attributes. The clustering algorithm has a wide range of applications in the field of image processing and other fields.

[0003] The current clustering algorithm usually adopts a traditional clustering algorithm, such as a similarity-based clustering algorithm (K-means), which needs to determine a clustering center. The representativeness of the clustering center directly affects the accuracy of subsequent clustering. Therefore, it is necessary to optimize the existing image clustering method. SUMMARY

[0004] The present application provides an image clustering method, device, electronic device and computer readable storage medium, and the technical solution is as follows:

[0005] In a first aspect, an image clustering method is provided, which includes:

[0006] Obtaining a target region of an image to be clustered, and obtaining a feature vector of the target region;

[0007] Clustering the obtained feature vector based on a graph convolution network to obtain a clustering result.

[0008] In a second aspect, an image clustering device is provided, which includes:

[0009] An acquisition module configured to obtain a target region of an image to be clustered, and obtain a feature vector of the target region;

[0010] A clustering module configured to cluster the obtained feature vector based on a graph convolution network to obtain a clustering result.

[0011] In a third aspect, an electronic device is provided, which includes:

[0012] One or more processors;

[0013] A memory;

[0014] One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to perform operations corresponding to the image clustering method according to the first aspect.

[0015] In a fourth aspect, a computer readable storage medium is provided, and the storage medium stores at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by a processor to implement the image clustering method according to the first aspect.

[0016] The technical scheme provided in the present application has the following beneficial effects:

[0017] The present application provides an image clustering method and device, electronic equipment and computer readable storage medium. Compared with the prior art, the present application obtains clustering of target regions of each image through a graph convolution network, without determining a clustering center, avoids deviation of the entire clustering result caused by deviation of the clustering center, and can effectively improve the accuracy of the clustering result.

[0018] Further, the cosine similarity is used to calculate the distance between nodes, and the cosine distance distinguishes the difference between features from the direction. The face feature difference of the same person under different postures, different illuminations and different time points is relatively large, but still represents the same person. Compared with the Euclidean distance, the cosine distance reflects the absolute difference in the numerical value of the feature. If the Euclidean distance is used, it will lead to clustering of two persons. Therefore, the use of the cosine distance can further improve the accuracy of clustering.

[0019] Further, the shot-cut (residual connection) mechanism is introduced on the graph convolution network to fuse the output results of different layers of the graph convolution network. The closer to the later layer of the graph convolution network, the smaller the difference between nodes of the same type. To some extent, the difference between different nodes can be maintained to avoid clustering different nodes into the same class, and the accuracy of clustering is further improved.

[0020] Further, the inter-layer clustering is introduced behind the original at least one layer of the graph convolution network. The inter-layer clustering can be used at any layer of the graph convolution network. It can fuse nodes belonging to the same class during the transmission of the graph convolution network, that is, the nodes have been clustered once inside the graph convolution network, thereby improving the accuracy and efficiency of the entire clustering work.

[0021] Further, on the basis of clustering, when a new target region to be clustered is obtained, a preset number of images can be obtained from each of the clustered categories, and the new target region to be clustered is clustered based on the clustered images to obtain a new clustering result, so as to determine the category of the new target region to be clustered. The method can be applied to small devices such as mobile phones with limited computing resources, improves the clustering efficiency, allows users to quickly see the clustering result, and enhances the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced.

[0023] Figure 1 is a schematic diagram of structured data of a digital graph;

[0024] Figure 2 is a schematic diagram of structured data of voice / text;

[0025] Figure 3 is a schematic diagram of unstructured data of network structure;

[0026] Figure 4 is a schematic diagram of a scheme in the prior art in which a clustering algorithm is applied to a face image;

[0027] Figure 5 is a schematic diagram of a scheme in the prior art in which a clustering algorithm is applied to a vocabulary;

[0028] Figure 6 is a schematic diagram of a scheme in the prior art in which a clustering algorithm is applied to voice information;

[0029] Figure 7 is a schematic diagram of a scheme in the prior art in which a clustering algorithm is applied to network terminal management;

[0030] Figure 8 is a flowchart of a scheme in the prior art in which full clustering is performed;

[0031] Figure 9 is a flowchart of a scheme in the prior art in which incremental clustering is performed;

[0032] Figure 10 is a flowchart of a scheme in the prior art in which a k-means clustering algorithm is performed;

[0033] Figure 11 is a schematic diagram of the influence of a threshold value on a clustering result in the prior art;

[0034] Figure 12a is a schematic diagram of a clustering error occurring in an example in which an existing clustering method is used;

[0035] Figure 12b is a schematic diagram of different persons being clustered into one person in an example in which an existing clustering method is used;

[0036] Figure 12c is a schematic diagram of different persons being clustered into one person in an example in which an existing clustering method is used;

[0037] Figure 13a is a schematic diagram of an existing k-means algorithm being affected by a clustering center in an example.

[0038] Figure 13b An example of using existing clustering methods to remove noisy nodes;

[0039] Figure 14 A flowchart of an image clustering method provided by an embodiment of the present application;

[0040] Figure 15 A subgraph structure diagram of Euclidean space data;

[0041] Figure 16 A subgraph structure diagram of non-Euclidean space data;

[0042] Figure 17 A structure diagram of a subgraph G in an example provided by the present application;

[0043] Figure 18 A structure diagram of a subgraph G in Figure 17 A weightless representation diagram of an adjacency matrix of the subgraph G in

[0044] Figure 19 A weighted representation diagram of an adjacency matrix of the subgraph G in Figure 17

[0045] A degree matrix diagram of the subgraph G in Figure 20 Figure 17 A Laplacian matrix diagram of the subgraph G in

[0046] Figure 21 Figure 17 A structure diagram of a subgraph G in another example provided by the present application;

[0047] Figure 22 A structure diagram of a subgraph G in another example provided by the present application;

[0048] Figure 23 A diagram of an adjacency matrix of the subgraph G in Figure 22 A diagram of an adjacency matrix of the subgraph G in

[0049] Figure 24 A diagram of an adjacency matrix of the subgraph G in Figure 22 A diagram of an adjacency matrix of the subgraph G in

[0050] Figure 25 A diagram of an adjacency matrix of the subgraph G in Figure 22 A diagram of an adjacency matrix of the subgraph G in

[0051] Figure 26 A flowchart of an image clustering method provided by an embodiment of the present application;

[0052] ​​Figure 27a This is a schematic diagram of an image clustering method in one example of this application;

[0053] Figure 27b This is a schematic diagram illustrating the comparison of Euclidean distance and cosine distance in an example.

[0054] Figure 27c This is a schematic diagram illustrating the comparison of Euclidean distance and cosine distance in an example.

[0055] Figure 28a This is a schematic diagram of an image clustering method in one example of this application;

[0056] Figure 28b This is a schematic diagram of the feature extraction process for data in one example of this application;

[0057] Figure 29 for Figure 28b A schematic diagram of the neural network structure used in the paper;

[0058] Figure 30 Here is a diagram illustrating the cosine distance in an example;

[0059] Figure 31 This is a schematic diagram illustrating the process of obtaining neighbor nodes in an example of this application;

[0060] Figure 32 This is a schematic diagram of a scheme for constructing a subgraph in one example of this application;

[0061] Figure 33 This is a schematic diagram illustrating the process of obtaining neighbor nodes in an example of this application;

[0062] Figure 34 This is a schematic diagram of a scheme for constructing a subgraph in one example of this application;

[0063] Figure 35 This is a schematic diagram comparing similarity results using Euclidean distance and cosine distance.

[0064] Figure 36 This is a schematic diagram of the structure of graph convolutional networks in the prior art;

[0065] Figure 37a This application provides a schematic diagram of the structure of a graph convolutional network that incorporates a shot-cut mechanism.

[0066] Figure 37b A schematic diagram of the graph convolutional network that incorporates a shot-cut mechanism and inter-layer clustering, as provided in an embodiment of this application;

[0067] Figure 37c This is a schematic diagram of an image clustering method in one example of this application;

[0068] Figure 37d A schematic diagram of the image clustering scheme introducing shot-cut mechanism and inter-layer clustering for an example of the present application;

[0069] Figure 38 A schematic diagram of the image clustering scheme introducing shot-cut mechanism and inter-layer clustering for an example of the present application;

[0070] Figure 39a A schematic diagram of the clustering using graph convolutional network introducing inter-layer clustering operation for an example of the present application;

[0071] Figure 39b A schematic diagram of the clustering using graph convolutional network introducing inter-layer clustering operation for an example of the present application;

[0072] Figure 40a A schematic diagram of the image clustering method flow for an example of the present application;

[0073] Figure 40b A schematic diagram of the dimension recovery scheme for an example of the present application;

[0074] Figure 41 A schematic diagram of the scheme for clustering according to connection probability for an example of the present application;

[0075] Figure 42a A schematic diagram of the scheme for removing noise nodes according to mean strategy for an example of the present application;

[0076] Figure 42b A schematic diagram of the method flow for removing noise nodes according to mean strategy for an example of the present application;

[0077] Figure 42c A schematic diagram of the scheme for dividing the clustering belonging to the same category into two categories for an example of the present application;

[0078] Figure 42d A schematic diagram of the scheme for updating the clustering result for an example of the present application;

[0079] Figure 43 A schematic diagram of the flow for clustering the new data to be clustered provided by the embodiment of the present application;

[0080] Figure 44 A schematic diagram of the flow for applying the image clustering method provided by the embodiment of the present application to mobile terminal;

[0081] Figure 45 A schematic diagram of the neural network structure applied to mobile terminal in the embodiment of the present application;

[0082] Figure 46A flowchart of a process of an image clustering method provided by an embodiment of the present application applied to a server is shown in FIG. 1.

[0083] Figure 47 A structural diagram of an image clustering device provided by an embodiment of the present application is shown in FIG. 2.

[0084] Figure 48 A structural diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 3. DETAILED DESCRIPTION

[0085] Embodiments of the present application are described in detail below with reference to the accompanying drawings, in which like or similar elements are denoted by the same or similar reference numerals, and the embodiments described below are examples only for explaining the present application, and cannot be interpreted as limiting the present application.

[0086] It should be understood by those skilled in the art that the singular forms "a," "an," and "the" used in the present application include plural referents unless the context clearly dictates otherwise. It should be further understood that the term "comprises" used in the specification of the present application means that the stated features, integers, steps, operations, elements, and / or components are present, but not excluding the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be an intermediate element. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The term "and / or" used herein includes all or any single unit and all combinations of the associated listed items.

[0087] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

[0088] A clustering algorithm is a process of aggregating individuals with the same physical or abstract attribute into the same class through computer technology, data analysis and processing technology. Each class generated by the clustering algorithm has similar abstract attributes. The clustering algorithm is one of the important methods for data analysis and mining, and has a wide range of applications in image processing, text analysis, speech recognition and network management.

[0089] The current clustering algorithm adopts a traditional clustering algorithm, for example: DBSCAN (Gensity-based Spatial Clustering of Application with Noise, density clustering algorithm based on high-density connection area), IGDCA (Incremental Grid Density-Based Clustering Algorithm, density-based incremental clustering), clustering algorithm based on similarity measurement (K-means) and clustering algorithm based on adjacency relationship measurement (Shared Nearest Neighbor) and the like. In practical application and research, it is found that the accuracy of the existing traditional clustering algorithm is relatively low, and it cannot better meet the actual commercial demand.

[0090] In recent years, with the improvement of algorithm, the convenience of data acquisition and the improvement of computer hardware speed, deep learning can obtain better accuracy and robustness in various tasks, and gradually replaces traditional algorithms to become mainstream. Convolutional Neural Networks (CNN) extracts features through convolution, which can extract better features than traditional feature extraction algorithms in image processing tasks. Recurrent Neural Network (RNN) is a deep learning model specially designed to solve sequence data, which can better capture sequence features. Images and voices belong to Euclidean space data, and their characteristics are structured rules. CNN and RNN can well process structured data such as images and voices, such as the digital graphics (also known as digital images) shown in Figure 1 and the voice or text shown in Figure 2 are structured data; but cannot process unstructured data such as the social relationship network (also known as social network) shown in Figure 3 Graph Convolutional Network (GCN) is specially proposed for solving unstructured data.

[0091] Clustering algorithm is widely used in image processing (such as face clustering, animal clustering and object scene clustering), text analysis (such as text clustering), speech recognition (such as language analysis) and network management (such as 5G terminal intelligent power saving management) and other fields.

[0092] Face clustering: one of the common ones in face clustering album management, as shown in Figure 4 After face detection, face alignment and feature extraction are performed on the images in the album, the faces are clustered by clustering algorithm (the example picture comes from YTB dataset). Each category contains its own photos, which is convenient for users to manage specific photos.

[0093] The clustering and face recognition system using graph convolutional networks in this application employs supervised learning and can be trained on annotated training data. It also enables unlabeled data management. While facial images are readily available in real-world scenarios, labeling face categories is extremely resource-intensive. Clustering and labeling unlabeled data using clustering algorithms not only saves significant manpower and resources but also improves the accuracy of face recognition.

[0094] Lexical clustering: such as Figure 5 As shown, specific words in a text are extracted and clustered to group words with similar attributes together, enabling document / text analysis. For example, in movie reviews, each review is clustered, and the clustering results reflect the user's evaluation of the movie, counting how many are positive, negative, and neutral reviews, facilitating movie ratings, etc.

[0095] Language analysis: such as Figure 6 As shown, there are many languages ​​around the world, and dialects vary greatly. Clustering algorithms are used to group the acquired speech information into multiple subclasses. For example, in speech recognition, the language of each subclass must first be determined before speech recognition and machine translation functions can be performed. A recording may contain multiple languages. First, the recording is parsed and clustered to determine which language each subclass belongs to before recognition is performed, thus improving the accuracy of speech recognition.

[0096] Network terminal management: such as Figure 7 As shown, 5G has begun commercial use, bringing convenience to people's lives. However, 5G base stations consume 12 times more power than 4G base stations. Therefore, intelligent power-saving management of base stations can effectively reduce the cost of 5G services. For example, a 5G base station consists of multiple components, each with different power consumption characteristics. By using clustering algorithms to group devices with similar power consumption attributes into subclasses, targeted analysis and management of power consumption attributes can be conducted, improving node management efficiency and avoiding resource waste.

[0097] To address the error of grouping faces of class A and class B into the same class during face clustering, existing technologies propose a two-stage clustering algorithm combining full clustering and incremental clustering. First, a subset of face data is selected for full clustering, followed by incremental clustering based on the full clustering. The specific implementation is as follows:

[0098] Full clustering, as the name suggests, involves clustering all facial data. It determines some cluster categories before incremental clustering and provides a category basis for clustering new facial data that comes later.

[0099] The implementation flowchart of the full-quantity clustering is as shown in Figure 8 The specific implementation process is as follows:

[0100] 1) Select N photos for face detection, face alignment and face feature extraction, and take the N faces as separate categories;

[0101] 2) Calculate the distance between each pair of face categories;

[0102] 3) Determine whether the distance between the two categories is less than a given threshold. If it is less than the given threshold, the two categories are merged into one category; if it is greater than the threshold, they are two separate categories;

[0103] 4) Repeat step 2) until the distance between all face categories is greater than the given threshold, indicating that the full-quantity clustering is complete.

[0104] Full-quantity clustering is equivalent to the initialization work of incremental clustering. Only the first face clustering uses full-quantity clustering, and subsequent clustering is incremental clustering based on it. The process flowchart of incremental clustering is as shown in Figure 9 The specific implementation method is as follows:

[0105] 1) Count the number of categories M in the last clustering, take one of the M categories as category A, and take the newly added P face feature information as P categories, take one of the P categories as category B;

[0106] 2) Use the density clustering algorithm to re-cluster the face features in category A, and set a lower threshold to obtain one or more categories;

[0107] 3) Count the number of subcategories in category A, and select the subcategory with the most faces as the representative of category A;

[0108] 4) Calculate the distance between each pair of face data in the representative subcategory of category A and category B;

[0109] 5) Determine whether the distance between each pair is less than a given threshold. If it is less than the given threshold, the two categories are merged into one category; if it is greater than the threshold, they are two separate categories;

[0110] 6) Repeat step 1) until the number of faces in all categories no longer changes, indicating that the clustering is complete.

[0111] The k-means clustering algorithm is a commonly used clustering algorithm, k represents clustering data into k classes, and means represents calculating the mean of all nodes in each sub-class as the clustering center of the sub-class. In order to solve the problem that the initial clustering center of the traditional clustering algorithm cannot guarantee the representativeness of the clustering center, resulting in a very poor clustering algorithm and clustering result, the prior art proposes an optimized algorithm for the problem that the clustering number k value is not easy to select. As shown in Figure 10 The specific implementation is as follows:

[0112] S1: all nodes to be clustered are grouped into a set M, the density function value of all nodes to be clustered is calculated, and nodes with a density function value greater than or equal to the average density function value of the set are grouped into a dense point set Y;

[0113] S2: two nodes in the dense point set Y are selected to form a set Q;

[0114] S3: m nodes are selected from the set M by the focal point statistics method, and the m nodes are grouped into a set of candidate initial clustering centers C;

[0115] S4: the nodes in the dense point set Y are divided into the categories in which the initial clustering centers in the set Q are located, and the average maximum similarity of the first clustering is obtained;

[0116] S5: a node is selected from the candidate initial clustering center set C and added to the set Q as a new initial clustering center in the set Q, and the node is deleted in C. Steps S4-S5 are repeated, and the minimum value of the average maximum similarity of each clustering is selected by comparing the average maximum similarity of each clustering;

[0117] S6: the minimum value of the average maximum similarity is selected as the clustering center in the set Q, and then k-means clustering is performed to obtain the clustering result.

[0118] Problems of existing clustering algorithms:

[0119] (A) In order to solve the problem of face clustering, as shown in Figure 11If the B-class face is contained in the A-class face, an error of clustering the A-class face and the B-class face into one class is caused, a two-stage clustering algorithm combining full clustering and incremental clustering is proposed. First, part of the face data is selected for full clustering, and then incremental clustering is performed on the basis of full clustering. This method can indeed avoid clustering two different classes into the same class to some extent. However, if the threshold is too high, the clustering algorithm will discard the edge face, which may cause a class to be clustered into two classes. If the threshold is too low, the same error of clustering two classes into the same class will occur. And this error is in an accumulative state in the later clustering.

[0120] (B) The existing clustering algorithm almost judges whether two nodes belong to the same class according to the distance between the clustering nodes, but the situation of "my neighbor's neighbor is not necessarily my neighbor" may occur, causing the clustering result to be wrong. As shown in Figure 12a No. 12 node belongs to the class of node 1, and No. 6 node belongs to the class of No. 8 node. However, the distance between No. 12 node and No. 6 node is small, and according to the traditional distance-based clustering algorithm, No. 12 node and No. 6 node will be clustered into one class. However, the distance between No. 6 node and No. 7 node, No. 8 node and No. 9 node is smaller, and the relationship is more intimate, so No. 6 node is the neighbor of the neighbor of No. 1 node (No. 12 node), but it should belong to the class of No. 8 node, not the class of No. 1 node.

[0121] (C) The existing clustering algorithm directly uses the original clustering data for clustering, and the clustering effect is poor and the accuracy is low. For example, in face clustering, face detection, face alignment and face feature extraction are performed on the collected images. However, due to the difference in light, front and side faces, the features of the faces are quite different. If the original face features are simply used for clustering or the Euclidean distance is used for clustering, the same person may be clustered into two classes or different persons may be clustered into one class.

[0122] As shown in Figure 12b The photos in the leftmost circle belong to the same person, but the background, lighting, image quality and shooting period of each photo are different. If the Euclidean distance or the original photo features are used for clustering, it is likely to cause the photos of the same person to be clustered into multiple classes.

[0123] As shown in Figure 12c Due to the influence of light and posture, different classes may be clustered into the same person. Class A is a photo of a little boy, and class B is a photo of a little girl. d1 and d2 represent the Euclidean distances between A3 and classes A and B respectively. Due to the influence of the external environment, d2 < d1, so the clustering result may be that A and B belong to the same class.

[0124] (D) Traditional k-means clustering algorithm is an iterative clustering algorithm, which needs to select appropriate K value and initial clustering center through continuous iteration, and has high algorithm complexity and low efficiency.

[0125] (E) Like traditional k-means algorithm, many clustering algorithms need to determine the clustering center, and the representativeness of the clustering center directly affects the accuracy of subsequent clustering. Especially in the sub-class of introducing negative samples, the existence of negative samples causes the whole clustering center to deviate, resulting in more and more inaccurate clustering. As shown in the left figure, node 13 represents the clustering center of the class, and when the clustering error introduces negative sample node 6, the whole clustering center will deviate to the right, which obviously leads to more and more inaccurate clustering. Figure 13a

[0126] (F) The prior art proposes to use a simple graph convolutional neural network (GCN) to extract the features of the subgraph, and the GCN will cluster the features between the nodes with the transmission of the network, reducing the difference between the features. If there are noise nodes in the subgraph, it will cause the difference between the noise nodes and other neighbor nodes to become smaller and smaller, and finally the noise nodes will be clustered into the class, resulting in clustering errors.

[0127] (G) The prior art proposes to use GCN to predict the probability between the center node and the neighbor node for clustering, but does not perform any post-processing. If there are noise nodes, it will inevitably lead to the occurrence of neural network prediction errors, resulting in final clustering errors. As shown in the left figure, A, B, C, D and E belong to the same category, and the connection probability between F and A is 0.9, but F does not exist connection with other nodes, and we have reason to believe that F is a noise node. But if there is no post-processing, F and other nodes will be clustered into one class, resulting in clustering errors. Figure 13b

[0128] The present application proposes a clustering algorithm with better accuracy and stronger robustness to solve the above problems, and the innovation points are:

[0129] A) A GCN-based clustering algorithm is proposed, which has higher accuracy and relatively lower complexity;

[0130] ​​B) In order to solve the problem of "my neighbor's neighbor is not necessarily my neighbor" and the poor clustering effect caused by external factors such as light, a coding-decoding graph neural network is proposed. The node features are mapped to another dimensional space by an encoder inside the neural network. An inter-layer clustering module is proposed in the conventional GCN layer to pre-cluster nodes with strong relationships and normalize the features to determine noise nodes. The inter-layer clustering module can extract hierarchical features of subgraphs and increase the difference between noise nodes and other nodes. Through the inter-layer clustering module, the difference between the features of nodes with the same attributes becomes smaller, and the difference between nodes with different attributes becomes larger and larger;

[0131] C) A post-processing algorithm based on Average Score is proposed to determine whether an isolated node is a noise node according to the neighbor relationship, so as to improve the accuracy and stability of clustering;

[0132] D) Full clustering is to cluster all nodes at the same time, and the complexity of the clustering algorithm increases with the increase of the number of nodes to be clustered. In order to reduce the time complexity of the algorithm, an efficient addition logic is proposed, which can not only ensure the accuracy of the algorithm, but also reduce the complexity of the algorithm;

[0133] E) Traditional clustering algorithms all need to determine the cluster center, and the representativeness of the cluster center directly affects the accuracy of subsequent clustering. The novel clustering algorithm proposed in the present application does not need to determine the cluster center, avoiding the risk of entire clustering error caused by the error of the cluster center;

[0134] F) A shot-cut mechanism is proposed inside the GCN to maintain the difference between features, which is conducive to excluding noise nodes and improving the accuracy of clustering. At the same time, the shot-cut mechanism can alleviate the problem of over-smoothing of GCN;

[0135] G) Use cosine distance instead of Euclidean distance process. Cosine distance is more accurate than Euclidean distance and can judge the similarity of nodes in the same class;

[0136] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes may not be described again in some examples. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0137] A possible implementation manner is provided in the embodiments of the present application, as shown in Figure 14 An image clustering method is provided, which can include the following steps:

[0138] In step S401, a target region of an image to be clustered is acquired, and a feature vector of the target region is acquired.

[0139] Specifically, the target region can be a region including a target to be detected, and the target can be a face, a specific object, or the like in the image.

[0140] In a specific implementation process, for at least one target region to be clustered, the target region can be extracted from the image, and then a feature vector is extracted from the target region.

[0141] In step S402, the acquired feature vector is clustered based on a graph convolutional network to obtain a clustering result.

[0142] The graph convolutional network (GCN) is a neural network structure that has gradually become popular in recent years. Unlike the traditional network model LSTM (Long Short-Term Memory) and CNN (Convolutional Neural Networks) that can only be used for grid-based data, the graph convolutional network can process data with a general topological graph structure and deeply explore its features and rules, such as PageRank citation networks, social networks, communication networks, protein molecular structures, and a series of irregular data with spatial topological graph structures. The graph convolutional network will be further introduced below.

[0143] Specifically, a subgraph corresponding to at least one feature vector can be acquired first, and then an adjacency matrix, a feature matrix, and a first-order neighbor vector of the subgraph are acquired. The adjacency matrix, the feature matrix, and the first-order neighbor vector of the subgraph are input into at least one layer of the graph convolutional network, and are sequentially transformed (which can also be regarded as feature extraction). The transformed matrix is input into at least one layer of a fully connected layer for classification to obtain a connection probability between a center node and each neighbor node of the subgraph.

[0144] In a specific implementation process, for at least one subgraph, the subgraph can include a center node and at least one neighbor node. The connection probability between the center node and each first-order neighbor node is acquired, and if the connection probability is greater than a preset threshold, the center node and the first-order neighbor node with the connection probability greater than the preset threshold can be classified into one class.

[0145] The first-order neighbor node is a neighbor node directly connected to the center node, and the neighbor node can also include a second-order neighbor node, i.e., a neighbor of the neighbor of the center node, which is not a neighbor node directly connected to the center node.

[0146] In the above embodiments, the target regions of each image are clustered by obtaining graph convolutional networks. It is not necessary to determine the cluster center, which avoids the deviation of the cluster center from causing the deviation of the entire clustering result and can effectively improve the accuracy of the clustering result.

[0147] The following section will provide a more detailed introduction to subgraphs and graph convolutional networks (also known as graph convolutional neural networks) with reference to the accompanying diagrams.

[0148] 1) Definition of a graph

[0149] The definition of a graph in this application is not the common definition of an image, but rather the mathematical definition of a graph.

[0150] Digital images, speech, and text belong to Euclidean space data, which have fixed dimensions. Therefore, neural networks such as CNNs or RNNs can be used to extract features from them. However, unstructured data such as social networks do not have fixed dimensions. In order to process this kind of unstructured data, previous researchers have proposed graph convolutional neural networks (GCNs).

[0151] If we treat each pixel of the image as a node in the graph, then... Figure 15 As we can see, each node has 8 neighborhoods, and features of the image can be extracted using a fixed-size convolutional kernel. For example: Figure 15 The neighboring nodes of node 1 are {2,3,4,5,6,7,8,9}, and features can be extracted from them using a 3x3 convolution kernel. Figure 15 The neighboring nodes of node 2 are {1,3,4,8,9,10,14,15}, and features can be extracted from them using a 3x3 convolution kernel. Points on the boundary can be padded with zeros to satisfy the 8-neighborhood feature, for example, node 8; Figure 16 As shown, for non-Euclidean spatial data, the number of neighboring nodes for each node varies, making it impossible to extract features using a fixed-size convolutional kernel. For example: Figure 16 The neighboring nodes of node 1 are {0,3,4,5}, and the number of its neighbors is 4. It needs to be characterized by convolutional kernels of 4 node sizes. Figure 16 Node 2 has only one neighbor, {3}, and requires a convolutional kernel of size 2 nodes for feature extraction. This demonstrates that for non-Euclidean space data, due to the varying number of neighbors for each node, different convolutional kernels are needed for feature extraction, which traditional CNNs and RNNs cannot meet. Graph Convolutional Neural Networks (GCNs), however, can measure the relationships between nodes in non-Euclidean space data using the graph's adjacency matrix and Laplacian matrix.

[0152] A graph G = (V, E) is defined, where V represents the set of nodes, and E represents the set of edges. Each node in the graph is represented by a vector of dimension D, and the result of the graph is usually represented by an adjacency matrix of the graph for easy calculation. As shown in Figure 17 The structure can be represented by a graph G, where V represents the set of nodes {0, 1, 2, 3, 4, 5}, E represents the set of edges {a, b, c, e, f, g, h}, where {a, b, c, e, f, g, h} represents the weight of the edge, N represents the number of nodes, A is the vector representation of node 4, and D is the dimension of the vector.

[0153] In graph theory, the adjacency matrix and Laplacian matrix of a graph are usually used to measure the relationship between nodes in the graph. Figure 18 The adjacency matrix of the graph G is a non-weighted representation, usually using 1 to represent the existence of a connection between two nodes, and 0 to represent the absence of a connection between two nodes, as shown in Figure 19 Figure 19 The adjacency matrix is represented with weights.

[0154] The degree of a node in a graph G represents the number of paths through that node, for example, the degree of node 0 is 3. The degree matrix D of the graph G is shown in Figure 20 The degree matrix only has values on the diagonal, representing the degree of the node, and the rest of the positions are 0. As shown in Figure 21 The Laplacian matrix L of the graph G is D - A.

[0155] 2) Definition of graph convolution

[0156] X (l+1) = σ (AX (l) W (l) +b (l) ) (1)

[0157] In the formula: X (l) represents the node features of the l-th layer; σ represents a nonlinear transformation; A represents the adjacency matrix; W (l) represents the weight of the l-th layer; b (l) represents the intercept of the l-th layer; X (l+1) represents the node features of the l+1-th layer.

[0158] The convolution operation on the graph is essentially an operation that represents the node in the graph with its own features and the features of the neighboring nodes. The result of the graph convolution is that the difference between the node features with neighbor relationships becomes smaller and smaller, and the difference between the unrelated nodes becomes larger and larger, which means that the graph convolution itself has a clustering function.

[0159] For example: there is a graph G as shown in Figure 22 ​As shown, the directed graph G is represented by the adjacency matrix A (it is generally assumed that each node is connected to itself), and X represents the eigenvector of each node in graph G. From graph G, it can be seen that nodes 0, 1, 2, and 3 belong to the same type of node, and nodes 4 and 5 belong to the same type of node.

[0160] Let's explore the mathematical principles of GCN from a very simple perspective. Assuming all weights W are 1 and all intercepts (biases) b are 0, when the nonlinear transformation is a linear function y = x, the formula for graph convolution can be expressed as:

[0161] X (l+1) =A*X (l) (2)

[0162] In the formula: X (l) Represents the node characteristics of the l-th layer; A represents the adjacency matrix; X represents the node characteristics of the l-th layer. (l+1) This represents the node characteristics of the (l+1)th layer.

[0163] use Figure 22 From the diagram G shown, we can obtain Figure 23 The adjacency matrix A and the node features X of the l-th layer are shown. (l) and the node features X of layer l+1 (l+1) .

[0164] From the above formula (2), we can deduce the following formula (3):

[0165] X(l+1)=A*X(l) (3)

[0166] Then according to Figure 23 The neighbor matrix A and the node features X of the (l+1)th layer are shown. (l+1) It can be obtained Figure 24 The node features X of layer l+2 shown (l+2) ;

[0167] Similarly, the following formula can be deduced:

[0168] X(l+3)=A*X(l+2) (4)

[0169] Then according to Figure 24 The neighbor matrix A and the node features X of the (l+2)th layer are shown. (l+2) It can be obtained Figure 25 The node feature X of layer l+3 shown (l+3) .

[0170] Through X (l+1) =A*X (l) It was found that the new feature representation of each node in graph G is the sum of the features of its neighbors and itself. Each X (l+1) =A*X(l) represents a layer of GCN network, and it is found through the above calculation that if there is a relationship between nodes, that is, they belong to the same type of nodes, the difference between the same type of nodes will become smaller and smaller with the forward transmission of the GCN network. For example: node 4 and node 5 belong to the same type, and their features will be the same in the subsequent GCN calculation; node 0, node 1, node 2 and node 3 belong to the same type of nodes, and their features will also become more and more similar. While the feature difference between different types will become larger and larger. Therefore, the GCN itself has the function of node clustering, so the application uses GCN to realize the clustering function, and has rigorous mathematical and scientific basis.

[0171] As can be seen from the above figure, the eigenvalue of the node will become larger and larger with the depth of the GCN network, which may cause gradient disappearance or gradient explosion when calculating the gradient in the backward propagation, seriously affecting the effectiveness of the algorithm. Therefore, in practical application, the features are usually normalized, but the characteristics of GCN do not change.

[0172] The adjacency matrix A is normalized by the degree matrix D of the graph, A=D -1 * A, so the formula of GCN becomes:

[0173] X (l+1) =σ(D -1 AX (l) W (l) +b (l) ) (5)

[0174] A=D -1 *A is transformed into A=D (-1 / 2) *A*D (-1 / 2) :

[0175]

[0176] In the formula: X (l) represents the node feature of the lth layer; σ represents a nonlinear transformation; D represents a degree matrix; A represents an adjacency matrix; W (l) represents the weight of the lth layer; b (l) represents the intercept of the lth layer; X (l+1) represents the node feature of the (l+1)th layer.

[0177] It is generally considered that each node itself has a connection with itself, and a self-loop is added, so the calculation formula of GCN is:

[0178]

[0179] In the formula: X (l) represents the node feature of the lth layer; σ represents a nonlinear transformation; A represents the adjacency matrix corresponding to node i; denotes A+I, self-loop, I denotes a diagonal matrix with value 1; denotes corresponding degree matrix; W (l) denotes the weight of the lth layer; b (l) denotes the intercept of the lth layer; X (l+1) denotes the node feature of the (l+1)th layer.

[0180] The above is a detailed introduction to the graph and the graph convolution network, and the specific process of constructing a subgraph will be further described below in combination with the drawings and embodiments.

[0181] In a possible implementation of the embodiment of the present application, the step S401 of acquiring the feature vector of the target region can include:

[0182] (1) detecting feature points of the target region, aligning the target region based on the detected feature points, and obtaining a corresponding target transformation region;

[0183] (2) extracting a vector of the target transformation region to obtain the feature vector of the target region.

[0184] In the specific implementation process, the transformation can be a process of direction conversion and alignment of the feature points. For example, for a face, if the target region containing the face is extracted, but the face is not in the front direction, the feature points of the target region containing the face are extracted, and the feature points are subjected to affine transformation to obtain a front image of the face.

[0185] In a possible implementation of the embodiment of the present application, as shown in Figure 26 the step S402 of clustering the acquired feature vector based on the graph convolution network to obtain a clustering result can include:

[0186] The step S210 of constructing a subgraph corresponding to the feature vector; the subgraph includes a center node corresponding to the feature vector and at least one neighbor node corresponding to another feature vector.

[0187] Specifically, the feature vector can be taken as the center node, the neighbor nodes corresponding to the center node are acquired, and the subgraph of the feature vector is constructed.

[0188] In the specific implementation process, the similarity between the feature vector and other feature vectors can be acquired respectively, wherein the other feature vectors refer to vectors other than the feature vector in the plurality of feature vectors corresponding to the target region to be clustered; the neighbor nodes can be determined from the other feature vectors according to the similarity between the feature vector and the other feature vectors.

[0189] In step S220, the connection probability between the center node of the subgraph and the neighbor nodes thereof is determined based on the graph convolution network, and the target region is clustered according to the determined connection probability, to obtain a clustering result.

[0190] The neighbor nodes can be first-order neighbor nodes, i.e., the neighbor nodes directly connected to the center node.

[0191] Specifically, the corresponding input feature of the subgraph can be obtained, the input feature is input into the graph convolution network for transformation, and then the transformed matrix is input into the fully connected network to obtain the connection probability between the center node of the subgraph and each neighbor node.

[0192] In the specific implementation process, the first-order neighbor node with a connection probability greater than a preset threshold can be determined as being in the same category as the center node.

[0193] The specific process of constructing the subgraph will be described in detail below in combination with specific embodiments.

[0194] In the specific implementation process, the subgraph corresponding to the feature vector constructed in step S210 can include:

[0195] (1) taking the feature vector as the center node, and obtaining the neighbor nodes of the center node from other feature vectors.

[0196] The other feature vectors refer to the feature vectors corresponding to other data except the center node.

[0197] Specifically, the neighbor nodes can be selected from the other feature vectors according to the similarity between the feature vector corresponding to the center node and the other feature vectors.

[0198] In the specific implementation process, the neighbor nodes of the center node are obtained from the other feature vectors by taking the feature vector as the center node, which can include:

[0199] a. respectively obtaining the cosine distance between the feature vector and the other feature vectors;

[0200] b. based on the cosine distance between the feature vector and the other feature vectors, selecting the neighbor nodes of the feature vector from the other feature vectors.

[0201] The process of specifically selecting the neighbor nodes will be described in detail below.

[0202] The cosine distance, also known as the cosine similarity, is an algorithm for evaluating the similarity between two face feature vectors by calculating the cosine value of the included angle between them; the greater the Euclidean distance between A and B, the greater the difference between them; the greater the cosine distance, the smaller the difference between them.

[0203] (2) constructing a subgraph according to the center node and the neighbor node.

[0204] Specifically, after the neighbor node is selected from the feature vectors corresponding to other data, a subgraph is constructed according to the center node and the at least one neighbor node selected.

[0205] The above embodiments illustrate the specific construction process of the subgraph. In the following, the image clustering method and the specific construction process of the subgraph provided by the present application will be further described by taking a face image as an example.

[0206] As shown in FIG. 1, Figure 27a in one example, the method of image clustering can include the following steps:

[0207] S1, performing face detection and cropping a face image;

[0208] S2, using a CNN to extract a feature vector of the face image, i.e., a facial feature shown in the figure, and taking the extracted facial feature as a node;

[0209] S3, taking each node as a center node, determining at least one neighbor node according to the cosine distance, and constructing a subgraph by connecting the center node and the neighbor node;

[0210] The step S3 can include:

[0211] (1) calculating the cosine distance between two nodes;

[0212] (2) taking each node as a center node and selecting a first-order neighbor node according to the cosine distance;

[0213] (3) selecting a neighbor node of the first-order neighbor node, i.e., a second-order neighbor node, according to the cosine distance;

[0214] (4) selecting k neighbor nodes for each node to establish a subgraph; wherein k is a natural number;

[0215] (5) repeating the steps (1)-(4) until the subgraph of each node is established;

[0216] S4, the feature of each subgraph is extracted by a GCN; a inter-layer clustering module is used to maintain the difference between the noise and other node features, and then the connection probability of the center node and its first-order neighbor node is obtained according to a softmax function;

[0217] S5, classifying the face image according to the connection probability of the center node and the first-order neighbor node.

[0218] Figure 27aOnly steps S2-S4 are shown, and using the Euclidean distance to calculate the distance between nodes in the traditional clustering method will cause clustering errors. In this example, the cosine distance is used instead of the Euclidean distance. The cosine distance is more accurate than the Euclidean distance and can determine the similarity of nodes in the same class.

[0219] In step S3 described above, the cosine distance is used instead of the Euclidean distance. A large number of experiments and academic research have shown that the cosine distance is more accurate than the Euclidean distance and can more accurately determine the similarity of nodes in the same class.

[0220] As shown in Figure 27b- Figure 27c , the numbers 0-9 represent faces with different identities. Figure 27b A, B, and C in Figure 27c correspond to A, B, and C in . LAB represents the Euclidean distance between A and B, and e represents the angle between A and B; LBC represents the Euclidean distance between B and C, and d represents the angle between B and C. The smaller the angle, the greater the cosine distance, and the higher the similarity between two features.

[0221] A and B represent the same person, while C and A are not the same person. Since the facial features of the same person are different at different poses, different illuminations, and different times, if the Euclidean distance is used, it will cause clustering errors. For example, in Figure 27c , the Euclidean distance between A and B is 1.45, and the Euclidean distance between B and C is 1.22. We can cluster B and C into a class, but the clustering result based on the Euclidean distance is incorrect, while the cosine distance does not have this problem. Using the cosine distance to calculate the similarity is more accurate. In the same class, the angle between the features is small, so using the cosine distance can more accurately cluster.

[0222] As shown in Figure 28a , in one example, taking a face image as an example, the image clustering method can include the following steps:

[0223] S10: performing feature extraction on the collected data;

[0224] S20: calculating the cosine distance between each node (face);

[0225] S30: taking each node as a center node, finding its k neighbor nodes, and constructing a subgraph;

[0226] S40: calculating the probability of connection between neighbor nodes and the center node through the GCN neural network to form each connection pair.

[0227] S50: if the connection probability between two nodes is greater than a set threshold, they are clustered into a class, otherwise they are two classes.

[0228] In the above examples, by calculating the cosine distance between the nodes, the framework constructs a subgraph with each node as the center node, thereby calculating the connection probability between the center node of each subgraph and its first-order neighbor nodes, and according to the connection probability, it is determined whether the two nodes of each connection pair belong to a category, which can improve the accuracy of clustering.

[0229] As shown in the Figure 28b feature extraction step S10 includes a data preprocessing submodule and a feature extraction submodule:

[0230] The data preprocessing module includes a face detection module, a face alignment module, and a face feature extraction module.

[0231] The face detection module is an algorithm for positioning the face in the collected image.

[0232] The original image contains a lot of background information. If the original image is used for clustering, different people in the same background condition may be clustered into the same person. The face detection algorithm is used to detect the face in the image, crop and save the face, and only the face is clustered in the subsequent clustering to avoid clustering errors caused by background interference.

[0233] The cropped face image is normalized to a specified resolution size.

[0234] The face alignment module is an algorithm for correcting face images with tilt. The face feature point detection algorithm is used to detect the feature points of the face, and the detected feature points are used for affine transformation to obtain the front face image.

[0235] The feature extraction module uses a deep learning algorithm to extract features from the aligned face. As the name implies, feature extraction is to represent the face image with a 128-dimensional vector. Currently, the example can also use other dimensional vectors for representation, which is not limited here.

[0236] Face feature extraction is learned from a large amount of face data. For example, a dataset containing 10000 people of black, white, and yellow races is constructed, where each face data is greater than or equal to 450, and each race includes child, teenager, young adult, middle-aged, and old data. The neural network performs face feature extraction training and learning on the dataset. After training, the deep learning model can realize feature extraction of each face image.

[0237] As shown in the Figure 29 feature extraction of the face image after face detection and face alignment; wherein block 1 represents a convolutional neural network layer; block 2 represents a depth separable convolution layer; block 6 represents a fully connected layer, and its output is a 128-dimensional face feature.

[0238] In step S2 in Figure 27a The face is represented by a 128-dimensional vector, in step S3 in Figure 27a The cosine distance between each two face nodes is calculated.

[0239] The cosine distance, also known as cosine similarity, is an algorithm for evaluating the similarity between two face feature vectors by calculating the cosine value of the angle between them.

[0240] As shown in Figure 30 , f1 represents the A face feature vector, f2 represents the B face feature vector, and a represents the angle between f1 and f2 vectors. The smaller the angle a, the higher the similarity between f1 and f2. When the angle a is 0, the cosine value is 1, indicating that f1 and f2 have the highest similarity; when the angle a is 90, the cosine value is 0, indicating that f1 and f2 are independent of each other, that is, f1 and f2 are not the same person. The calculation formula is:

[0241]

[0242] Where x and y represent two vectors whose similarity is to be calculated; sim(X, Y) represents the cosine similarity between the two vectors.

[0243] In step S3, each node is taken as the center node P, and m1 neighbor nodes with the largest cosine distance, m2 neighbor nodes with the smallest cosine distance, and m3 neighbor nodes with intermediate random cosine distance are found; the number of neighbor nodes of P is k1 = m1 + m2 + m3;

[0244] For each neighbor node of P, find n nodes with the largest cosine distance, and k2 = k1*n. Through the above steps, the number of first-order neighbor nodes of center node P is determined as k1, the number of second-order neighbor nodes is k2, and the theoretical total number of neighbor nodes of P is k = k1 + k2. In reality, there may be the same neighbors between each first-order node, so the number of neighbor nodes of P is less than or equal to k;

[0245] For each center node P, select p neighbor nodes to construct a connection between them to construct a subgraph, which is represented by an adjacency matrix A, and calculate the degree matrix D.

[0246] Repeat the above steps until all nodes are completed as center nodes to build subgraphs.

[0247] In the S4 step, different GCN neural network models are designed according to application scenarios and computing platforms, the probability of connection between neighbor nodes and the center node is calculated through the GCN neural network for each subgraph, each connection pair is formed, and the stability of the algorithm is improved through the average score algorithm. Finally, the clustering result is obtained by traversing each connection pair.

[0248] From the above formula (7), in order to extract the feature X of the node in the graph (l+1) The degree matrix D and the adjacency matrix A of the graph need to be calculated, the weight W and the bias b are parameters learned by the GCN through training, and X (l) is the feature representation of the node in the current state.

[0249] The application proposes a novel clustering algorithm based on GCN, which converts the clustering algorithm problem into a node connection prediction problem, that is, whether two nodes belong to the same category is determined by judging whether there is a connection between the two nodes. If the probability of connection between two nodes is greater than a set threshold, it is considered that there is a connection between the two nodes and they belong to the same category, otherwise they do not belong to the same category. Therefore, the application takes each node as a center node, finds the k neighbor nodes with the largest cosine distance to construct a subgraph, and then judges the probability of connection between the center node and its first-order neighbor nodes through GCN.

[0250] The application proposes two ways to construct a subgraph: fixed number of neighbor nodes and non-fixed number of nodes.

[0251] The fixed number of neighbor nodes means that a certain number of neighbor nodes are selected to construct a subgraph. In order to balance the number of positive and negative samples, the application not only finds the nearest nodes, but also finds the farthest nodes as negative samples. As shown in Figure 31 The algorithm flow is as follows:

[0252] S1: Calculate the cosine distance between each pair of nodes;

[0253] S2: Take each node as a center node P, find m1 neighbor nodes with the largest cosine distance, m2 neighbor nodes with the smallest cosine distance, and m3 neighbor nodes with the intermediate random cosine distance; the number of neighbor nodes of P is k1=m1+m2+m3;

[0254] S3: Find n nodes with the largest cosine distance for each neighbor node of P, k2=k1*n. Through the above steps, the number of first-order neighbor nodes of the center node P is k1, the number of second-order neighbor nodes is k2, and the theoretical total number of neighbor nodes of P is mp=k1+k2. In the real situation, there may be the same neighbor nodes between each first-order node, so the number of neighbor nodes of P is less than or equal to mp;

[0255] S4: Select k neighboring nodes for each central node P and build connections between them to construct a subgraph. Represent the subgraph with an adjacency matrix A and calculate its degree matrix D.

[0256] S5: Repeat steps S2-S4 until all nodes are used as center nodes to complete the subgraph construction.

[0257] like Figure 32 As shown in the example, each node is labeled 1-12, and the cosine distance between each of these 12 nodes is calculated. For each node as the center node, the two neighboring nodes with the largest cosine distance (k1=2) are calculated. Then, the two neighboring nodes with the largest cosine distance (k2=2) of each node are calculated. Finally, the k nearest neighbors of each node are selected and connected. For example, node 1 is taken as the center node, and the two neighboring nodes with the largest cosine distance (nodes 2 and 12) are found. Then, the neighbors of node 2 (nodes 6 and 12), the neighbors of node 12 (node ​​7 and node 2) are calculated. Finally, all nodes except the center node 1 are connected to complete the subgraph construction.

[0258] The non-fixed-number-node method is also a way to construct a subgraph when the number of selected neighbors is uncertain. This application uses the cosine distance radius method to find the neighbor nodes of the center node. A cosine distance radius R is set; if a node's distance from the center node is greater than radius R, it is considered a neighbor node of the center node. Most of the neighbor nodes found using this method are neighbors of the center node. To balance the number of positive and negative samples, a larger radius R1 is set, and nodes with a cosine distance less than R and greater than R1 are searched again. For example... Figure 33 As shown, the algorithm flow is as follows:

[0259] S1: Calculate the cosine distance between any two nodes;

[0260] S2: Treat each node as the center node P, set a cosine distance radius threshold R, and if it is less than the threshold, it is treated as a neighbor of the center node P.

[0261] S3: Expand the search range, reset the cosine distance radius threshold R1, and select nodes whose cosine distance is greater than R and less than R1;

[0262] S4: Select k nodes for each node to connect, constructing a subgraph. Represent the subgraph using an adjacency matrix A, and calculate its degree matrix D;

[0263] S5: Repeat steps S2-S4 until all nodes are used as center nodes to complete the subgraph construction.

[0264] like Figure 34As shown, for example, the nodes to be clustered are numbered 1-10, and the cosine distance between each pair of the 10 nodes is calculated; each node is taken as a center node P, and a cosine distance radius threshold R is set, and if it is less than the threshold, it is taken as a neighbor of the center node P; the search range is expanded, the cosine distance radius threshold R1 is reset, and the nodes with cosine distance greater than R and less than R1 are selected; and k nodes are selected for each node to be connected. For example, take node 1 as the center node, set the cosine distance radius to R1, and select nodes 2, 3, 4, and 5 as its neighbor nodes; reset the radius R2 to a larger range, and select node 6 as a neighbor node of node 1. However, the nearest neighbor node of node 6 is not within the range of k1 neighbor nodes, so node 6 is a negative sample and is not connected to any node.

[0265] The prior art also uses a fixed number of neighbor nodes to construct a subgraph for each node, but it uses Euclidean distance to calculate the distance between nodes, while the present application uses cosine similarity to calculate the distance between nodes. Euclidean distance reflects the absolute difference between numerical features, and is more used to reflect the difference from the size of the dimension value. Cosine distance distinguishes the difference between features from the direction. For example, the face feature difference of the same person at different poses, different illuminations, and different times is large, but still represents the same person, and if Euclidean distance is used, it will result in clustering into two people. Therefore, using cosine distance can effectively improve the accuracy of clustering.

[0266] Euclidean distance is based on the absolute numerical value of each dimension feature to measure feature similarity, and the calculation formula is:

[0267]

[0268] where dist(X, Y) represents the Euclidean distance between vectors x and y; x i represents the numerical value of the i-th dimension of vector x; y i represents the numerical value of the i-th dimension of vector y.

[0269] For two features A and B, the greater the Euclidean distance between them, the greater the difference between them; the greater the cosine distance, the smaller the difference between them. As Figure 35 shown, it can be seen that the cosine distance of the face of the same person at different poses, different illuminations, and different times is relatively large, indicating that it is the same person. The Euclidean distance is also relatively large, indicating that it may not be the same person. Figure 35 where O represents the Euclidean distance; and C represents the cosine distance.

[0270] The construction process of the subgraph of the image clustering method is described in detail above in combination with the drawings and examples. The process of obtaining the connection probability between the center node and each neighbor node based on the subgraph will be further described below.

[0271] In a possible implementation of the embodiment of the application, the step S220 of determining the connection probability between the center node and the neighbor nodes of the subgraph based on the graph convolution network can include the following steps.

[0272] (1) Obtain the node embedding of the subgraph according to the at least one layer of graph convolution network.

[0273] In the present application, the node embedding can be in the form of a matrix, and the node embedding can also be referred to as a feature representation or a feature matrix.

[0274] Specifically, the input feature corresponding to the subgraph can be obtained, and the input feature can include a feature matrix, an adjacency matrix, and a first-order neighbor vector. The input feature is input into the at least one layer of graph convolution network to obtain the node embedding of the subgraph.

[0275] In a specific embodiment, the feature matrix, the adjacency matrix, and the first-order neighbor vector can be input into the at least one layer of graph convolution network for feature extraction, and the output result of the last layer of graph convolution network can be taken as the node embedding.

[0276] Specifically, the matrix can be input into the multi-layer graph convolution network for transformation in sequence, and the structure of the graph convolution network can also be improved, for example, the pooling is added, or the results between two layers of graph convolution network are fused. The improvement of the graph convolution network will be described in detail below.

[0277] (2) Obtain the connection probability between the center node and the neighbor nodes of the subgraph based on the node embedding (also referred to as the feature representation or the feature matrix).

[0278] The neighbor node can be a first-order neighbor node, i.e., a neighbor node directly connected to the center node.

[0279] Specifically, the node embedding can be input into the at least one layer of fully connected layer (softmax layer) to obtain the connection probability between the center node and each first-order neighbor node of each subgraph.

[0280] In order to more intuitively represent the improvement of the structure of the graph convolution network in the present application, the graph convolution network in the prior art will be described first.

[0281] In the prior art solution, Figure 36As shown, a 4-layer GCN is used to extract node features. The adjacency matrix, feature matrix, and first-order neighbor vector of the subgraph are fed into the GCN network layer as input. Then, after two fully connected layers and a softmax layer, the probability of whether there is a connection between the first-order neighbor node and the center node is predicted. Cross-entropy is used as the loss function to train the entire neural network. Its network structure is shown in the figure.

[0282] In this diagram, block 1 represents the adjacency matrix A of the subgraph, which is an N*N matrix where N represents the number of nodes; block 2 represents the feature matrix F of the subgraph, which is an N*d matrix where d represents the dimension of the features (in the prior art, d = 512); block 3 represents the first-order neighbor vector, which is a 1*k1 matrix where k1 represents the number of first-order neighbors of the center node; block 4 represents the GCN network layer, which extracts node features; and block 5 represents the fully connected layer.

[0283] Depend on Figure 36 It is known that when nodes are related, meaning they belong to the same category, the differences between nodes of the same category decrease as the GCN network propagates forward, while the differences between nodes of different categories increase. If Euclidean distance is used, or if factors such as pose, lighting, and face photos taken at different times influence the process, it may lead to two nodes of different categories having very similar features, resulting in a connection between the nodes. As the GCN network propagates forward, the differences between these two nodes of different categories decrease, causing subsequent clustering errors. Therefore, this application adopts the following network structure to reduce the occurrence of this error.

[0284] The improvements to graph convolutional networks will be further explained in detail below with reference to the accompanying drawings and embodiments.

[0285] In one implementation, for at least one graph convolutional network other than the first graph convolutional network, the output of the previous graph convolutional network is used as the input of the at least one graph convolutional network.

[0286] Alternatively, the output of the previous graph convolutional network can be fused with the output of at least one graph convolutional network preceding the previous one, and then used as the input of the at least one graph convolutional network.

[0287] Specifically, the outputs of any two graph convolutional networks can be merged and used as the input to the next layer of the subsequent graph convolutional network.

[0288] In one example, such as Figure 37aAs shown, the shot-cut mechanism is introduced in the prior art, which can effectively reduce the occurrence of the above errors and improve the clustering accuracy. The multi-layer (such as 5 layers) GCN is used to extract node features. The adjacency matrix, feature matrix and first-order neighbor vector of the subgraph are transmitted as input to the GCN network layer, and then the connection probability between the first-order neighbor node and the center node is predicted through two fully connected layers and a softmax layer. Cross-entropy is used as the loss function to train the entire neural network, and the features of the shallow layer and the features of the later layer are fused. Therefore, the closer to the later layer of the GCN, the smaller the difference between the nodes of the same type, and the fusion of the shallow layer features into the later layer features can maintain the difference between different nodes to a certain extent, and avoid clustering different nodes into the same class. For example, the features extracted by the first layer of GCN are fused with the features extracted by the third layer and input to the fourth layer, and the features extracted by the second layer are fused with the features extracted by the fifth layer and output.

[0289] It can be understood that, Figure 37a The graph convolution network structure shown in the embodiment is 5 layers, and only for illustration, the specific number of layers of the graph convolution network structure is not limited in the specific implementation process. The output of any two layers of graph convolution network can be fused as the input of the next layer of the graph convolution network, which is not limited here.

[0290] As shown in Table 1, for the graph convolution network without introducing the shot-cut mechanism, the algorithm can effectively improve the precision, recall and f1-score of clustering.

[0291] Precision = Correct number of clustering / Number of clustering

[0292] Recall = Correct number of clustering / Number of positive samples in clustering

[0293] F1-score = Precision * Recall * 2 / (Precision + Recall)

[0294] Table 1: Comparison of effects before and after introducing the shot-cut mechanism

[0295]

[0296] In the above embodiment, the shot-cut mechanism is introduced on the graph convolution network, and the output results of different layers of the graph convolution network are fused. The closer to the later layer of the graph convolution network, the smaller the difference between the nodes of the same type, which can maintain the difference between different nodes to a certain extent, avoid clustering different nodes into the same class, and further improve the accuracy of clustering.

[0297] In another implementation process, inter-layer clustering can also be introduced to improve the structure of the graph convolution network.

[0298] Specifically, the clustering layer is arranged between at least two adjacent graph convolution networks, and the clustering layer clusters the feature vectors of the nodes contained in the output result of the previous layer of the graph convolution network (hereinafter also referred to as inter-layer clustering), and inputs the clustering result to the next layer of the graph convolution network.

[0299] Specifically, the feature vector of each node is of a first dimension; and the clustering result output by the clustering layer is of a second dimension; and the second dimension is smaller than the first dimension.

[0300] Specifically, at least two adjacent graph convolution networks after the clustering layer can also be provided with a dimension recovery layer, and the dimension recovery layer is used to recover the dimension of the output result of the clustering layer. The feature input to the dimension recovery layer is of the second dimension; and the input feature of the dimension recovery layer is of the first dimension, thereby improving the accuracy and efficiency of the entire clustering work, and making the clustering result of the final graph convolution network consistent with the dimension input to the graph convolution network.

[0301] As shown in FIG. 1, Figure 37b The inter-layer clustering module is introduced after each original layer of the GCN, and a differentiable inter-layer clustering operation is introduced in the present application. The inter-layer clustering can be used in any layer of the GCN, and it can fuse nodes belonging to the same class during the GCN transmission, that is, the nodes are clustered once in each GCN, thereby improving the accuracy and efficiency of the entire clustering work.

[0302] It can be understood that, Figure 37b The graph convolution network structure with the inter-layer clustering module shown in FIG. 1 has four layers, and only for the purpose of illustration, the specific number of layers of the graph convolution network structure is not limited in the specific implementation process. The output of any two layers of the graph convolution network can be fused as the input of the next layer of the next layer of the graph convolution network. In addition, each layer of the graph convolution network can have the inter-layer clustering operation, or only one layer or part of the graph convolution network can have the inter-layer clustering operation, which is not limited here.

[0303] In view of the problem that the conventional GCN can aggregate the features of the nodes, reduce the difference between the noise nodes and other nodes, and may cause clustering errors, the following improvements are made to the GCN structure in the present application:

[0304] 1. The shot-cut mechanism is used to maintain the difference between different nodes in the subgraph;

[0305] 2. The inter-layer clustering module in the GCN is proposed to extract the hierarchical features of the subgraph and increase the difference between the noise nodes and other nodes.

[0306] Figure 1 shows a schematic diagram of an image clustering method according to an embodiment of the present application. Figure 37b and 37c As shown in Figure 1, in one example, taking a face image as an example, the image clustering method can include the following steps:

[0307] S1, face detection is performed, and the face image is cropped and aligned;

[0308] S2, a feature vector of the face image is extracted using a CNN, i.e., a face feature shown in the figure, and the extracted face feature is taken as a node;

[0309] S3, taking each node as a center node, at least one neighbor node is determined according to the cosine distance, and a subgraph is constructed by connecting the center node and the neighbor node;

[0310] S4, the feature of each subgraph is extracted by a GCN; a shot-cut mechanism and an inter-layer clustering module are used to maintain the difference between the noise and other node features, and then a connection probability of the center node and its first-order neighbor nodes is obtained according to a softmax function;

[0311] The S4 step can include:

[0312] (1) taking an adjacency matrix, a feature matrix, and a neighbor node (which can be a first-order neighbor node) vector as the input of the GCN;

[0313] (2) using the first layer of the GCN to extract the feature of the subgraph;

[0314] (3) using the inter-layer clustering module to perform inter-layer clustering and feature extraction;

[0315] (4) in the second layer and the third layer, the features of the clustered nodes are extracted and the shallow layer features are aggregated through the shot-cut mechanism;

[0316] (5) the number of nodes is upgraded to the original dimension through a dimension lifting module;

[0317] (6) the features of the upgraded subgraph are extracted using the fourth layer of the GCN, and the features extracted by the fourth layer are fused with the features of the first layer of the GCN to maintain the gap between the nodes;

[0318] (7) the connection probability between the center node and its first-order neighbor nodes is predicted using a softmax.

[0319] S5, according to the connection probability of the center node and the first-order neighbor nodes, the face image is classified.

[0320] According to the connection probability between each node and its neighbor nodes, the connection relationship between all nodes can be obtained, and then the face image is classified according to the connection relationship between all nodes, wherein the average score can be used to remove the noise nodes (which will be described in detail later).

[0321] Figure 37c Only steps S3-S5 are shown, and the shot-cut mechanism in the GCN can fuse the features of different layers to maintain the difference between the noise and other node features.

[0322] As Figure 37d shown, Figure 37d shows a processing process of simulating a GCN in an example, where G represents a subgraph, and node 2 represents a noise node. A represents a feature matrix extracted from the second-to-last layer of the GCN, and B represents a feature matrix after shot-cut.

[0323] As shown in Table 2, AA represents the cosine distance between features without shot-cut, and BB represents the cosine distance between features after adopting the shot-cut mechanism. After adding the shot-cut, the similarity between the noise node and other nodes is reduced, and the clustering performance is improved.

[0324] Table 2: Comparison of cosine distances before and after introducing the shot-cut mechanism

[0325]

[0326] As can be seen from Table 1, the shot-cut mechanism can preserve the difference between the features of different nodes, effectively eliminate noise nodes, and improve the accuracy of subsequent clustering.

[0327] As Figure 38 shown, in an example, an inter-layer clustering module can be introduced in a regular GCN layer to cluster nodes with strong relationships and normalize features to determine noise nodes.

[0328] We only simply simulate the inter-layer clustering process of the GCN. B represents the result of inter-layer clustering, and C represents that if the node belongs to the same category, the node is represented by the feature of the center node. It can be seen that the inter-layer clustering module can effectively remove noise nodes, solve the problem that the traditional GCN aggregates the features of nodes, reduces the difference between noise nodes and other nodes, and ultimately leads to clustering errors.

[0329] As Figure 39aAs shown, nodes in boxes 1 and 2 belong to the same node class A, nodes in box 3 belong to the same node class B, and nodes in box 4 belong to the same node class C. One node in class A is close to a node in box 2, indicating a connection. If processed through a GCN network, the features of nodes in box 2 and box 3 would become increasingly similar, affecting subsequent clustering results. However, the inter-layer clustering module in this application performs a small clustering of each node within the GCN, fusing the features of similar nodes and reducing the impact of a few erroneous nodes.

[0330] like Figure 39b As shown, node 3 does not belong to the same class as nodes 0, 1, and 2. However, due to factors such as lighting and pose, the cosine distance between nodes 0 and 3 is relatively large. If a traditional clustering algorithm is used, the high similarity between nodes 0 and 3 would lead to clustering the two classes into one, resulting in a clustering error. However, this invention employs inter-layer clustering, determining whether two nodes belong to the same class by observing the information of their neighboring nodes. If they belong to the same class, they are clustered into the same subclass in the GCN; otherwise, they are not clustered together. Simultaneously, a dimension recovery module is used to represent the features of the neighboring nodes, increasing the inter-class distance. For example, although nodes 0 and 3 have high similarity, their neighbor information indicates that they belong to different classes. Inter-layer clustering groups them into separate subclasses, represented by neighboring node information, increasing the difference between the features of nodes 0 and 3. Figure 39b The value 0.2 represents the average connection probability between nodes 0 and 3. Since it is less than the set threshold, nodes 0 and 3 can be disconnected. Therefore, in subsequent clustering, nodes 0 and 3 will be avoided from being clustered into the same class.

[0331] Based on the above GCN neural network model, this scheme reduces the number of GCN network layers by one, and adds an inter-layer clustering module and a dimension recovery module after each GCN layer, as shown in Table 3. This module improves the F1-score by 0.0126 compared to the above improved model.

[0332] Table 3: Comparison of the effects of existing technologies, the introduction of inter-layer clustering, and shot-cut mechanism

[0333]

[0334]

[0335] Define a subgraph G, and for each subgraph use an adjacency matrix A( Figure 37b In the square (1), A represents an N*N matrix whose elements are related to whether nodes are connected. A value of 1 indicates a connection between nodes, and 0 indicates a connection between nodes. Therefore, A ∈ {0, 1}. N×N. The feature matrix F is constructed by the feature vector of each node in the graph G Figure 37b Block 2), F is an N*D dimensional matrix, D represents the dimension of the feature vector of each node, The present application is a way of clustering by predicting the probability between neighbor nodes and the center node, so Node represents the first-order neighbor node label Figure 37b Block 3), Node is a 1*n1 dimensional vector, n1 represents the number of first-order neighbor nodes of the center node, Figure 37b Block 4 represents the GCN layer, the formula is:

[0336] H l+1 =σ(MH l W l +b l ) (10)

[0337] Where H l+1 represents the feature of each node in the next layer of GCN, H l represents the feature representation of the current layer node, H l =F; W l represents the weight matrix of the lth layer, which is a DxDl dimensional matrix, b l represents the bias of the lth layer, M represents the state value of the normalized node adjacency matrix, represents the adjacency matrix, represents the degree matrix of the adjacency matrix, and σ represents a nonlinear activation layer, and the present application uses ReLU as the activation function.

[0338] Because the graph structure is an unstructured data, the pooling operation in digital image cannot be directly used, so the present application realizes the interlayer clustering function through a learnable conversion factor S (l) . S (l) is an N*N1 dimensional matrix, representing the probability of each node belonging to different categories, N1 represents clustering N nodes in the graph into N1 nodes. S (l) is learned by a layer of GCN, so it can be directly embedded into any layer of the entire GCN neural network, Z l represents the node clustering factor:

[0339] S (l) =softmax(GCN(A l ,X l )) (11)

[0340] Z (l) =GCN(A l ,X l(12)

[0341] Clustering nodes using transformation factors:

[0342] X (l+1) =S (l)T Z l (13)

[0343] Therefore, the node feature matrix after intra-layer GCN clustering is X. l+1 , is an N1*D dimension matrix. Because X l+1 It is the feature matrix of the nodes after clustering, so it is necessary to reconstruct the adjacency matrix A between the nodes. (l+1) :

[0344] A (l+1) =S (l)T A l S (l) (14)

[0345] Among them, S (l) This represents the transformation matrix, which is also the inter-layer clustering matrix.

[0346] The adjacency matrix A of the clustered nodes can be obtained using the above formula. l+1 and characteristic matrix X l+1 .

[0347] In the above embodiments, inter-layer clustering is introduced after each layer of the original graph convolutional network. Inter-layer clustering can be used in any layer of the graph convolutional network. It can merge nodes of the same type during the transmission of the graph convolutional network. In other words, the clustering function of the nodes has been completed once within each graph convolutional network, thereby improving the accuracy and efficiency of the entire clustering work.

[0348] like Figure 40a As shown, in one example, the process of inter-layer clustering and dimensionality recovery may include the following steps:

[0349] S1 takes the adjacency matrix, the feature matrix, and the neighbor node (specifically, the first-order neighbor node) vector as input to the GCN;

[0350] S1, use the first layer of GCN to extract features from the subgraph; use the inter-layer clustering module to perform inter-layer clustering and feature extraction;

[0351] S3, in the second and third layers of GCN, extracts the features of cluster nodes and aggregates shallow features through a shot-cut mechanism;

[0352] S4 restores the number of nodes to the original dimension through the dimension enhancement module.

[0353] In S1, the neighbor matrix, feature matrix, and first-order neighbor matrix are used to extract node features through GCN. In S2, the inter-layer clustering module is used to cluster nodes with the same identity into one class. In S3, the clustered feature nodes represent the features of the original nodes. In S4, the dimension restoration module is used to restore the subgraph dimension, and the clustered features represent the features of nodes in the same subclass.

[0354]

[0355] Among them, M (l) This indicates the category to which each node belongs after inter-layer clustering.

[0356] Using M in the Dimension Recovery module (l) and X (l+1) The recovered feature matrix is ​​obtained. For example, in Figure 40a In step S2, all nodes (0,1,2,3) are clustered into two classes (class 0 and class 1) using the inter-layer clustering module. (l) ={0,0,1,0}, indicating that nodes 0,1,3 belong to category 0, and node 2 belongs to category 1. Through M (l) As can be seen, nodes 0, 1, and 3 have the same identity, so in S4, the characteristics of node 4 are represented as those of nodes 0, 1, and 3.

[0357] like Figure 40b As shown, Figure 40b The process of dimension recovery is illustrated. According to Formula 15, the category to which each node belongs can be determined. During dimension recovery, the feature vector of the node at the corresponding position is used as the feature vector of the recovered position. For example, the inter-layer clustering module clusters all nodes (0,1,2,3) into two categories (category 0 and 1). The clustered categories are defined by M in step S2. (l) Let {0,0,1,0} represent the clustering pattern, where nodes 0, 1, and 3 belong to category 0, and node 2 belongs to category 1. The feature vector at position 0 after clustering is used as the feature vector of the node after dimension recovery, i.e., the feature vector of nodes 0, 1, and 3. The same logic applies to other nodes. M in Formula 15... (l) It can also be used to calculate the recovered neighbor matrix representation in the dimension recovery module:

[0358]

[0359]

[0360] Among them, A (r) The sum of the adjacency matrix of the subgraph after the dimension recovery module and the adjacency matrix of the subgraph in the previous layer represents the final adjacency matrix of the dimension recovery module; A (l+1) A is used to represent the adjacency matrix of the subgraph in the previous layer; (′)a subgraph adjacency matrix after representing the dimension recovery module; A (′) [i][j] has the same meaning as A (′) .

[0361] The above embodiments elaborate on the improvement of the graph convolution network, including the shot-cut mechanism and the introduction of the inter-layer clustering module. The process of clustering target regions according to connection probabilities will be described below.

[0362] In one implementation process, the step S220 of clustering each target region according to the determined connection probability can include:

[0363] For at least one target region to be clustered, if the connection probability between any first-order neighbor node and the center node in the subgraph corresponding to the target region is greater than a preset threshold, the target region corresponding to the first-order neighbor node and the target region are regarded as the same category.

[0364] Specifically, as long as the connection probability between the first-order neighbor node and the center node is greater than the preset threshold, the target region corresponding to the first-order neighbor node and the target region corresponding to the center node are the same category.

[0365] After passing through the GCN neural network, the probability value pair between each center node and its first-order neighbor node is obtained, and then each prediction result is traversed. If it is lower than the set threshold, it means that there is no connection between the center node and the neighbor node. Finally, through the breadth-first algorithm, each edge is traversed, and the nodes with a connection probability greater than the threshold are connected, and the clustering is completed.

[0366] As Figure 41 shown, each clustered node is taken as a center node, and its neighbor nodes are found through the cosine distance to construct a subgraph with the center node as a unit. Each subgraph is predicted by the GCN neural network to obtain the probability of the existence of a connection between the center node and the first-order neighbor node. If the probability of the existence of a connection between two nodes is greater than a threshold, the two nodes are connected. Finally, each connection pair is traversed to complete the clustering.

[0367] In another implementation process, the step S220 of clustering each target region according to the determined connection probability can include:

[0368] (1) For at least one target region to be clustered, a same-category neighbor node set is obtained according to the connection probability between the corresponding center node and the neighbor node.

[0369] Among the same-category neighbor node set, at least one neighbor node determined to be in the same category as the center node is included.

[0370] Specifically, for at least one target region to be clustered, a same-class neighbor node set can be obtained according to the connection probability between the corresponding center node and the neighbor node, which can include:

[0371] For at least one target region to be clustered, a neighbor node with a connection probability with the center node not less than a preset threshold is determined, and the determined neighbor node is grouped into a same-class neighbor node set.

[0372] (2) For at least one neighbor node in the same-class neighbor node set, the connection probability between the at least one neighbor node and each neighbor node in the same-class neighbor node set is adjusted.

[0373] Specifically, for at least one neighbor node in the same-class neighbor node set, the average connection probability of the at least one neighbor node and each neighbor node in the same-class neighbor node set can be determined, and the average connection probability can be used as the connection probability between the at least one neighbor node and the center node.

[0374] (3) Based on the adjusted connection probability, the target region is clustered to determine whether the at least one neighbor node is a same-class neighbor node.

[0375] Specifically, if the adjusted connection probability is greater than or equal to a first preset threshold, the neighbor node corresponding to the adjusted connection probability can be set as a same-class neighbor node.

[0376] (4) According to the same-class neighbor nodes respectively corresponding to each target region to be clustered, the target regions to be clustered are clustered.

[0377] Specifically, according to the same-class neighbor nodes respectively corresponding to each target region to be clustered, the target regions to be clustered are clustered, which can include:

[0378] a. According to the same-class neighbor nodes respectively corresponding to each target region to be clustered, the connection probability between the center nodes respectively corresponding to each target region to be clustered is determined.

[0379] b. Based on the connection probability between the center nodes, the target regions to be clustered are clustered.

[0380] Specifically, if the connection probability between the center nodes is greater than or equal to a second preset threshold, the target region corresponding to the center node can be set as the same category.

[0381] Specifically, the application proposes an average score algorithm to enhance the stability of the algorithm. For each subgraph, the probability between the center node and its neighbor nodes is predicted by GCN, and the set C of all nodes connected to A (nodes with a probability greater than the threshold) is obtained. If the probability of connection between nodes A and B is greater than the set threshold, the average probability of all nodes connected to B in set C is calculated as the value of the probability of connection between nodes A and B. Finally, clustering is completed by traversing each connection pair. By considering the connection information of multiple nodes with B, the stability of A and B not being nodes of the same class can be enhanced.

[0382] As shown in Figure 42a , for example: A, B, C, D, E are nodes of the same class, the probability of connection between F node and A node is greater than 0.9, but the probability of connection between F and B, C, D, E is very small, which is 0.05, 0.1, 0.2, 0.15. The average probability 0.1 is taken as the probability value of A and F node, which can indicate that the probability of connection between A and F is very small, thereby improving the accuracy of the algorithm.

[0383] As shown in Figure 42b , in an example, the process of determining noise nodes can include the following steps:

[0384] S1, traverse all nodes, for any node, set the node whose connection probability is greater than the preset threshold and the number of neighbor nodes of the node is less than the preset node number as an isolated node; for example, the preset node number can be 2;

[0385] S2, sort the isolated nodes and their second-order neighbor nodes (i.e. neighbors of the neighbors) according to the connection probability;

[0386] S3, take the average value of the connection probability as the connection probability of the isolated node and its neighbor nodes;

[0387] S4, if the connection probability is less than the set threshold, the node is a noise node (Noise).

[0388] In order to improve the accuracy and stability of clustering, the average algorithm is proposed in the above example to determine whether the isolated node is a noise node according to the neighbor relationship. A, B, C, D and E are nodes of the same class, the connection probability between F and A is greater than 0.9, but the connection probability between F and B, C, D, E is very small, which is 0.05, 0.1, 0 and 0. The average value of the connection probability of F and A, B, C, D, E is taken as the connection probability of F and A, based on which it can be determined that F is a noise node of class A, i.e. F does not belong to class A. The average algorithm can effectively avoid clustering errors and improve the accuracy and stability of clustering.

[0389] The above illustrates different ways of clustering according to connection probability. The above clustering process is clustering all face images to be clustered, but in a small device such as a mobile phone with limited computing resources, an initial clustering and a later additional clustering mode is usually used. As the number of face images to be clustered increases, the time and space complexity of the clustering algorithm proposed in the present application also increases, requiring more consumption of computing resources. At the same time, when users use a mobile phone to take pictures, they do not take all the pictures at once, but gradually accumulate over time. Therefore, through the additional mode, users can quickly see the clustering results and enhance the user experience.

[0390] Post-processing:

[0391] After clustering, the same person may be divided into two categories, as shown in Figure 42c The main reasons for the above problems are: (1) In the case of a large number of photos of the same face, the neural network can effectively learn the global context relationship of the constructed subgraph, but may ignore the influence of local information on the clustering results to some extent. By extracting the global context relationship, the difference of the features of the face with a large pose is gradually enlarged, and finally the same class is clustered into two classes.

[0392] Therefore, in view of the above problems, the present application proposes a feasible post-processing scheme.

[0393] In one possible implementation of the embodiment of the present application, the image clustering method further comprises:

[0394] (1) respectively determining clustering centers of each class obtained by clustering.

[0395] The clustering result can be a plurality of classes obtained by clustering the images, and each class corresponds to a plurality of images.

[0396] Specifically, respectively determining the clustering centers of each class obtained by clustering can include: for any class obtained by clustering, setting the mean of the feature vectors contained in the class as the clustering center of the class.

[0397] In the specific implementation process, the images corresponding to each class are obtained, and the mean of the feature vectors of the images corresponding to each class is calculated. The calculated mean is set as the clustering center of the one class.

[0398] (2) If the similarity of two clustering centers is not less than a set value, the classes corresponding to the two clustering centers are merged into the same class, and the clustering result is updated.

[0399] Specifically, for all classes in the clustering result, the similarity between the clustering centers of each two classes can be calculated respectively, for example, the similarity can be calculated in the form of cosine distance, if the similarity is not less than a set value, it means that the two classes can be merged into the same class.

[0400] As shown in Figure 42d , all images are clustered into 3 classes, i.e. class 1, 2 and 3 in the figure, but the error result of the same person being clustered into two classes occurs. As the photos of the same person in class "1" and "2" are, but are two classes that are wrongly clustered. We propose a post-processing algorithm to solve this problem. The first step is to calculate the clustering center of each class. As shown in Figure 42d , the feature vectors of the images in each class are summed and averaged to obtain the clustering center 1, the clustering center 2 and the clustering center 3 of the three classes. The second step is to calculate the cosine distance between each two clustering centers. As shown in Figure 42d , the cosine distance between each clustering center and other clustering centers is calculated respectively to obtain d1, d2 and d3. The third step is to determine whether to merge two classes according to the cosine distance. The cosine distance d1 is greater than a set value th, which means that class 1 and class 2 belong to the same class, and class 1 and class 2 are merged; the cosine distances d2 and d3 are less than the set threshold, which means that class 2 and class 3, and class 1 and class 3 do not belong to the same class, and the original classes remain unchanged. Through the above process, the clustering result is updated to obtain the final clustering result.

[0401] In one possible implementation of the embodiments of the present application, the image clustering method further includes:

[0402] (1) obtaining a new target region to be clustered;

[0403] (2) obtaining a preset number of target regions in each of the at least one clustered class;

[0404] (3) clustering the obtained clustered target regions and the new target region to be clustered based on a graph convolution network to obtain a new clustering result;

[0405] (4) determining the class of the new target region to be clustered based on the new clustering result and the classes corresponding to the clustered target regions.

[0406] Specifically, based on the obtained clustering result, when the new target region to be clustered needs to be clustered, a preset number of images can be obtained from each of the clustered classes, and the clustered target regions and the new target region to be clustered are clustered to obtain a new clustering result, so as to determine the class of the new target region to be clustered, which can be applied to small devices such as mobile phones with limited computing resources, improve the clustering efficiency, enable the user to quickly see the clustering result, and enhance the user experience.

[0407] like Figure 43 As shown, at the initial clustering node, a set number N faces are selected for clustering, resulting in n face categories, i.e., the existing clusters. When a new photo is detected, face detection, face alignment, and face feature extraction are performed on the new photo to obtain the features of each face. From the n already clustered categories, n2 face images are randomly selected as representatives of that category and merged with the new face image. The merged image is then clustered simultaneously. The proportion (Ratio) of the new photo in the existing clustering results is calculated. If it is greater than a set proportion of 1 (e.g., 0.5), it is merged into the existing clustering results. If it is less than a set proportion of 2 (e.g., 0.2), it is treated as a new category. If it is greater than 0.2 and less than 0.5, no action is taken, and it will be re-clustered in the next clustering.

[0408] To better understand the application of the image clustering method of this application, the following will illustrate the image clustering method of this application with examples.

[0409] In one possible implementation of this application embodiment, the image clustering method may further include:

[0410] (1) Receive keywords for image search input by the user;

[0411] (2) Determine the categories associated with the keywords based on the clustering results;

[0412] (3) Search for images that match the keywords in the determined categories of images.

[0413] Specifically, when users need to search for images that match keywords among a large number of images, direct searching may result in low accuracy and long processing time. Instead, they can first identify the category associated with the keywords from multiple clustered categories.

[0414] For example, when you need to search for a person's name among a large number of images of various categories, you can identify the category "female celebrity" from the already clustered categories, and then perform a keyword search from the identified category. This can effectively improve search efficiency and reduce search time.

[0415] In one possible implementation of this application embodiment, the image clustering method may further include:

[0416] (1) Obtain multiple labels to be labeled;

[0417] (2) Based on the clustering results, the labels are marked on the target regions of each of the clustered categories.

[0418] Specifically, in the process of labeling a large number of images of various categories, the target regions can be clustered first to determine which categories the target regions can be clustered into, and then the closest category of each label to be labeled is determined respectively for automatic labeling, which can effectively improve the labeling efficiency.

[0419] According to the requirements of application scenarios and hardware environments, different schemes can be used to complete corresponding tasks.

[0420] 1) High-performance clustering scheme for mobile terminal

[0421] The mobile terminal integration method needs low computational resource consumption, and uses a low-channel, low-dimension and low-input scheme in network model design. In the clustering logic, an initial clustering and an additional clustering scheme are used, as shown in Figure 44 The specific workflow is as follows:

[0422] S1: Perform face detection, face alignment and face feature extraction (extract 128-dimensional face features) on the photos in the mobile terminal album;

[0423] S2: Use GCN to predict the probability of node existence connection for the existing face features, complete clustering through a post-processing algorithm, and obtain an initial clustering result;

[0424] S3: If a new face feature is obtained, select 6 face features as its representatives in the already clustered categories to form a new clustering set, and determine whether the number of face features to be clustered is greater than a set threshold (such as 20). If it is less than 20, set the id of all faces to -1 and wait for the next clustering. If it is greater than 20, perform clustering;

[0425] S4: Use GCN to predict the probability of node existence connection, and complete clustering through a post-processing algorithm;

[0426] S5: Determine whether the number of clustered categories is greater than a set number (such as 5). If it is less than 5, set the id of all faces to -1 and wait for the next clustering. If it is greater than 5, proceed to the next step;

[0427] S6: Calculate the proportion of the new photo in the existing clustering result in the new clustering result. If it is greater than a set proportion 1 (such as 0.5), merge it into the existing clustering result. If it is less than a set proportion 2 (such as 0.2), treat it as a new category. If it is greater than 0.2 and less than 0.5, do not perform any operation, and perform re-clustering in the next clustering.

[0428] The neural network model of the mobile terminal is as shown in Figure 45As shown in the figure, wherein block 1 represents the adjacency matrix A of the subgraph (subgraph), A is an N*N matrix, N=31 represents the number of nodes; block 2 represents the feature matrix F of the subgraph, F is an N*d matrix, d=128 represents the dimension of the feature; block 3 represents the label Node of the first-order neighbor, Node is a 1*k1 matrix, k1=10 represents the number of first-order neighbors of the center node; block 4, block 5 and block 6 all represent GCN network layers, which perform node feature extraction, different color depths represent different output channels of the GCN network layer, and the narrower it is, the smaller the channel is, and the corresponding calculation amount is also small; block 7 represents a fully connected layer, which predicts the probability of the existence of a connection between the center node and its first-order neighbor nodes.

[0429] 2) Server-side high-performance solution

[0430] Large servers have strong computing power and large memory, so large neural network models are used to improve clustering accuracy. The neural network structure of inter-layer clustering is introduced, that is, Figure 37b The neural network structure shown in the figure has a high F1-score on multiple test sets and can be used for tasks that require F1-score indicators. The neural network with a shot-cut mechanism, that is, Figure 37a The neural network shown in the figure has high accuracy and is suitable for tasks that require accuracy indicators. As shown in Figure 46 The deployment process is as follows:

[0431] S1: Perform face detection, face alignment and face feature extraction on the photos in the album;

[0432] S2: Use GCN to predict the probability of the existence of a connection between the existing face features, complete clustering through post-processing algorithms, and obtain the initial clustering result;

[0433] S3: When a new face photo is obtained, merge it with the original face photo and re-cluster.

[0434] The above image clustering method, the present application obtains the clustering of the target region of each image through the graph convolution network, does not need to determine the clustering center, avoids the deviation of the entire clustering result caused by the deviation of the clustering center, and can effectively improve the accuracy of the clustering result.

[0435] Further, the cosine similarity is used to calculate the distance between nodes. The cosine distance distinguishes the difference between features from the direction. The face feature difference of the same person in different poses, different illuminations and different time points is large, but still represents the same person. Compared with the Euclidean distance, which reflects the absolute difference in numerical values, if the Euclidean distance is used, it will lead to clustering into two people, therefore, the use of cosine distance can further improve the accuracy of clustering.

[0436] Further, the shot-cut mechanism is introduced on the graph convolution network to fuse the output results of different layers of the graph convolution network. The difference between nodes of the same type becomes smaller and smaller as the layer of the graph convolution network is closer to the rear layer, which can maintain the difference between different nodes to some extent, avoid clustering different nodes into the same class, and further improve the accuracy of clustering.

[0437] Further, the inter-layer clustering is introduced behind the original at least one layer of the graph convolution network. The inter-layer clustering can be used at any layer of the graph convolution network. The inter-layer clustering can fuse nodes belonging to the same class during the transmission of the graph convolution network, that is, the nodes are clustered once in the graph convolution network, thereby improving the accuracy and efficiency of the entire clustering work.

[0438] Further, based on the clustering, when a new target region to be clustered is obtained, a preset number of images can be obtained from each category of the clustered images, and the new target region to be clustered is clustered based on the clustered images and the new target region to be clustered to obtain a new clustering result, so as to determine the category of the new target region to be clustered. The method can be applied to a small device such as a mobile phone with limited computing resources, improve the clustering efficiency, enable the user to quickly see the clustering result, and enhance the user experience.

[0439] The above embodiment introduces the image clustering method from the perspective of method flow. The following is introduced from the perspective of a virtual module, and is specifically as follows:

[0440] An image clustering apparatus 700 is provided in the embodiment of the present application, as shown in the figure, the apparatus 700 can include an acquisition module 701 and a first clustering module 702, wherein: Figure 47

[0441] The acquisition module 701 is configured to acquire a target region of an image to be clustered, and acquire a feature vector of the target region.

[0442] The first clustering module 702 is configured to cluster the acquired feature vector based on a graph convolution network to obtain a clustering result.

[0443] In one possible implementation of the embodiment of the present application, when the first clustering module 702 clusters the acquired feature vector based on the graph convolution network to obtain the clustering result, the first clustering module 702 is specifically configured to:

[0444] construct a subgraph corresponding to the feature vector, wherein the subgraph includes a center node corresponding to the feature vector and at least one neighbor node corresponding to another feature vector;

[0445] determine a connection probability between the center node of the subgraph and the neighbor node of the subgraph based on the graph convolution network, and cluster the target region according to the determined connection probability to obtain the clustering result. ​

[0446] In a possible implementation of the application, the first clustering module 702, when constructing the subgraph corresponding to the feature vector, is specifically configured to:

[0447] obtain neighbor nodes of the center node from other feature vectors, taking the feature vector as the center node.

[0448] construct the subgraph according to the center node and the neighbor nodes.

[0449] In a possible implementation of the application, the first clustering module 702, when obtaining neighbor nodes of the center node from other feature vectors, taking the feature vector as the center node, is specifically configured to:

[0450] respectively obtain cosine distances between the feature vector and other feature vectors;

[0451] obtain neighbor nodes of the feature vector from other feature vectors based on the cosine distances between the feature vector and other feature vectors.

[0452] In a possible implementation of the application, the first clustering module 702, when determining the connection probability between the center node and the neighbor nodes of the subgraph based on the graph convolution network, is specifically configured to:

[0453] obtain node embedding of the subgraph according to at least one layer of the graph convolution network;

[0454] obtain the connection probability between the center node and the neighbor nodes of the subgraph based on the node embedding.

[0455] In a possible implementation of the application, the first clustering module 702, when obtaining the node embedding of the subgraph according to at least one layer of the graph convolution network, is specifically configured to:

[0456] obtain input features corresponding to the subgraph;

[0457] input the input features into at least one layer of the graph convolution network for feature extraction, and take an output result of the last layer of the graph convolution network as the node embedding.

[0458] In a possible implementation of the application, for at least one layer of the graph convolution network except the first layer of the graph convolution network, an output result of a previous layer of the graph convolution network is taken as an input of the at least one layer of the graph convolution network.

[0459] Or the output result of the previous layer of the graph convolution network and an output result of at least one layer of the graph convolution network before the previous layer of the graph convolution network are fused and taken as the input of the at least one layer of the graph convolution network.

[0460] In a possible implementation of the embodiment of the application, the clustering layer is arranged between at least two adjacent graph convolution networks, and the clustering layer clusters feature vectors of nodes included in an output result of a previous graph convolution network and inputs a clustering result to a next graph convolution network.

[0461] In a possible implementation of the embodiment of the application, the first clustering module 702, when clustering the target region according to the determined connection probability, is specifically configured to:

[0462] For at least one target region to be clustered, a same-class neighbor node set is obtained according to a connection probability between the corresponding center node and a neighbor node.

[0463] For at least one neighbor node in the same-class neighbor node set, a connection probability between the at least one neighbor node and the center node is adjusted according to a connection probability between the at least one neighbor node and each neighbor node in the same-class neighbor node set.

[0464] The target region is clustered based on the adjusted connection probability to determine whether the at least one neighbor node is a same-class neighbor node.

[0465] The target regions to be clustered are clustered according to the same-class neighbor nodes corresponding to the target regions respectively.

[0466] In a possible implementation of the embodiment of the application, the first clustering module 702, when obtaining the same-class neighbor node set for at least one target region to be clustered according to a connection probability between the corresponding center node and a neighbor node, is specifically configured to:

[0467] For at least one target region to be clustered, a neighbor node with a connection probability not less than a preset threshold value is determined between the center node, and the determined neighbor node is composed into a same-class neighbor node set.

[0468] In a possible implementation of the embodiment of the application, the first clustering module 702, when adjusting the connection probability between the at least one neighbor node and the center node for at least one neighbor node in the same-class neighbor node set according to a connection probability between the at least one neighbor node and each neighbor node in the same-class neighbor node set, is specifically configured to:

[0469] For at least one neighbor node in the same-class neighbor node set, an average connection probability of the at least one neighbor node and each neighbor node in the same-class neighbor node set is determined, and the average connection probability is taken as the connection probability between the at least one neighbor node and the center node.

[0470] In a possible implementation of the embodiment of the application, the first clustering module 702 is specifically configured to:

[0471] determine the connection probability between the center nodes corresponding to the target regions to be clustered according to the same type neighbor nodes corresponding to the target regions to be clustered respectively;

[0472] cluster the target regions to be clustered based on the connection probability between the center nodes.

[0473] In a possible implementation of the embodiment of the application, the image clustering apparatus further includes a searching module, configured to:

[0474] receive a keyword input by a user for image searching;

[0475] determine a category associated with the keyword based on the clustering result;

[0476] search for images meeting the keyword in the images of the determined category.

[0477] In a possible implementation of the embodiment of the application, the image clustering apparatus further includes a labeling module, configured to:

[0478] obtain a plurality of labels to be labeled;

[0479] label the labels on the target regions of the images of each category clustered based on the clustering result.

[0480] In a possible implementation of the embodiment of the application, the image clustering apparatus further includes an updating module, configured to:

[0481] determine the clustering center of each category obtained by clustering respectively;

[0482] if the similarity of two clustering centers is not less than a set value, merge the categories corresponding to the two clustering centers into the same category, to update the clustering result.

[0483] In a possible implementation of the embodiment of the application, the updating module is specifically configured to:

[0484] for the category obtained by clustering, set the mean of the feature vectors contained in the category as the clustering center of the category.

[0485] In a possible implementation of the embodiment of the application, the image clustering apparatus further includes a second clustering module, configured to:

[0486] obtain a target region of a new image to be clustered;

[0487] Respectively obtain a preset number of target regions in the at least one clustered category;

[0488] Cluster the obtained clustered target regions and the new target region to be clustered based on the graph convolution network to obtain a new clustering result;

[0489] Determine the category of the new target region to be clustered based on the new clustering result and the category corresponding to the clustered target region.

[0490] The image clustering device described above obtains clustering of target regions of each image through the graph convolution network, does not need to determine a clustering center, avoids deviation of the entire clustering result caused by deviation of the clustering center, and can effectively improve the accuracy of the clustering result.

[0491] Further, the cosine similarity is used to calculate the distance between nodes. The cosine distance distinguishes the difference between features from the direction. The face feature difference of the same person in different poses, different illuminations and different time points is large, but still represents the same person. Compared with the Euclidean distance, the cosine distance reflects the absolute difference in the numerical value of the feature. If the Euclidean distance is used, it will lead to clustering of two people. Therefore, the use of the cosine distance can further improve the accuracy of clustering.

[0492] Further, the shot-cut mechanism is introduced on the graph convolution network to fuse the output results of the graph convolution network of different layers. The difference between nodes of the same category becomes smaller and smaller as the layer of the graph convolution network is closer to the rear layer. To some extent, the difference between different nodes can be maintained, and different nodes can be avoided from being clustered into the same category, thereby further improving the accuracy of clustering.

[0493] Further, the inter-layer clustering is introduced behind the original at least one layer of the graph convolution network. The inter-layer clustering can be used at any layer of the graph convolution network. It can fuse nodes belonging to the same category during the transmission of the graph convolution network, that is, the nodes have been clustered once inside the graph convolution network, thereby improving the precision and efficiency of the entire clustering work.

[0494] Further, on the basis of clustering, when a new target region to be clustered is obtained, a preset number of images can be obtained from each category of the clustered images. The new target region to be clustered is clustered based on the clustered images to obtain a new clustering result, thereby determining the category of the new target region to be clustered. This method can be applied to devices with limited computing resources such as mobile phones, improves the clustering efficiency, allows users to quickly see the clustering result, and enhances the user experience.

[0495] The image clustering apparatus can perform the image clustering method provided by the embodiments of the present disclosure, and the implementation principles are similar. The actions performed by each module in the image clustering apparatus in each embodiment of the present disclosure correspond to the steps in the image clustering method in each embodiment of the present disclosure. For the detailed functions of each module of the image clustering apparatus, refer to the description of the corresponding image clustering method in the foregoing, which will not be described here again.

[0496] The apparatus provided in the embodiments of the present application can implement at least one of the plurality of modules through an AI (Artificial Intelligence) model. The functions associated with AI can be performed through a non-volatile memory, a volatile memory, and a processor.

[0497] The processor can include one or more processors. At this time, the one or more processors can be a general-purpose processor such as a central processing unit (CPU), an application processor (AP), etc., or a pure graphics processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-dedicated processor such as a neural processing unit (NPU).

[0498] The one or more processors control the processing of the input data according to a predefined operation rule or an artificial intelligence (AI) model stored in the non-volatile memory and the volatile memory. The predefined operation rule or the artificial intelligence model is provided through training or learning.

[0499] Here, the provision through learning means that a predefined operation rule or an AI model having a desired characteristic is obtained by applying a learning algorithm to a plurality of learning data. The learning can be performed in the apparatus itself according to the embodiments, and / or can be implemented through a separate server / system.

[0500] The AI model can include a plurality of neural network layers. Each layer has a plurality of weight values, and the calculation of one layer is performed by the calculation result of the previous layer and the plurality of weights of the current layer. Examples of neural networks include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a generative adversarial network (GAN), and a deep Q-network.

[0501] The learning algorithm is a method of training a predetermined target apparatus (e.g., a robot) using a plurality of learning data to enable, allow, or control the target apparatus to determine or predict. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0502] The image clustering apparatus provided by the embodiments of the present application is introduced from the perspective of functional modularization above. Next, the electronic device provided by the embodiments of the present application is introduced from the perspective of hardware materialization, and the computing system of the electronic device is also introduced.

[0503] Based on the same principle as the method shown in the embodiments of the present disclosure, an electronic device is also provided in the embodiments of the present disclosure, which can include but is not limited to: a processor and a memory; the memory is used to store computer operation instructions; the processor is used to execute the image clustering method shown in the embodiments by calling the computer operation instructions. Compared with the prior art, the image clustering method in the present application can avoid the deviation of the entire clustering result caused by the deviation of the clustering center, and can effectively improve the accuracy of the clustering result.

[0504] In an optional embodiment, an electronic device is provided, as shown in Figure 48 The electronic device 1000 shown in Figure 48 The electronic device 1000 shown in the embodiments includes a processor 1001 and a memory 1003. The processor 1001 and the memory 1003 are connected, for example, through a bus 1002. Optionally, the electronic device 1000 can also include a transceiver 1004. It should be noted that in actual applications, the transceiver 1004 is not limited to one, and the structure of the electronic device 1000 does not constitute a limitation to the embodiments of the present application.

[0505] The processor 1001 can be a CPU (Central Processing Unit, central processor), a general-purpose processor, a DSP (Digital Signal Processor, data signal processor), an ASIC (Application Specific Integrated Circuit, application specific integrated circuit), an FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure of the present application. The processor 1001 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0506] The bus 1002 can include a path that transmits information between the above-described components. The bus 1002 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 1002 can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 48 Only one thick line is used in the middle, but it does not mean that there is only one bus or one type of bus.

[0507] The memory 1003 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0508] The memory 1003 is used to store application program codes for implementing the scheme of the present application, and is controlled to execute by the processor 1001. The processor 1001 is used to execute the application program codes stored in the memory 1003 to realize the content shown in the foregoing method embodiments.

[0509] The electronic device includes, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a car terminal (for example, a car navigation terminal), etc., and a fixed terminal such as a digital TV, a desktop computer, etc. Figure 48 The electronic device shown is only an example, and should not bring any limitation to the function and use range of the embodiments of the present disclosure.

[0510] The embodiment of the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is executed on a computer, the computer can execute the corresponding content in the foregoing method embodiment. Compared with the prior art, the image clustering method in the present application can avoid deviation of the entire clustering result caused by deviation of the clustering center, and can effectively improve the accuracy of the clustering result.

[0511] It should be understood that, although each step in the flowchart of the accompanying drawings is shown in sequence according to the indication of the arrow, these steps are not necessarily executed in sequence according to the indication of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other sequences. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0512] It should be noted that the computer readable medium of the present disclosure described above can be a computer readable signal medium or a computer readable storage medium or any combination of the above two. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component. In the present disclosure, the computer readable signal medium can include a data signal propagating in a baseband or as a carrier wave in a carrier, which carries computer readable program code. Such a propagating data signal can take various forms, including but not limited to electromagnetic signals, optical signals or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or component. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to: wire, cable, RF (radio frequency), etc., or any suitable combination of the above.

[0513] The computer readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device and be mounted in the electronic device.

[0514] The computer readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the embodiments described above.

[0515] Computer program code for carrying out operations of the present disclosure can be written in one or more programming languages or combinations of languages including object oriented programming languages such as Java, Smalltalk, C++ as well as conventional procedural programming languages such as "C" or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0516] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a part of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that noted in the figures. For example, two blocks noted in succession can in fact be executed substantially concurrently or in the opposite order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0517] The modules involved in the embodiments of the present disclosure can be implemented in the form of software or in the form of hardware. In some cases, the name of a module does not constitute a limitation on the module itself, for example, a first clustering module can also be described as a "module for clustering a target region".

[0518] The above description is merely exemplary of the disclosure and the application of the principles thereof and it is not intended to limit the scope of the disclosure to the specific forms set forth. The disclosure is susceptible to numerous modifications and alternative forms of implementation, all of which are intended to be within the scope of the disclosure. Indeed, the specific forms set forth are only exemplary embodiments of the disclosure and other embodiments are possible. The scope of the disclosure is therefore not intended to be limited to the specific forms set forth and any modifications and / or improvements are intended to be within the scope of the disclosure.

Claims

1. A method performed by an electronic device, the method comprising: The method comprises: acquiring a plurality of images for clustering; acquiring a plurality of target regions, wherein each target region corresponds to one of the plurality of images; acquiring a plurality of feature vectors corresponding to the plurality of target regions; acquiring a plurality of center nodes corresponding to the plurality of feature vectors; acquiring neighbor nodes, wherein each neighbor node is associated with one of the plurality of center nodes; constructing a subgraph based on the plurality of center nodes and the neighbor nodes; determining a connection probability between a center node and a neighbor node of the center node in the subgraph based on a graph convolutional network; for each target region to be clustered, acquiring a same-class neighbor node set corresponding to the target region according to the connection probability between the corresponding center node and the neighbor node; for each neighbor node in the same-class neighbor node set, adjusting the connection probability between the neighbor node and the center node according to the connection probability between the neighbor node and each neighbor node in the same-class neighbor node set; and clustering the target region based on the adjusted connection probability to determine whether the neighbor node is a same-class neighbor node; clustering each target region to be clustered based on the same-class neighbor nodes corresponding to the target region, to obtain a clustering result.

2. The method of claim 1, wherein, The method further comprises: using an inter-layer clustering module to maintain the difference between the noise node and other node features. The method further comprises: selecting a first-order neighbor node according to the cosine distance with each node as the center node; 3. The method of claim 2, wherein, selecting a second-order neighbor node of the first-order neighbor node as the neighbor node according to the cosine distance; and selecting K neighbor nodes for each node to construct a subgraph corresponding to the node. The method further comprises:

4. The method of claim 1, wherein, determining the connection probability between the center node and the first-order neighbor node of the center node according to a softmax function. The method further comprises:

5. The method of claim 3, wherein, repeating the step of constructing a subgraph corresponding to the center node with each node as the center node until the subgraph of each node is established. The method further comprises: acquiring node embeddings of the subgraph according to at least one layer of graph convolutional network; obtaining the connection probability between the center node and the neighbor node of the center node in the subgraph based on the node embeddings.

6. The method of claim 5, wherein, The method further comprises: acquiring input features corresponding to the subgraph; 7. The method of claim 5, wherein, ​ ​ 8. The method according to any one of claims 1 to 7, characterized in that, ​ ​ ​ 9. The method of claim 8, wherein, ​ ​ inputting the input features into the at least one layer of graph convolutional network for feature extraction, and taking the output result of the last layer of graph convolutional network as the node embedding.

10. The method of claim 9, wherein, wherein, for at least one layer of graph convolutional network other than the first layer of graph convolutional network, taking the output result of the previous layer of graph convolutional network as the input of the at least one layer of graph convolutional network; or taking the output result of the previous layer of graph convolutional network and the output result of at least one layer of graph convolutional network before the previous layer of graph convolutional network as the input of the at least one layer of graph convolutional network.

11. The method of claim 9, wherein, wherein, the clustering layer is arranged between at least two adjacent layers of graph convolutional network, and the clustering layer clusters the feature vectors of each node contained in the output result of the previous layer of graph convolutional network, and inputs the clustering result into the next layer of graph convolutional network.

12. The method according to any one of claims 1 to 7, characterized in that, for each target region to be clustered, a set of homogenous neighbor nodes is obtained according to the connection probability between the corresponding center node and the neighbor nodes, including: for each target region to be clustered, the neighbor nodes with the connection probability not less than a preset threshold value are determined, and the determined neighbor nodes are grouped into a set of homogenous neighbor nodes.

13. The method according to any one of claims 1 to 7, characterized in that, for each neighbor node in the set of homogenous neighbor nodes, the connection probability between the neighbor node and the center node is adjusted according to the connection probability between the neighbor node and each neighbor node in the set of homogenous neighbor nodes, including: for each neighbor node in the set of homogenous neighbor nodes, the average connection probability between the neighbor node and each neighbor node in the set of homogenous neighbor nodes is determined, and the average connection probability is taken as the connection probability between the neighbor node and the center node.

14. The method according to any one of claims 1 to 7, characterized in that, each target region to be clustered is clustered according to the corresponding set of homogenous neighbor nodes of each target region to be clustered, including: the connection probability between the center nodes corresponding to each target region to be clustered is determined according to the corresponding set of homogenous neighbor nodes of each target region to be clustered; each target region to be clustered is clustered based on the connection probability between the center nodes.

15. The method according to any one of claims 1 to 7, characterized in that, Further comprising: receiving a keyword input by a user for image search; determining a category associated with the keyword based on the clustering result; searching for images meeting the keyword in images of the determined category.

16. The method according to any one of claims 1 to 7, characterized in that, Further comprising: obtaining a plurality of labels to be labeled; labeling the labels on the target regions of the images of each category obtained by clustering based on the clustering result.

17. The method according to any one of claims 1 to 7, characterized in that, Further comprising: determining the clustering centers of each category obtained by clustering respectively; if the similarity of two clustering centers is not less than a set value, merging the categories corresponding to the two clustering centers into the same category to update the clustering result.

18. The method of claim 17, wherein, The determination of the clustering centers of each category obtained by clustering respectively includes: for a category obtained by clustering, taking the mean of the feature vectors contained in the category as the clustering center of the category.

19. The method according to any one of claims 1 to 7, characterized in that, Further comprising: obtaining a target region of a new image to be clustered; obtaining a preset number of target regions in at least one category obtained by clustering respectively; clustering the obtained target regions obtained by clustering and the new target region to be clustered based on the graph convolutional network to obtain a new clustering result; Based on the new clustering result and the category corresponding to the clustered target region, a category of a new target region to be clustered is determined.

20. An electronic device, comprising: It includes: one or more processors; a memory; one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to execute the method performed by the electronic device according to any one of claims 1 to 19.

21. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by the processor to realize the method performed by the electronic device according to any one of claims 1 to 19.

Images