Data processing method, model training method, electronic equipment, medium and product

Abstract

The invention provides a data processing method, a model training method, electronic equipment, a medium and a product, and the data processing method comprises the steps: carrying out the feature extraction of a to-be-processed image through employing a plurality of feature extraction modes, and obtaining a plurality of image features; performing initial clustering on the to-be-processed image based on part of the image features in the multiple image features to obtain a first clustering category; according to the first clustering category and the multiple image features, obtaining a second clustering category to which the to-be-processed image belongs and a clustering graph of the to-be-processed image; determining the probability that the to-be-processed image belongs to the second clustering category based on the second clustering category and the clustering graph; and determining a clustering result of the to-be-processed image based on the probability. According to the embodiment of the invention, the accuracy of image clustering can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a data processing method, a model training method, an electronic device, a medium and a product. Background Art

[0002] Supervised learning requires sample labeling, but sample labeling is costly, difficult, and has data security issues. Self-supervised learning can avoid these problems. Self-supervised learning includes self-supervised contrastive learning, which extracts discriminative representations from unlabeled data and enhances them. The same sample pairs with different enhancement methods are regarded as positive samples, and different sample pairs are regarded as negative samples. Through instance-level comparison, the distance between positive samples is minimized, and the distance between negative samples is increased, so that discriminative representation information can be learned from unlabeled data.

[0003] Contrastive Clustering (CC) introduces self-supervised learning based on deep clustering to learn representations and cluster assignments from unlabeled data. However, CC cannot learn global and local information in complex visual data sets, so the application of image clustering in the visual field is limited. Graph Contrastive Clustering (GCC) and the Nearest Neighbor Matching Neural Network Model (NNM) are representation clustering methods based on adjacent samples. They can only learn the compactness within the category, but cannot learn the differences and complex associations between categories and subcategories. Prototype-based representation clustering (such as deep clustering ProPos) can only use prototypes for learning, ignoring the deviations within the category and the complex associations between data. Therefore, the current clustering methods each have different problems and cannot improve the accuracy of clustering. Summary of the invention

[0004] The present disclosure provides a data processing method, a model training method, an electronic device, a medium and a product to improve the accuracy of clustering results.

[0005] In a first aspect, the present disclosure provides a data processing method, comprising:

[0006] Use multiple feature extraction methods to extract features from the image to be processed to obtain multiple image features;

[0007] Performing initial clustering on the image to be processed based on some of the multiple image features to obtain a first clustering category;

[0008] According to the first clustering category and the multiple image features, obtaining a second clustering category to which the image to be processed belongs and a clustering graph of the image to be processed;

[0009] Based on the second cluster category and the cluster graph, determining the probability that the image to be processed belongs to the second cluster category;

[0010] A clustering result of the image to be processed is determined based on the probability.

[0011] In a second aspect, the present disclosure provides a model training method, the method comprising:

[0012] Using image samples to train the clustering model to be trained;

[0013] Among them, the model training includes: extracting features from the image samples to obtain multiple image sample features; performing initial clustering processing on the image samples based on the multiple image sample features to obtain a first cluster category; determining the second cluster category to which the image sample belongs and the cluster graph of the image sample based on the first cluster category and the multiple image sample features; determining the probability that the image sample belongs to the second cluster category based on the second cluster category and the cluster graph; determining a total loss value based on the multiple image sample features, the first cluster category, and the probability that the image sample belongs to the second cluster category; and adjusting model parameters based on the loss value.

[0014] In a third aspect, the present disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and one or more of the computer programs are executed by the at least one processor so that the at least one processor can execute the above-mentioned data processing method and clustering model training method.

[0015] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned data processing method and clustering model training method when executed by a processor.

[0016] In a fifth aspect, the present disclosure provides a computer program product, characterized in that it includes a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned data processing method and model training method.

[0017] The data processing method provided by the embodiment of the present disclosure utilizes a plurality of feature extraction methods to extract features of the image to be processed, and can obtain features of the image to be processed from different angles. The image to be processed is initially clustered based on some image features among the plurality of image features to obtain a first clustering category (local feature). On the basis of the first clustering category, the second clustering category to which the image to be processed belongs and a clustering graph (global feature) of the image to be processed are obtained according to the first clustering category and the plurality of image features. The accuracy of the second clustering category to which the image to be processed belongs can be improved. Moreover, the clustering graph of the image to be processed is obtained on the basis of the plurality of image features and the first clustering category, so that the clustering graph is more accurate, and the probability that the image to be processed belongs to the second clustering category determined on the basis of the clustering graph is more accurate, thereby making the clustering result of the image to be processed more accurate.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings, in which:

[0020] Figure 1 A flowchart of a data processing method provided by an embodiment of the present disclosure;

[0021] Figure 2 A flowchart of a model training method provided in an embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of a clustering model provided in an embodiment of the present disclosure;

[0023] Figure 4 A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0025] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0026] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0027] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0028] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0029] Contrastive Clustering (CC) introduces self-supervised learning based on deep clustering to learn representation and cluster assignment from unlabeled data. However, CC cannot learn global and local information in complex visual datasets, so the application of image clustering in the visual field is limited.

[0030] Graph Contrastive Clustering (GCC) and Neural Network Model (NNM) are representational clustering methods based on adjacent samples. They can only learn the compactness within a category, but cannot learn the differences and complex associations between categories and subcategories.

[0031] Prototype-based representation clustering (such as deep clustering ProPos) can only use prototypes for learning, ignoring the deviations within the category and the complex connections between data.

[0032] Large-Scale Learnable Graph Convolutional Networks (L-GCN) uses a supervised learning clustering method and requires labels for sample data.

[0033] Therefore, the clustering accuracy of the related clustering methods is low, and each has other problems.

[0034] The data processing method according to the embodiment of the present disclosure can be executed by an electronic device such as a terminal device or a server, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a cloud server capable of cloud computing. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory.

[0035] In a first aspect, an embodiment of the present disclosure provides a data processing method.

[0036] Figure 1 A flowchart of a data processing method provided by an embodiment of the present disclosure. Figure 1 As shown, the data processing method includes:

[0037] Step S101, extracting features from the image to be processed using a variety of feature extraction methods to obtain a variety of image features.

[0038] The image to be processed may include multiple images, and feature extraction is performed on the multiple images to obtain multiple image features.

[0039] The disclosed embodiment can utilize multiple trained feature extraction models for feature extraction. For example, the trained feature extraction models can be student models and teacher models with the same structure. The extracted features include but are not limited to color features, shape features, texture features, and spatial relationship features.

[0040] Step S102: performing initial clustering on the image to be processed based on some of the multiple image features to obtain a first cluster category.

[0041] Part of the image features are extracted from the multiple image features, and initial clustering is performed based on the part of the image features, similar image features are clustered together, and the first cluster category is obtained. The number of the first cluster categories is related to the similarity of the image features. The more the number of the first cluster categories, the worse the similarity of the image features and the more dispersed the image features.

[0042] For example, a k-means clustering algorithm is used to cluster some image features to obtain a first clustering category, which may include multiple clustering categories, such as the first clustering category including but not limited to people, animals, flowers, grass, etc.

[0043] Step S103: obtaining the second cluster category to which the image to be processed belongs and a cluster diagram of the image to be processed according to the first cluster category and the plurality of image features.

[0044] Clustering is performed again based on the first clustering category using all the image features to obtain the second clustering category. Since the second clustering category is based on the first clustering category and is obtained based on more image features, the second clustering category is more accurate.

[0045] In some embodiments, obtaining a second cluster category to which the image to be processed belongs according to the first cluster category and multiple image features includes: clustering the multiple image features based on the first cluster category to obtain the second cluster category to which the image to be processed belongs.

[0046] The image features are all extracted from the image to be processed, and based on the first cluster category, multiple image features are clustered again. For example, using the k-means clustering algorithm, the similarity between each image feature and the first cluster category is calculated respectively, and the cluster category with the highest similarity is obtained, and the image feature is divided into the cluster category with the highest similarity. For any image feature, the similarity between the image feature and each cluster category in the first cluster category is calculated, and the cluster category with the highest similarity with the image feature is obtained. The image feature is then divided into the cluster category with the highest similarity, and finally the second cluster category to which the image to be processed belongs is obtained.

[0047] In some embodiments, a clustering graph of the image to be processed is obtained based on the first clustering category and multiple image features, including: calculating the similarity between each image feature and the first clustering category to generate an adjacency matrix; and generating a clustering graph of the image to be processed based on the first clustering category and the adjacency matrix.

[0048] Similarity refers to the degree of proximity between the image feature and the first cluster category, including but not limited to Euclidean distance and cosine similarity. In some embodiments, the similarity between each cluster category in the first cluster category and the image feature can be calculated using a K-Nearest Neighbor (KNN) algorithm or a proximity algorithm, and then an adjacency matrix is ​​generated based on the similarity between the image feature and each cluster category.

[0049] In some embodiments, the adjacency matrix is ​​calculated by formula (1), formula (2), formula (3), formula (4) and formula (5).

[0050]

[0051] Among them, K m Indicates a total of K m cluster categories, i=1,2,…,K m ,j=1,2,…,K m , the adjacency matrix A is K m Row K m The matrix of columns, a ij represents the weight connecting the i-th image feature and the j-th cluster category, h i represents the i-th image feature, c j represents the jth cluster category, Dist(h i ,c j ) represents the image feature h i With cluster category c j The Euclidean distance between i ) represents the image feature h i The set of Euclidean distances between the nearest u cluster categories, u is a pre-set value, Represents the image feature h i With cluster category c j The Euclidean distance between them does not belong to the image feature h i The set of Euclidean distances between the nearest u cluster categories, at this time a of the adjacency matrix A ij =0, Dist(h i ,c j )∈KNN(h i ) represents the image feature h i With cluster category c j The Euclidean distance between them does not belong to the image feature h iThe set of Euclidean distances between the nearest u cluster categories, at this time the adjacency matrix A σ represents the preset variance, D represents the degree matrix, diag() represents the diagonal matrix, d i Represents the image feature h i The degree, Represents the adjacency matrix.

[0052] After the adjacency matrix is ​​generated based on the similarity between the cluster categories and the image features, any cluster center in the first cluster category is used as a vertex of the cluster graph, and the adjacency matrix is ​​used as a vector to obtain the cluster graph.

[0053] Step S104: determining the probability that the image to be processed belongs to the second cluster category based on the second cluster category and the cluster graph.

[0054] In some embodiments, the image features of the image to be processed can be encoded through a Visual Geometry Group (VGG) network model, and then the encoded features can be converted into numerical vectors through a convolutional neural network to obtain an image feature vector, and then the image feature vector is used to determine the position of the image to be processed in the clustering diagram, and then based on the distance between the position of the image to be processed in the clustering diagram and each clustering category in the second clustering category, the probability that the image to be processed belongs to the second clustering category is determined.

[0055] Step S105: determining the clustering result of the image to be processed based on probability.

[0056] In some embodiments, the second clustering category with the greatest probability is determined as the clustering result of the image to be processed.

[0057] The data processing method provided by the embodiment of the present disclosure utilizes a variety of feature extraction methods to extract features from the image to be processed, and can obtain features of the image to be processed from different angles. The image to be processed is initially clustered based on some image features among the multiple image features to obtain a first clustering category (local feature). On the basis of the first clustering category, the second clustering category to which the image to be processed belongs and a clustering graph of the image to be processed (global feature) are obtained according to the first clustering category and the multiple image features. The accuracy of the second clustering category to which the image to be processed belongs can be improved, and the clustering graph of the image to be processed is obtained on the basis of the multiple image features and the first clustering category, so that the clustering graph is more accurate, and the probability that the image to be processed belongs to the second clustering category determined on the basis of the clustering graph is more accurate, thereby making the clustering result of the image to be processed more accurate.

[0058] In a second aspect, an embodiment of the present disclosure provides a model training method.

[0059] Figure 2A flow chart of a model training method provided in an embodiment of the present disclosure. Figure 2 As shown, the model training method includes:

[0060] Step S200: Perform model training on the clustering model to be trained using image samples.

[0061] The image samples are image samples extracted from the image training set. When training the model, the image samples in the image training set can be divided into multiple image groups, and the model can be trained batch by batch using the image groups.

[0062] Among them, model training includes:

[0063] Step S201, extracting features from image samples to obtain multiple image sample features.

[0064] The embodiments of the present disclosure may use multiple trained feature extraction models to perform feature extraction, such as using trained student models and teacher models with the same structure to perform feature extraction.

[0065] Step S202: performing initial clustering processing on the image samples based on multiple image sample features to obtain a first cluster category.

[0066] The disclosed embodiment may utilize a k-means clustering algorithm to cluster features of some image samples to obtain a first clustering category, which may include multiple clustering categories.

[0067] Step S203: determining the second cluster category to which the image sample belongs and a cluster diagram of the image sample based on the first cluster category and multiple image sample features.

[0068] Using the k-means clustering algorithm, the similarity between each image sample feature and the first cluster category is calculated to obtain the cluster category with the highest similarity, and the image sample feature is divided into the cluster category with the highest similarity. For any image sample feature, the similarity between the image sample feature and each cluster category in the first cluster category is calculated to obtain the cluster category with the highest similarity to the image sample feature. The image sample feature is then divided into the cluster category with the highest similarity, and finally the second cluster category to which the image sample belongs is obtained.

[0069] After the adjacency matrix is ​​generated based on the similarity between the cluster categories and the image sample features, any cluster center in the first cluster category is used as a vertex of the cluster graph, and the adjacency matrix is ​​used as a vector to obtain the cluster graph.

[0070] Step S204: determining the probability that the image sample belongs to the second cluster category based on the second cluster category and the cluster graph.

[0071] In some embodiments, determining the probability that the image sample belongs to the second cluster category based on the second cluster category and the cluster graph includes:

[0072] Step S2041, based on the second clustering categories and the clustering graph, determine the information features of the second clustering categories and the adjacency matrix between different second clustering categories; step S2042, determine the aggregation features based on the information features and the adjacency matrix, and obtain the clustering graph of the current round based on the aggregation features. Step S2043, compare the clustering graph of the current round with the clustering graph of the previous round. Figure 1 If the clustering diagram of the current round is inconsistent with the clustering diagram of the previous round, return to execute steps S2041 and S2042.

[0073] In step S2041, a K-Nearest Neighbor (KNN) algorithm or a proximity algorithm may be used to calculate the similarity between each cluster category in the first cluster category and the image sample feature, and then an adjacency matrix is ​​generated based on the similarity between the image sample feature and each cluster category.

[0074] In some embodiments, formula (6) and formula (7) may also be used to determine information features of the second clustering categories and an adjacency matrix between different second clustering categories.

[0075]

[0076]

[0077] Among them, H (l) is the feature output by the graph convolutional network layer at layer l, l is a positive integer, when l = 1, H (1) is the second cluster category, is the normalized adjacency matrix, σ is the activation function of the graph convolutional network layer, usually ReLU (Rectified Linear Unit), W (l) is the weight of the graph convolutional network layer at layer l, b (l) is the bias of the l-th graph convolutional network layer, and combine() is the aggregation function.

[0078] In step S2042, the information features of the second clustering category and the adjacency matrix between different second clustering categories are aggregated to obtain aggregated features, which fuse the information features of the second clustering category and the features of different second clustering categories connected to the second clustering category, that is, the aggregated features contain inter-class information between different clustering categories, so that the resulting clustering diagram is more accurate.

[0079] In step S2043, determine whether the clustering diagram of the current round is consistent with the clustering diagram of the previous round. If not, continue to execute steps S2041 and S2042; if they are consistent, that is, the clustering diagram of the current round is unchanged from the clustering diagram of the previous round, indicating that the clustering diagram has reached a convergence state, then determine the probability that the image sample belongs to the second clustering category based on the clustering features obtained in step S2042.

[0080] Step S205, determining a total loss value based on multiple image sample features, the first cluster category, and the probability that the image sample belongs to the second cluster category; and adjusting the model parameters based on the loss value.

[0081] In some embodiments, a total loss value is determined based on multiple image sample features, a first cluster category, and a probability that the image sample belongs to a second cluster category, including: determining a first loss value based on some image sample features among the multiple image sample features; determining a second loss value based on the first cluster category and the multiple image sample features; determining a third loss value based on the probability that the image sample belongs to the second cluster category; and determining a total loss value based on the first loss value, the second loss value, and the third loss value.

[0082] In some embodiments, a first loss function (contrastive loss function) is used to obtain a first loss value of an image sample based on multiple image sample features.

[0083] For example, the first loss value is obtained by formula (8), formula (9) and formula (10), and formula (8), formula (9) and formula (10) are as follows:

[0084]

[0085] in, represents the first loss value, represents the first loss function, g() represents the predictor in the clustering model, and z a represents the first image sample feature, z b represents the second image sample feature, represents the third image sample feature, represents the fourth image sample feature, ‖‖2 represents the L2 norm, represents the square of the L2 norm, and <> represents the dot product of the vectors.

[0086] In some embodiments, a second loss function (prototype loss function) is used to obtain a second loss value based on the clustering category, the first image sample feature, the second image sample feature, the third image sample feature, and the fourth image sample feature.

[0087] For example, the second loss value is calculated by formula (11), formula (12) and formula (13).

[0088]

[0089]

[0090] in, represents the second loss value, represents the second loss function, Represents the fourth image sample feature After clustering, K m The set of the first prototype centers corresponding to the prototype belonging categories, Represents the features of the third image sample After clustering, K m The set of second prototype centers corresponding to the prototype belonging categories, j=1,2,…,K m , Represents the fourth image sample feature A collection of represents the set of features of the third image samples, represents the j first prototype centers, represents the jth second prototype center, Represents the feature of the fourth image sample of the i-th Represents the hth third image sample feature Represents the feature of the fourth image sample of the i-th The first prototype center corresponding to the prototype belonging category, Represents the hth third image sample feature The second prototype center corresponding to the prototype belonging category, τ c represents a hyperparameter.

[0091] In some embodiments, the third loss value is calculated by formula (14), formula (15) and formula (16):

[0092]

[0093]

[0094] in, represents the third loss value, represents the second loss function, Represents the first prototype center set After clustering, the set of third prototype centers corresponding to the K second cluster categories is Represents the second prototype center set After clustering, the set of fourth prototype centers corresponding to the K second cluster categories is j = 1, 2, ..., K, Indicates the fourth enhanced feature A collection of Indicates the third enhancement feature A collection of represents the jth third prototype center, represents the jth fourth prototype center, represents the fourth enhanced feature of the i-th represents the hth third enhanced feature represents the fourth enhanced feature of the i-th The third prototype center corresponding to the second cluster category, represents the hth third enhanced feature The fourth prototype center corresponding to the second cluster category, τ c represents a hyperparameter.

[0095] In some embodiments, the total loss value is calculated using formula (17).

[0096]

[0097] in, represents the total loss value, represents the first loss value, represents the second loss value, Represents the third loss value.

[0098] In some embodiments, based on the first loss value, the second loss value and the third loss value, before determining the total loss value, it also includes: calculating the distribution entropy based on the probability that the image sample belongs to the second cluster category; determining the clustering regularization loss value based on the distribution entropy and the number of second cluster categories.

[0099] In some embodiments, the distribution entropy is calculated using formula (18), formula (19) and formula (20).

[0100]

[0101] Where Q represents the probability matrix, i = 1, 2, ..., N, j = 1, 2, ..., K, N is a positive integer, N represents a total of N second cluster categories, K is a positive integer, K represents a total of K second cluster categories, q ij represents the probability that the i-th image sample feature belongs to the j-th second cluster category, represents the distribution probability of the jth second cluster category, represents the distribution entropy.

[0102] In some embodiments, based on the distribution entropy and the number of second clustering categories, the clustering regularization loss is obtained using formula (21), and formula (21) is as follows:

[0103]

[0104] Among them, K means there are v second cluster categories, represents the distribution entropy, represents the clustering regularization loss.

[0105] By minimizing the clustering regularization loss The image sample features can be divided into different second cluster categories more evenly, avoiding the majority of image sample features being divided into a few second cluster categories, thereby causing model degradation.

[0106] Based on the first loss value, the second loss value and the third loss value, a total loss value is determined, including: obtaining the weight of the second loss value and the weight of the third loss value; determining the total loss value based on the first loss value, the weight of the first loss value, the second loss value, the weight of the third loss value, the third loss value and the clustering regularization loss value.

[0107] In some embodiments, based on the first loss value Second loss value weight, second loss value Third loss value weight, third loss value and clustering regularization loss, and the total loss is obtained using formula (22), which is as follows:

[0108]

[0109] in, represents the total loss, represents the first loss value, represents the second loss value weight, represents the second loss value, λ global represents the third loss value weight, represents the third loss value, represents the clustering regularization loss.

[0110] In some embodiments, before training the clustering model, the image samples are enhanced, and then the model training is performed based on the enhanced image samples. The image enhancement methods include but are not limited to cropping, rotating, flipping, scaling, translating, and brightness adjustment.

[0111] In some embodiments, multiple enhancement methods are used to enhance image samples, which can enrich the information content of the image samples and increase the robustness of the model.

[0112] Figure 3 A schematic diagram of a clustering model provided by an embodiment of the present disclosure. Figure 3 The clustering model includes a feature extraction network 301, a clustering network 302, a graph convolutional neural network 303 and a determination module 304, wherein:

[0113] The feature extraction network 301 is used to extract features from images using multiple feature extraction methods to obtain multiple image features.

[0114] In some embodiments, a feature extraction network is constructed based on a resnet (Residual Network, deep residual network) network.

[0115] In some embodiments, the clustering model further includes an image enhancement network 305 for performing image enhancement processing, wherein image enhancement methods include but are not limited to cropping, rotation, flipping, scaling, translation, and brightness adjustment.

[0116] In some embodiments, before the feature extraction network 301 extracts image features, the image enhancement network is used to enhance the image sample to obtain an enhanced image sample. During the enhancement process, two or more image enhancement methods can be selected, and then feature extraction is performed on the two or more enhanced image samples.

[0117] In some embodiments, the feature extraction network 301 includes a first encoder 311 and a second encoder 312, and the first encoder and the second encoder have the same structure. The first encoder may use a student model encoder, and the second encoder may use a teacher model encoder.

[0118] In some embodiments, the clustering model further includes a feature adjustment module 306 for adjusting the parameters of the first encoder 311, for example, the weight θ1 of the first encoder can be adjusted. Based on a stochastic gradient descent (SGD) optimizer or an adaptive moment estimation optimizer, the cosine annealing algorithm is used to adjust the learning rate, minimize the preset total loss function, and optimize the weight of the feature extraction encoder through a back propagation algorithm.

[0119] In some embodiments, the parameters of the feature extraction network 301 include the weight θ2 of the first encoder. Based on the exponential moving average (EMA) algorithm and the preset momentum m, the weight θ2 of the first encoder is updated using formula (23). Wherein, formula (8) is as follows:

[0120] θ2 ′=m*θ2+(1-m)*θ1 (23)

[0121] Among them, θ2 ′ is the weight of the first encoder after updating, θ2 is the weight of the first encoder before updating, m is the preset momentum, m is greater than 0 and less than 1, preferably, m can be set to 0.996.

[0122] In some embodiments, after multiple rounds of optimization of the weights of the feature extraction encoder, the weights of the first encoder are updated based on an exponential moving average algorithm and a preset momentum m, so that the weights of the feature extraction encoder are different from those of the first encoder, thereby increasing the difference between the image features extracted by the first encoder and the second encoder.

[0123] The clustering network 302 is used to cluster images based on image features, such as using part of the image features to initially cluster the images to obtain a first clustering category; obtaining a second clustering category to which the image belongs based on the first clustering category and multiple image features; and obtaining a clustering graph of the image based on the first clustering category and multiple image features.

[0124] In the disclosed embodiment, the clustering network 302 can extract the features of the structural data in the image, aggregate the features of each node with the features of the neighboring nodes based on the connection relationship between each node and the neighboring nodes (adjacent nodes) in the structural data, and update the features of each node in the structural data. The features of each node in the updated structural data fuse the self information of each node and the information of the neighboring nodes. Therefore, the second clustering category fuses the inter-class information between different clustering categories.

[0125] The clustering network 302 is clustered by steps S202 to S203, which will not be described in detail here.

[0126] The graph convolutional neural network 303 is used to determine the probability that the image sample belongs to the second cluster category based on the second cluster category and the cluster graph.

[0127] In some embodiments, the graph convolutional neural network 303 includes a graph convolution subnetwork and a graph clustering subnetwork. The graph convolution subnetwork includes multiple layers of graph convolutional network layers, each of which has corresponding weights, biases, and activation functions, and the graph clustering subnetwork includes a multilayer perceptron (MLP). The multilayer perceptron MLP can be one or more layers of fully connected layers. The multilayer perceptron MLP can be used for classification tasks, the input of the multilayer perceptron MLP is the image features to be classified, and the output of the multilayer perceptron MLP is the probability that each image feature belongs to a different category.

[0128] In some embodiments, the clustering model also includes a projection layer network. The projection layer network includes a feature linear layer, a relu (rectified linear units) activation function layer, a normalization layer, and a target linear layer. The projection layer network is used to project the features extracted by the feature extraction network from a high-dimensional feature space to a low-dimensional vector space. Among them, the feature linear layer and the target linear layer can both be fully connected layers. The feature linear layer is used to convert the features extracted by the feature extraction network into features in the hidden variable layer. The target linear layer is used to map the features output by the normalization layer to the output variables of the low-dimensional vector space.

[0129] The determination module 304 is used to determine the clustering result of the image to be processed based on probability.

[0130] A feature extraction network provided by an embodiment of the present disclosure includes a feature extraction encoder and a target feature encoder. An unlabeled complex image set includes image instances. An image enhancement transformation method is randomly used to enhance the image instances to obtain first enhanced data. Another image enhancement transformation method is randomly used to enhance the image instances to obtain second enhanced data. The feature extraction encoder is used to extract a first enhanced feature of the first enhanced data and a second enhanced feature of the second enhanced data. The target feature encoder is used to extract a third enhanced feature of the first enhanced data and a fourth enhanced feature of the second enhanced data. The first enhanced feature and the fourth enhanced feature are used as a feature pair, and the second enhanced feature and the third enhanced feature are used as another feature pair. When training the feature extraction network, the distance between two features in the feature pair is minimized, thereby improving the feature extraction effect of the feature extraction network in the clustering model.

[0131] In some embodiments, the image sample includes at least one of image data and video data.

[0132] The above clustering model training method can be used for clustering of image data and clustering of multiple frames of images in video data.

[0133] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0134] In addition, the present disclosure also provides electronic devices and computer-readable storage media, which can be used to implement any clustering model training method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0135] Figure 4A block diagram of an electronic device provided in an embodiment of the present disclosure.

[0136] Reference Figure 4 An embodiment of the present disclosure provides an electronic device, which includes: at least one processor 401; at least one memory 402, and one or more I / O interfaces 403, connected between the processor 401 and the memory 402; wherein the memory 402 stores one or more computer programs that can be executed by at least one processor 401, and the one or more computer programs are executed by at least one processor 401 so that the at least one processor 401 can execute the above-mentioned clustering model training method.

[0137] Each module in the above electronic device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0138] The present disclosure also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above clustering model training method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0139] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned clustering model training method.

[0140] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0141] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0142] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0143] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0144] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0145] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0146] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0147] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0148] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0149] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A data processing method, characterized in that: include: Use multiple feature extraction methods to extract features from the image to be processed to obtain multiple image features; Performing initial clustering on the image to be processed based on some of the multiple image features to obtain a first clustering category; According to the first clustering category and the multiple image features, obtaining a second clustering category to which the image to be processed belongs and a clustering graph of the image to be processed; Based on the second cluster category and the cluster graph, determining the probability that the image to be processed belongs to the second cluster category; A clustering result of the image to be processed is determined based on the probability.

2. The method according to claim 1, characterized in that: The obtaining, according to the first clustering category and the multiple image features, a second clustering category to which the image to be processed belongs includes: The plurality of image features are clustered based on the first clustering category to obtain a second clustering category to which the image to be processed belongs.

3. The method according to claim 1, characterized in that The step of obtaining a clustering diagram of the image to be processed according to the first clustering category and the plurality of image features includes: Calculating the similarity between each of the image features and the first clustering category to generate an adjacency matrix; A clustering graph of the image to be processed is generated based on the first clustering category and the adjacency matrix.

4. A model training method, characterized in that: The method comprises: Using image samples to train the clustering model to be trained; Among them, the model training includes: extracting features from the image samples to obtain multiple image sample features; performing initial clustering processing on the image samples based on the multiple image sample features to obtain a first cluster category; determining the second cluster category to which the image sample belongs and the cluster graph of the image sample based on the first cluster category and the multiple image sample features; determining the probability that the image sample belongs to the second cluster category based on the second cluster category and the cluster graph; determining a total loss value based on the multiple image sample features, the first cluster category, and the probability that the image sample belongs to the second cluster category; and adjusting model parameters based on the loss value.

5. The method according to claim 4, characterized in that The determining of the total loss value based on the multiple image sample features, the first cluster category, and the probability that the image sample belongs to the second cluster category includes: Determining a first loss value based on some image sample features among the plurality of image sample features; determining a second loss value based on the first cluster category and the plurality of image features; Determining a third loss value based on a probability that the image sample belongs to the second cluster category; The total loss value is determined based on the first loss value, the second loss value and the third loss value.

6. The method according to claim 4, characterized in that The determining, based on the second cluster category and the cluster graph, the probability that the image sample belongs to the second cluster category comprises: Based on the second clustering categories and the clustering graph, determining information features of the second clustering categories and an adjacency matrix between different second clustering categories; Determine an aggregation feature based on the information feature and the adjacency matrix, and obtain a clustering graph of the current round based on the aggregation feature; When the clustering diagram of the current round is consistent with the clustering diagram of the previous round, determining the probability that the image sample belongs to the second cluster category based on the aggregated features; When the clustering diagram of the current round is inconsistent with the clustering diagram of the previous round, return to execute the process of determining the information features of the second clustering categories and the adjacency matrix between different second clustering categories based on the second clustering categories and the clustering diagram; and determine the aggregation features based on the information features and the adjacency matrix, and update the clustering diagram based on the aggregation features.

7. The method according to claim 5, characterized in that Before determining the total loss value based on the first loss value, the second loss value and the third loss value, the method further includes: Calculating distribution entropy based on the probability that the image sample belongs to the second cluster category; Determining a clustering regularization loss value based on the distribution entropy and the number of the second cluster categories; The determining the total loss value based on the first loss value, the second loss value and the third loss value comprises: Obtaining the second loss value weight and the third loss value weight; The total loss value is determined based on the first loss value, a weight of the first loss value, the second loss value, a weight of the third loss value, the third loss value, and the cluster regularization loss value.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor so that the at least one processor can execute the data processing method described in any one of claims 1-3 and the clustering model training method described in any one of claims 4-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When executed by a processor, the computer program implements the data processing method according to any one of claims 1 to 3 and the clustering model training method according to any one of claims 4 to 7.

10. A computer program product, characterized in that It includes a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the data processing method as described in any one of claims 1 to 3 and the clustering model training method as described in any one of claims 4 to 7.

Images