Semantic-based image clustering method and device, storage medium and electronic device

Abstract

The invention provides a semantic-based image clustering method and device, a storage medium and an electronic device.The method comprises the steps that an original image of a clustering task is determined; performing feature enhancement on the original image to obtain an enhanced image, and performing feature reconstruction on the original image to obtain a reconstructed image; calculating a first loss distance between the original image and the enhanced image, and calculating a second loss distance between the original image and the reconstructed image; inputting the first loss distance and the second loss distance into a pre-training network, and outputting a neighbor image set of the original image; and inputting the neighbor image set into a pre-trained target clustering network to obtain the category of the original image. According to the embodiment of the invention, the technical problem of low image clustering precision in the prior art is solved, and the accuracy of the image clustering result is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a semantic-based image clustering method and device, a storage medium, and an electronic device. Background Art

[0002] In the related technology, image clustering is the most basic unsupervised learning task in the field of machine learning. Its core idea is to classify similar data into the same category and dissimilar data into different categories based on similarity measurement. In recent years, with the development of big data and artificial intelligence, people have an increasing demand for analyzing and processing a large amount of unlabeled data, and have paid more and more attention to clustering. In the past few decades, classic methods such as K-means clustering (K-means), spectral clustering (SC), Gaussian Mixed Model (GMM) and non-negative matrix factorization clustering (NMF) have been developed to effectively solve the problem of data clustering in lower dimensions.

[0003] In the related art, traditional clustering methods all rely on manually extracted features. However, for high-dimensional, large-scale and complex data, traditional clustering methods face the following problems: First, in high-dimensional space, the distances between data points will tend to be equal, resulting in the failure of distance-based clustering methods; second, high-dimensional data causes a sharp increase in computational complexity; finally, high-dimensional data often has redundant features and noise, which seriously affects clustering performance. The current methods for solving high-dimensional data clustering are mainly: dimensionality reduction or feature transformation of high-dimensional data, mapping the original high-dimensional data into a low-dimensional feature space, and then clustering the obtained low-dimensional features using traditional methods. When clustering, the related art cannot accurately extract the semantic features of the image, resulting in the loss of key features during dimensionality reduction mapping, which in turn leads to inaccurate clustering.

[0004] With respect to the above-mentioned problems existing in the related technologies, no efficient and accurate solutions have been found yet. Summary of the invention

[0005] The present invention provides a semantic-based image clustering method and device, a storage medium, and an electronic device to solve technical problems in related technologies.

[0006] According to one embodiment of the present invention, a semantic-based image clustering method is provided, comprising: determining an original image of a clustering task; performing feature enhancement on the original image to obtain an enhanced image, and performing feature reconstruction on the original image to obtain a reconstructed image; calculating a first loss distance between the original image and the enhanced image, and calculating a second loss distance between the original image and the reconstructed image; inputting the first loss distance and the second loss distance into a pre-trained network, and outputting a set of neighbor images of the original image; and inputting the set of neighbor images into a pre-trained target clustering network to obtain the category to which the original image belongs.

[0007] Optionally, before inputting the first loss distance and the second loss distance into the pre-trained network, the method also includes: determining a sample image; performing feature enhancement on the sample image to obtain an enhanced sample image, and performing feature reconstruction on the sample image to obtain a reconstructed sample image; calculating a first loss distance between the sample image and the enhanced sample image, and calculating a second loss distance between the sample image and the reconstructed sample image; constructing a loss function using the first loss distance and the second loss distance; and training a pre-trained network based on the sample image and the loss function.

[0008] Optionally, performing feature reconstruction on the sample image to obtain a reconstructed sample image includes: determining an encoder for a feature extraction network, and a decoder corresponding to the encoder; using the encoder to extract image features of the sample image; and using the decoder to reconstruct the image features to obtain a reconstructed sample image.

[0009] Optionally, before inputting the neighbor image set into the pre-trained target clustering network, the method further includes: for each sample image in the sample image set, constructing a neighbor sample set corresponding to the sample image by a nearest neighbor algorithm; and training the target clustering network using the sample image and the neighbor sample set.

[0010] Optionally, using the sample image and the neighbor sample set to train a target clustering network includes: randomly selecting a neighbor image from the neighbor sample set; and using the sample image and the neighbor image to update an initial clustering network to obtain a target clustering network.

[0011] Optionally, the sample image and the neighbor image are used to update the initial clustering network to obtain the target clustering network, including: calculating the feature error between the sample image and the neighbor image; and updating the initial clustering network using the feature error as a loss function to obtain the target clustering network.

[0012] Optionally, the initial clustering network is updated using the feature error as a loss function to obtain a target clustering network, including: determining a clustering cluster M={1,...,c}; and updating the initial clustering network using the following loss function L to obtain a target clustering network:

[0014] Among them, the probability of sample X being assigned to cluster c is expressed as , Φ η (X) is the predicted probability of sample X, Φ η (k) is the predicted probability of the neighboring image, Φ η (X)∈[0,1] M , X represents the sample image, |D| represents the number of samples, k represents the neighboring image of the sample image, N x represents the neighbor sample set of the sample image, c represents the number of clusters, <,> represents the dot product operation, and δ is a hyperparameter.

[0015] According to another embodiment of the present invention, a semantic-based image clustering device is provided, including: a first determination module, used to determine an original image of a clustering task; a processing module, used to perform feature enhancement on the original image to obtain an enhanced image, and perform feature reconstruction on the original image to obtain a reconstructed image; a calculation module, used to calculate a first loss distance between the original image and the enhanced image, and to calculate a second loss distance between the original image and the reconstructed image; a mining module, used to input the first loss distance and the second loss distance into a pre-trained network, and output a set of neighbor images of the original image; and a clustering module, used to input the set of neighbor images into a pre-trained target clustering network to obtain the category to which the original image belongs.

[0016] Optionally, the device also includes: a second determination module, used to determine the sample image before the extraction module inputs the first loss distance and the second loss distance into the pre-trained network; a processing module, used to perform feature enhancement on the sample image to obtain an enhanced sample image, and to perform feature reconstruction on the sample image to obtain a reconstructed sample image; a calculation module, used to calculate the first loss distance between the sample image and the enhanced sample image, and to calculate the second loss distance between the sample image and the reconstructed sample image; a first construction module, used to construct a loss function using the first loss distance and the second loss distance; and a first training module, used to train the pre-trained network based on the sample image and the loss function.

[0017] Optionally, the processing module includes: a determination unit, used to determine an encoder of a feature extraction network, and a decoder corresponding to the encoder; an extraction unit, used to extract image features of the sample image using the encoder; and a reconstruction unit, used to reconstruct the image features using the decoder to obtain a reconstructed sample image.

[0018] Optionally, the device also includes: a second construction module, used to construct a neighbor sample set corresponding to each sample image in the sample image set by a nearest neighbor algorithm before the clustering module inputs the neighbor image set into the pre-trained target clustering network; and a second training module, used to use the sample image and the nearest neighbor sample set to train to obtain a target clustering network.

[0019] Optionally, the second training module includes: a selection unit, used to randomly select a neighbor image from the neighbor sample set; and an updating unit, used to update the initial clustering network using the sample image and the neighbor image to obtain a target clustering network.

[0020] Optionally, the updating unit includes: a calculation subunit, used to calculate the feature error between the sample image and the neighboring image; and an updating subunit, used to update the initial clustering network using the feature error as a loss function to obtain a target clustering network.

[0021] Optionally, the updating subunit is further used to: determine a clustering cluster M={1, ..., c}; and update the initial clustering network using the following loss function L to obtain a target clustering network:

[0023] Among them, the probability of sample X being assigned to cluster c is expressed as , Φ η (X) is the predicted probability of sample X, Φ η (k) is the predicted probability of the neighboring image, Φ η (X)∈[0,1] M , X represents the sample image, |D| represents the number of samples, k represents the neighboring image of the sample image, N x represents the neighbor sample set of the sample image, c represents the number of clusters, <,> represents the dot product operation, and δ is a hyperparameter.

[0024] According to another aspect of an embodiment of the present application, a storage medium is further provided, which includes a stored program, and the above steps are executed when the program is run.

[0025] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein: the memory is used to store computer programs; and the processor is used to execute the steps in the above method by running the program stored in the memory.

[0026] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps in the above method.

[0027] Beneficial effects of the present invention:

[0028] 1. Ability to model image features of the current task without human annotation of image categories, reducing the need for human annotation;

[0029] 2. By introducing a regularization loss and using reconstruction to increase the connection between representation and input, more non-shared clustering task-related information is introduced, and the image clustering results are more accurate;

[0030] 3. In the fine-tuning stage, the method of updating the clustering algorithm and the network simultaneously is abandoned. Instead, the similarity difference between image neighbors is used as the loss to update the clustering network to better utilize semantic features and obtain more accurate image clustering results. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0032] Figure 1 is a hardware structure block diagram of a car according to an embodiment of the present invention;

[0033] Figure 2 is a flow chart of a semantic-based image clustering method according to an embodiment of the present invention;

[0034] Figure 3 It is a schematic diagram of the overall model of an embodiment of the present invention;

[0035] Figure 4 is a schematic diagram of clustering accuracy of different nearest neighbor values ​​on a data set according to an embodiment of the present invention;

[0036] Figure 5 Schematic diagram of comparison between the solution of the embodiment of the present invention and the SCAN algorithm;

[0037] Figure 6 is a schematic diagram of a neighbor image in an embodiment of the present invention;

[0038] Figure 7 It is a structural block diagram of a semantic-based image clustering device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0040] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] Example 1

[0042] The method embodiment provided in the first embodiment of the present application can be executed in a car, a car controller, a server or a similar processing device. Taking running on a car as an example, Figure 1 1 is a hardware structure diagram of a car according to an embodiment of the present invention. Figure 1 As shown, the server may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the server may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0043] The memory 104 can be used to store server programs, for example, software programs and modules of application software, such as a server program corresponding to a remote control method for a car in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the server program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0044] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0045] In this embodiment, a semantic-based image clustering method is provided. Figure 2 is a flowchart of a semantic-based image clustering method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0046] Step S202, determining the original image of the clustering task;

[0047] The solution of this embodiment can be applied in the scenario of vehicle visual recognition and application, and the original image is an image to be classified and recognized, such as an original image captured by a camera.

[0048] Step S204, performing feature enhancement on the original image to obtain an enhanced image, and performing feature reconstruction on the original image to obtain a reconstructed image;

[0049] Step S206, calculating a first loss distance between the original image and the enhanced image, and calculating a second loss distance between the original image and the reconstructed image;

[0050] Optionally, the first loss distance and the second loss distance are semantic features of the original image, where the semantic features are features that can characterize key content of the original image, such as regional features or discriminative features of significant targets in the original image.

[0051] Step S206, inputting the first loss distance and the second loss distance into a pre-trained network, and outputting a set of neighboring images of the original image;

[0052] The neighbor image set of this embodiment is a set of neighbor images similar to the original image, and there is correlation in the environmental information of the original image and the neighbor images, such as the same background area, the same foreground area, the same posture of the target in the image, the same color distribution, and local and global images of each other.

[0053] Step S208: input the neighbor image set into a pre-trained target clustering network to obtain the category to which the original image belongs.

[0054] Through the above steps, the original image of the clustering task is determined; the features of the original image are enhanced to obtain an enhanced image, and the features of the original image are reconstructed to obtain a reconstructed image; the first loss distance between the original image and the enhanced image is calculated, and the second loss distance between the original image and the reconstructed image is calculated; the first loss distance and the second loss distance are input into a pre-trained network, and a neighbor image set of the original image is output; the neighbor image set is input into a pre-trained target clustering network to obtain the category to which the original image belongs, and the semantic features of the original image are extracted and the neighbor image set of the original image is obtained by mining, and the category to which the original image belongs is obtained by using the neighbor image set, and the characteristic that there is feature similarity between the original image and its neighbors is used to solve the technical problem of low image clustering accuracy in related technologies and improve the accuracy of image clustering results.

[0055] In one implementation of the present embodiment, before the first loss distance and the second loss distance are input into the pre-trained network, it also includes: determining a sample image; performing feature enhancement on the sample image to obtain an enhanced sample image, and performing feature reconstruction on the sample image to obtain a reconstructed sample image; calculating a first loss distance between the sample image and the enhanced sample image, and calculating a second loss distance between the sample image and the reconstructed sample image; constructing a loss function using the first loss distance and the second loss distance; and training based on the sample image and the loss function to obtain a pre-trained network.

[0056] In one example, reconstructing features of the sample image to obtain a reconstructed sample image includes: determining an encoder for a feature extraction network and a decoder corresponding to the encoder; extracting image features of the sample image using the encoder; and reconstructing the image features using the decoder to obtain a reconstructed sample image.

[0057] In the pre-training stage, contrastive learning usually uses two different enhancement strategies to pre-train the network to extract better image features. However, in practice, some pre-training tasks are based on specific image transformations, which causes the learned features to be covariant to the adopted transformations, i.e. different enhancement transformations of the same image will lead to different output predictions of the network. This makes the learned feature representations less suitable for semantic clustering. To overcome this problem, contrastive loss is used in the pre-training stage to minimize the distance between the features of the image itself and the features of their enhanced views.

[0058] Since the minimum sufficient representation obtained by contrastive learning usually cannot meet the clustering task in the fine-tuning stage, the reconstruction loss is introduced as a regularization term to retain more information in the original image, thereby obtaining a better feature representation. In this embodiment, the feature extraction network is used as an encoder and a corresponding decoder network is designed. The original image is passed through the encoder network to obtain features, and these features are reconstructed through the decoder network. The reconstructed data is compared with the original data for error calculation, thereby obtaining the reconstruction loss to obtain more features of the original image.

[0059] In an implementation scenario, the sample image data D = {X1, ..., X D}, build a pre-trained network Φ θ ;

[0060] In the pre-training phase, a contrastive loss is used to minimize the image X i Own features and their enhanced views T[X i ] is the distance of the feature. It is expressed as:

[0061] Use the feature extraction network as the encoder and design the corresponding decoder network H ω The original image is passed through the encoder network to obtain features, and these features are reconstructed through the decoder network. The error between the reconstructed data and the original data is calculated to obtain the reconstruction loss, so as to obtain more features of the original image. The loss calculation is as follows:

[0062] The network loss calculation expression for the final pre-training phase is as follows:

[0063]

[0064] Among them, λ is a hyperparameter used to adjust the relationship between contrast loss and reconstruction loss, X represents image data, i represents the image index in the entire image dataset, θ represents the network parameters of the pre-trained network, ω represents the network parameters of the decoder network, D represents the entire image dataset, dist() represents the Euclidean distance, and λ is a hyperparameter. The pre-trained network obtained by pre-training can extract semantic feature information from the image.

[0065] The applicant found that applying the K-Means algorithm directly on these features will lead to cluster degradation. When learning the decision boundary, a discriminative model may assign all its probabilities to the same cluster, resulting in one cluster dominating other clusters. In order to overcome this problem, this embodiment chooses a better strategy to update the network by using the principle of feature similarity between similar samples. Through the representation learning in the pre-training stage, a network is obtained. For each sample, the features obtained in the network are mined by the K-nearest neighbor algorithm to obtain the K nearest neighbors of the sample, and the adjacent sample set of each sample in the data set is obtained. In the semantic fine-tuning stage, the goal is to learn a clustering network, which includes a ResNet-18 network and a clustering head composed of several linear layers. The sample and a neighbor randomly selected from the neighbor matrix are sent to the network for training. The characteristic of feature similarity between the original image and its neighbor sample is used to calculate the error between the features of the two as the loss function to update the clustering network. Finally, the data is clustered into the category to which it belongs through the softmax function.

[0066] In one implementation of this embodiment, before the neighbor image set is input into the pre-trained target clustering network, it also includes: for each sample image in the sample image set, constructing a neighbor sample set corresponding to the sample image through a neighbor algorithm; and using the sample image and the neighbor sample set to train to obtain the target clustering network.

[0067] The process of training the target clustering network is to fine-tune the network based on the pre-trained network obtained through training.

[0068] In one example, using the sample image and the neighbor sample set to train a target clustering network includes: randomly selecting a neighbor image from the neighbor sample set; and updating an initial clustering network using the sample image and the neighbor image to obtain a target clustering network.

[0069] Optionally, the sample image and the neighbor image are used to update the initial clustering network to obtain the target clustering network, including: calculating the feature error between the sample image and the neighbor image; and updating the initial clustering network using the feature error as a loss function to obtain the target clustering network.

[0070] In the semantic fine-tuning stage, the goal is to learn a clustering network Φ η , which consists of a ResNet-18 network and a clustering head composed of several linear layers. The sample and a neighbor randomly selected from the neighbor matrix are sent to the network for training. The feature similarity between the original image and its neighbor samples is used to calculate the error between the features of the two as the loss function to Φ η to update.

[0071] In one example, the initial clustering network is updated using the feature error as a loss function to obtain a target clustering network, including: determining a cluster cluster M={1,...,c}; and updating the initial clustering network using the following loss function L to obtain a target clustering network: Among them, samples X are assigned to clusters

[0073] The probability of c is expressed as , Φ η (X) is the predicted probability of sample X, Φ η (k) is the predicted probability of the neighboring image, Φ η (X)∈[0,1] M , X represents the sample image, |D| represents the number of samples, k represents the neighboring image of the sample image, N x represents the neighbor sample set of the sample image, c represents the number of clusters, <,> represents the dot product operation, and δ is a hyperparameter.

[0074] In this embodiment, the network Φ η It terminates with a softmax function and learns Φ by minimizing the following objective function η The first term Φ η For a sample X and its neighboring samples N x Make consistent predictions. Note that the dot product will be maximum when the predictions are confident and assigned to the same cluster. To avoid Φ η Assigning all samples to one class includes an entropy term (the second term in the above formula) that evenly distributes the predicted probabilities among the clusters M.

[0075] The scheme of this embodiment provides a semantic feature optimization algorithm for image clustering, including: using contrast loss in the pre-training stage to minimize the distance between the features of the image itself and the features of their enhanced views. Since the minimum sufficient representation obtained by contrastive learning usually cannot meet the clustering task in the fine-tuning stage, reconstruction loss is introduced as a regularization term. The original image is passed through the encoder network to obtain features, and these features are reconstructed through the decoder network. The reconstructed data is calculated with the original data for error, thereby obtaining the reconstruction loss to obtain more meaningful features of the original image.

[0076] Extract semantic feature information from the image from the pre-trained network. Update the network based on the principle that there are feature similarities between samples of the same type. For each sample, the features obtained in the network are used to mine the K nearest neighbors of the sample through the K-nearest neighbor algorithm to obtain the set of neighboring samples for each sample in the data set. In the semantic fine-tuning stage, the clustering network contains a ResNet-18 network and a clustering head composed of several linear layers. The sample and a neighbor randomly selected from the neighbor matrix are sent to the network for training, and the clustering network is updated by calculating the error between the features of the two as the loss function. The data is clustered into the categories to which it belongs through the softmax function.

[0077] Figure 3 It is a schematic diagram of the overall model of an embodiment of the present invention, including a pre-trained network obtained in the pre-training stage, and a target clustering network obtained by training in the semantic fine-tuning stage. The reconstruction loss and the contrast loss are combined to pre-train the unsupervised representation learning model to produce a richer and more semantic feature representation. Prepare for subsequent semantic clustering. In the fine-tuning stage, the extracted prior knowledge is integrated into the new loss function, each image and its nearest neighbor are put into the network for classification, and the difference in feature similarity between the image and its nearest neighbor is used as the loss function to update the clustering network, thereby obtaining the final semantic clustering assignment result.

[0078] Figure 4 Figure 1 is a schematic diagram of the clustering accuracy of different neighbor values ​​on a data set according to an embodiment of the present invention. In order to better verify the relationship between the clustering accuracy of the model and the number of neighbor points, an experiment was conducted on the present invention. The results are shown in Figure 1. Figure 4 As shown. Experimental results show that the model is not sensitive to the choice of the number of neighbors, and the clustering accuracy remains stable even when K is increased to 50. This is very useful because it is not necessary to fine-tune the K value on new datasets. In fact, as the K value increases, the robustness of the model and the clustering accuracy will increase. It is worth noting that when the number of neighbors is 0, the sample is only calculated with its own enhanced view. The clustering performance decreases compared to K = 10, which shows that better feature representations can be learned by enforcing consistent predictions between a sample and its nearest neighbors.

[0079] The enhanced view of this embodiment is to cover, blur, crop, rotate, grayscale and disturb the original image. The main functions are: to improve the generalization ability and robustness of the model; to avoid overfitting, reduce the deviation and variance of the data, and make the model more stable; to improve the utilization and value of data.

[0080] Figure 5: is a schematic diagram comparing the scheme of an embodiment of the present invention and the SCAN algorithm, which shows the comparison of the accuracy of the clustering scheme of the present embodiment and the SCAN method when selecting neighbors after pre-training. It can be seen that the present method has achieved a great lead in the accuracy evaluation index of selecting 20 neighbors, especially in the STL10 training set. Compared with the SCAN algorithm, the accuracy of the first 20 neighbors of the sample of the present method is improved by 3.2%, which shows that the semantic features obtained by pre-training of the present method are more beneficial for neighbor selection, and the high-precision neighbors ensure the stability and accuracy of the neighbor similarity calculation in the fine-tuning stage.

[0081] Figure 6 is a schematic diagram of a neighbor image in an embodiment of the present invention. After the pre-training is completed, some images in the STL10 data set and their selected neighbors are visualized. The neighbor result visualization is as follows: Figure 6 As shown in the figure, it can be seen from the neighbor images that there is a correlation between the environmental information of the image and its neighbors. The more similar the environment and posture of the image, the higher the ranking of its neighbors, and the higher the accuracy of the same category. This shows that increasing the mutual information between the feature representation and the input is effective for neighbor selection and clustering tasks.

[0082] The solution of this embodiment designs a semantically optimized deep clustering algorithm to address the problem that the semantic features learned by the contrastive learning method in pre-training are insufficient and the previous deep clustering algorithms have failed to make good use of the semantic relationship between image features. The algorithm increases the information between the feature representation and the input by adding a reconstruction regularization term in the pre-training stage to introduce more non-shared feature information related to the clustering task. At the same time, in order to solve the problem that the direct effect of features on the clustering algorithm may cause cluster degradation, the present invention uses the nearest neighbor matching method to update the clustering network, abandoning the method of combining feature extraction with traditional clustering algorithms. Finally, the effectiveness of this method was demonstrated by experiments on multiple data sets.

[0083] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0084] Example 2

[0085] In this embodiment, a semantic-based image clustering device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0086] Figure 7 is a structural block diagram of a semantic-based image clustering device according to an embodiment of the present invention. Figure 7 As shown, the device comprises:

[0087] A first determination module 70, used to determine the original image of the clustering task;

[0088] The processing module 72 is used to perform feature enhancement on the original image to obtain an enhanced image, and to perform feature reconstruction on the original image to obtain a reconstructed image;

[0089] A calculation module 74, configured to calculate a first loss distance between the original image and the enhanced image, and to calculate a second loss distance between the original image and the reconstructed image;

[0090] A mining module 76, configured to input the first loss distance and the second loss distance into a pre-trained network, and output a set of neighboring images of the original image;

[0091] The clustering module 78 is used to input the neighbor image set into a pre-trained target clustering network to obtain the category to which the original image belongs.

[0092] Optionally, the device also includes: a second determination module, used to determine the sample image before the extraction module inputs the first loss distance and the second loss distance into the pre-trained network; a processing module, used to perform feature enhancement on the sample image to obtain an enhanced sample image, and to perform feature reconstruction on the sample image to obtain a reconstructed sample image; a calculation module, used to calculate the first loss distance between the sample image and the enhanced sample image, and to calculate the second loss distance between the sample image and the reconstructed sample image; a first construction module, used to construct a loss function using the first loss distance and the second loss distance; and a first training module, used to train the pre-trained network based on the sample image and the loss function.

[0093] Optionally, the processing module includes: a determination unit, used to determine an encoder of a feature extraction network, and a decoder corresponding to the encoder; an extraction unit, used to extract image features of the sample image using the encoder; and a reconstruction unit, used to reconstruct the image features using the decoder to obtain a reconstructed sample image.

[0094] Optionally, the device also includes: a second construction module, used to construct a neighbor sample set corresponding to each sample image in the sample image set by a nearest neighbor algorithm before the clustering module inputs the neighbor image set into the pre-trained target clustering network; and a second training module, used to use the sample image and the nearest neighbor sample set to train to obtain a target clustering network.

[0095] Optionally, the second training module includes: a selection unit, used to randomly select a neighbor image from the neighbor sample set; and an updating unit, used to update the initial clustering network using the sample image and the neighbor image to obtain a target clustering network.

[0096] Optionally, the updating unit includes: a calculation subunit, used to calculate the feature error between the sample image and the neighboring image; and an updating subunit, used to update the initial clustering network using the feature error as a loss function to obtain a target clustering network.

[0097] Optionally, the updating subunit is further used to: determine a clustering cluster M={1, ..., c}; and update the initial clustering network using the following loss function L to obtain a target clustering network:

[0099] Among them, the probability of sample X being assigned to cluster c is expressed as , Φ η (X) is the predicted probability of sample X, Φ η (k) is the predicted probability of the neighboring image, Φ η (X)∈[0,1] M , X represents the sample image, |D| represents the number of samples, k represents the neighboring image of the sample image, N x represents the neighbor sample set of the sample image, c represents the number of clusters, <,> represents the dot product operation, and δ is a hyperparameter.

[0100] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0101] Example 3

[0102] An embodiment of the present invention further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0103] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0104] S1, the original image for clustering task;

[0105] S2, performing feature enhancement on the original image to obtain an enhanced image, and performing feature reconstruction on the original image to obtain a reconstructed image;

[0106] S3, calculating a first loss distance between the original image and the enhanced image, and calculating a second loss distance between the original image and the reconstructed image;

[0107] S4, inputting the first loss distance and the second loss distance into a pre-trained network, and outputting a set of neighboring images of the original image;

[0108] S5, inputting the neighbor image set into a pre-trained target clustering network to obtain the category to which the original image belongs.

[0109] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0110] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0111] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0112] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0113] S1, the original image for clustering task;

[0114] S2, performing feature enhancement on the original image to obtain an enhanced image, and performing feature reconstruction on the original image to obtain a reconstructed image;

[0115] S3, calculating a first loss distance between the original image and the enhanced image, and calculating a second loss distance between the original image and the reconstructed image;

[0116] S4, inputting the first loss distance and the second loss distance into a pre-trained network, and outputting a set of neighboring images of the original image;

[0117] S5, inputting the neighbor image set into a pre-trained target clustering network to obtain the category to which the original image belongs.

[0118] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0119] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0121] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0122] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A semantic-based image clustering method, characterized in that: include: Determine the original image for the clustering task; Performing feature enhancement on the original image to obtain an enhanced image, and performing feature reconstruction on the original image to obtain a reconstructed image; Calculating a first loss distance between the original image and the enhanced image, and calculating a second loss distance between the original image and the reconstructed image; Inputting the first loss distance and the second loss distance into a pre-trained network, and outputting a set of neighboring images of the original image; The neighbor image set is input into a pre-trained target clustering network to obtain the category to which the original image belongs.

2. The method according to claim 1, characterized in that Before inputting the first loss distance and the second loss distance into a pre-trained network, the method further includes: determining a sample image; Performing feature enhancement on the sample image to obtain an enhanced sample image, and performing feature reconstruction on the sample image to obtain a reconstructed sample image; Calculating a first loss distance between the sample image and the enhanced sample image, and calculating a second loss distance between the sample image and the reconstructed sample image; Constructing a loss function using the first loss distance and the second loss distance; A pre-trained network is obtained based on the sample image and the loss function.

3. The method according to claim 2, characterized in that Performing feature reconstruction on the sample image to obtain a reconstructed sample image includes: Determine an encoder of a feature extraction network and a decoder corresponding to the encoder; extracting image features of the sample image using the encoder; The decoder is used to reconstruct the image features to obtain a reconstructed sample image.

4. The method according to claim 1, characterized in that Before inputting the neighbor image set into the pre-trained target clustering network, the method further includes: For each sample image in the sample image set, a nearest neighbor sample set corresponding to the sample image is constructed by a nearest neighbor algorithm; The target clustering network is obtained by training the sample image and the nearest neighbor sample set.

5. The method according to claim 4, characterized in that The target clustering network is obtained by training the sample image and the neighbor sample set, including: Randomly select a neighbor image from the neighbor sample set; The sample image and the neighbor image are used to update the initial clustering network to obtain a target clustering network.

6. The method according to claim 5, characterized in that The initial clustering network is updated by using the sample image and the neighbor image to obtain a target clustering network including: Calculating a feature error between the sample image and the neighboring image; The initial clustering network is updated using the feature error as a loss function to obtain a target clustering network.

7. The method according to claim 6, characterized in that The initial clustering network is updated using the feature error as the loss function to obtain the target clustering network, including: Determine the clustering cluster M = {1, ..., c}; The following loss function L is used to update the initial clustering network to obtain the target clustering network: Among them, the probability of sample X being assigned to cluster c is expressed as Φ η (X), Φ η (X) is the predicted probability of sample X, Φ η (k) is the predicted probability of the neighboring image, Φ η (X)∈[0,1] M , X represents the sample image, |D| represents the number of samples, k represents the neighboring image of the sample image, N x represents the neighbor sample set of the sample image, c represents the number of clusters, <,> represents the dot product operation, and δ is a hyperparameter.

8. A semantic-based image clustering device, characterized in that: include: A first determination module, used to determine the original image of the clustering task; A processing module, used for performing feature enhancement on the original image to obtain an enhanced image, and performing feature reconstruction on the original image to obtain a reconstructed image; A calculation module, used for calculating a first loss distance between the original image and the enhanced image, and calculating a second loss distance between the original image and the reconstructed image; A mining module, configured to input the first loss distance and the second loss distance into a pre-trained network, and output a set of neighboring images of the original image; The clustering module is used to input the neighbor image set into a pre-trained target clustering network to obtain the category to which the original image belongs.

9. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 7.

Images