Image clustering method, device and computer readable storage medium

Abstract

The application provides an image clustering method, device and computer readable storage medium in the field of artificial intelligence; the method comprises: obtaining semantic information of each image to be processed in a set of images to be processed, and performing semantic cluster division on the set of images to be processed based on the semantic information of each image to be processed, to obtain n semantic sub-clusters; performing image clustering in each semantic sub-cluster of the n semantic sub-clusters, to obtain C i corresponding to each semantic sub-cluster, and further obtaining a set of clustering centers corresponding to the n semantic sub-clusters; performing clustering on the set of images to be processed based on the set of clustering centers, to obtain an image subclass corresponding to each clustering center in the set of clustering centers; obtaining a set of image subclasses based on the image subclass corresponding to each clustering center; the set of image subclasses comprises the image subclass corresponding to each clustering center; and the set of image subclasses is used for image retrieval. Through the application, the accuracy and efficiency of image clustering can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image clustering method, apparatus and computer-readable storage medium. Background Technology

[0002] Large-scale image retrieval often relies on bucketing, which involves dividing the original large dataset into multiple non-overlapping subsets, each belonging to a bucket. During retrieval, the system only needs to find matching samples from the bucket that best matches the target sample. Therefore, bucketing improves retrieval efficiency. Currently, the commonly used bucketing method is k-means clustering. For example, if 1 million samples are divided into 10,000 buckets, the cluster centers will be 10,000. It can be seen that the effectiveness of bucketing has a significant impact on the final retrieval results. Ideally, bucketing should result in high similarity within each bucket, allowing samples with similar features to be grouped into the same bucket, making the recall of a particular bucket similar to the true samples. However, conventional k-means clustering methods produce unstable clustering results and require significant time overhead when processing massive amounts of data, thus reducing the accuracy and efficiency of image clustering. Summary of the Invention

[0003] This application provides an image clustering method, apparatus, and computer-readable storage medium, which can improve the accuracy and efficiency of image clustering.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides an image clustering method, including:

[0006] Semantic information of each image in the image set to be processed is obtained, and based on the semantic information of each image, the image set to be processed is divided into semantic clusters to obtain n semantic sub-clusters; where n is a positive integer greater than or equal to 1.

[0007] Image clustering is performed within each of the n semantic subclusters to obtain the corresponding semantic subclusters. The set of cluster centers corresponding to the n semantic subclusters is obtained by using i as cluster centers; where i is a positive integer greater than or equal to 1 and less than or equal to n; It is a positive integer greater than or equal to 1;

[0008] Based on the cluster center set, the image set to be processed is clustered to obtain image subclasses corresponding to each cluster center in the cluster center set;

[0009] Based on the image subclass corresponding to each cluster center, an image subclass set is obtained; the image subclass set includes the image subclass corresponding to each cluster center in the cluster center set; the image subclass set is used for image retrieval.

[0010] This application provides an image clustering apparatus, comprising:

[0011] The classification and recognition module is used to extract semantic information from each image in the image set to be processed, and to divide the image set to be processed into semantic clusters based on the semantic information of each image to be processed, thereby obtaining n semantic sub-clusters; where n is a positive integer greater than or equal to 1.

[0012] The intra-cluster clustering module is used to perform image clustering within each of the n semantic sub-clusters, obtaining the image clustering corresponding to each semantic sub-cluster. The set of cluster centers corresponding to the n semantic subclusters is obtained by using i as cluster centers; where i is a positive integer greater than or equal to 1 and less than or equal to n; It is a positive integer greater than or equal to 1;

[0013] The inter-cluster clustering module is used to cluster the image set to be processed based on the cluster center set to obtain an image subclass corresponding to each cluster center in the cluster center set; and to obtain an image subclass set based on the image subclass corresponding to each cluster center; the image subclass set includes the image subclass corresponding to each cluster center in the cluster center set; and the image subclass set is used for image retrieval.

[0014] In the above device, the classification and recognition model is obtained by self-supervised collaborative training of the initial classification and recognition model using the labeled semantic classification results and labeled cluster centers corresponding to the sample image set; the labeled cluster centers are obtained by clustering the sample embedding feature vectors of the sample images in the sample image set; the classification and recognition model is also used to extract features from each image to be processed to obtain an embedded feature vector; the embedded feature vector contains the semantic information; based on the embedded feature vector and n preset semantic classifications, each image to be processed is classified to obtain the n semantic subclusters corresponding to the image set to be processed, and each semantic subcluster corresponds to one preset semantic type among the n preset semantic classifications.

[0015] In the above-described device, the image clustering device further includes a model training module, wherein,

[0016] The model training module is used to extract sample embedding feature vectors from each sample image in the sample image set using the initial classification and recognition model before extracting features from each image to be processed using the classification and recognition model to obtain the embedded feature vectors; each sample image contains the labeled semantic classification result; based on a preset number of centers and the sample embedding feature vectors, each sample image is clustered to obtain a preset number of sample cluster centers, and the sample cluster center with the smallest distance to the sample embedding feature vector is taken as the labeled cluster center corresponding to each sample image; in the current round of training, based on the sample embedding feature vectors, semantic classification prediction is performed on each sample image to obtain the semantic classification prediction result corresponding to each sample image, and then the semantic classification is determined according to the semantic classification. The class prediction results yield a set of sample semantic subclusters for each round of training. Within each sample semantic subcluster, cluster prediction is performed based on the sample embedding feature vector of each sample image to obtain the predicted cluster center for each sample image. The semantic classification prediction results are compared with the labeled semantic classification results to obtain a classification loss. The predicted cluster centers are compared with the labeled cluster centers to obtain an equilibrium loss. The equilibrium loss characterizes the distribution difference between the predicted cluster centers and the labeled cluster centers. The classification loss and the equilibrium loss are used as the total loss for the current round of training. The model parameters of the initial classification and recognition model are updated based on the total loss, and the next round of training is performed based on the updated model parameters until the total loss is less than a preset training threshold, thus obtaining the classification and recognition model.

[0017] In the aforementioned apparatus, the intra-cluster clustering module is further configured to, for each semantic sub-cluster, multiply the number of sub-cluster images in each semantic sub-cluster by the proportion of cluster centers, as... The number of cluster centers; the cluster center ratio is the ratio of the total number of images to be processed in the image set to the preset total number of cluster centers; the sub-cluster image is the image to be processed contained in each semantic sub-cluster; based on the The number of cluster centers and the image features of each sub-cluster image are used to cluster the sub-cluster images within each semantic sub-cluster to obtain the... Cluster centers.

[0018] In the above device, the inter-cluster clustering module is further configured to calculate the distance from each image to be processed to each cluster center in the cluster center set; take the cluster center corresponding to the minimum distance as the target cluster center corresponding to each image to be processed; divide the images to be processed corresponding to the same target cluster center into an image subclass, thereby obtaining the image subclass corresponding to each cluster center.

[0019] In the above apparatus, the inter-cluster clustering module is further configured to: after obtaining an image subclass set based on the image subclass corresponding to each cluster center, calculate the number of images contained in each image subclass; designate image subclasses with fewer than a preset threshold number of images as subclasses to be merged, calculate the center distance from the cluster center to be merged of the subclass to be merged to at least one other cluster center in the cluster center set; merge the subclass to be merged with the image subclasses whose center distance is less than a preset distance threshold to obtain merged image subclasses, update the image subclass set using the merged image subclasses; calculate the merged cluster center of the merged image subclasses, and update the cluster center set using the merged cluster center.

[0020] In the above-described device, the image clustering device further includes a retrieval module. The retrieval module is used to: after obtaining an image subclass set based on the image subclass corresponding to each cluster center, acquire the image to be retrieved and the retrieval features corresponding to the image to be retrieved; compare the retrieval features with the cluster center set to determine the retrieval cluster center matching the retrieval features; take the image subclass corresponding to the retrieval cluster center in the image subclass set as the target image subclass; and determine the target image matching the image to be retrieved in the target image subclass according to the retrieval features.

[0021] This application provides an image clustering apparatus, including:

[0022] Memory, used to store executable instructions;

[0023] The processor, when executing executable instructions stored in the memory, implements the image clustering method provided in the embodiments of this application.

[0024] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the image clustering method provided in this application.

[0025] The embodiments of this application have the following beneficial effects:

[0026] Before performing image clustering, the entire set of images to be processed is pre-clustered using semantic information. This allows images with similar semantic content to be grouped into the same category. The semantic similarity within each semantic sub-cluster improves the accuracy of image clustering. Therefore, when performing retrieval based on the image clustering results, samples with more similar semantics can be retrieved as samples in the same or similar buckets, resulting in more accurate retrieval results. Furthermore, by clustering within each semantic sub-cluster, the semantic content of each sub-cluster is obtained. By identifying cluster centers and then obtaining the complete set of cluster centers, we can reduce the instability and uncertainty caused by randomly initializing cluster centers on the entire dataset, thus improving the accuracy of image clustering. Furthermore, clustering can be performed within a limited overhead, enabling efficient clustering of large-scale data samples with limited resources, thereby improving the efficiency of image clustering. Moreover, by re-clustering the entire set of images to be processed based on the cluster center set determined within the semantic sub-clusters, we can simultaneously improve the clustering balance within and between sub-clusters, achieving a global clustering balance effect. Attached Figure Description

[0027] Figure 1 This is a schematic diagram illustrating the current process of using image clustering for large-scale image retrieval;

[0028] Figure 2 This is an optional structural diagram of the image clustering retrieval system architecture provided in the embodiments of this application;

[0029] Figure 3 This is an optional structural schematic diagram of the image clustering device provided in the embodiments of this application;

[0030] Figure 4 This is an optional flowchart illustrating the image clustering method provided in this application embodiment;

[0031] Figure 5 This is an optional flowchart illustrating the image clustering method provided in this application embodiment;

[0032] Figure 6 This is an optional flowchart illustrating the image clustering method provided in this application embodiment;

[0033] Figure 7 This is a schematic diagram illustrating the initial classification and recognition model training process provided in the embodiments of this application;

[0034] Figure 8 This is an optional flowchart illustrating the image clustering method provided in this application embodiment;

[0035] Figure 9 This is an optional flowchart illustrating the image clustering method provided in this application embodiment;

[0036] Figure 10 This is an optional flowchart illustrating the image clustering method provided in this application embodiment;

[0037] Figure 11 This is a schematic diagram comparing the image clustering method provided in this application embodiment with the implementation process of current clustering methods. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0040] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0042] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0043] 1) Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0044] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0045] 2) Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0046] 3) Image recognition: Category-level recognition, which does not consider the specific instance of an object, but only the category of the object (such as human, dog, cat, bird, etc.) and identifies the category to which the object belongs. A typical example is the recognition task in the large-scale general object recognition open-source dataset ImageNet, which identifies which of 1000 categories an object belongs to.

[0047] 4) ImageNet: A large-scale, general-purpose open-source dataset for object recognition.

[0048] 5) ImageNet pre-trained model: A deep learning network model is trained based on ImageNet, and the parameter weights of this model are the ImageNet pre-trained model.

[0049] 6) Self-Supervised Learning: Self-supervised learning is supervised learning without human labeling; it can be viewed as supervised learning without human intervention. Self-supervised learning utilizes auxiliary tasks (pretext) to mine its own supervisory information from large-scale unsupervised data. This constructed supervisory information is used to train the network, thereby learning representations valuable for downstream tasks. Self-supervised learning does not rely on any label values; the supervisory information in self-supervised learning is not manually labeled, but rather the algorithm automatically constructs supervisory information (pretext) from the intrinsic features of large-scale unsupervised data for supervised learning or training. This is useful for finding relationships between samples, such as in clustering-related tasks.

[0050] 7) Supervised learning: Supervised learning uses a large amount of labeled data to train the model. After the model's prediction and the true label of the data produce a loss, backpropagation is performed (calculating gradients and updating parameters). Through continuous learning, the model can eventually gain the ability to identify new samples.

[0051] 8) Feature Representation: The task of feature representation is to abstract the samples to be processed into numerical vectors for use by machine learning models. Therefore, the feature representation task needs to take into account both feature attributes and model requirements.

[0052] Currently, the process of applying image clustering for large-scale image retrieval can be as follows: Figure 1 As shown. For a large number of pre-stored images in the image library, a pre-defined number of clusters M can be established. Pre-stored image features are extracted from each image, and the k-means clustering algorithm is used to cluster the images based on the sample features and the number of clusters M, resulting in M ​​cluster centers and corresponding image subsets. When searching the image library using a query image, the query image features can be projected onto the feature space of the M cluster centers to obtain the corresponding projected features. Through cluster comparison, the projected features are compared with the M cluster centers to determine the target cluster center corresponding to the projected features. The image subset containing the target cluster center is then used as the recall class. Among the multiple pre-stored image features in the recall class, the target feature matching the projected features is determined through feature comparison. The pre-stored image corresponding to the target feature is then used as the target image to match the query image, completing the image matching process. However, since the k-means algorithm randomly initializes the cluster centers, current methods of direct clustering using k-means make the clustering results highly dependent on initialization, exhibiting instability and uncertainty. Furthermore, when dealing with massive datasets, k-means clustering requires significant computational resources, resulting in excessive overhead and low efficiency in image clustering. Moreover, current image clustering methods lack semantic information, failing to guarantee that semantically similar content will be clustered into the same class; they also cannot guarantee a balanced number of clusters, potentially leading to extreme imbalances where all data is distributed only into the same clustered subset of images (bugging), thus failing to leverage the efficiency-enhancing effects of bucketing.

[0053] This application provides an image clustering method, apparatus, and computer-readable storage medium, which can improve the accuracy and efficiency of image clustering. The exemplary application of the image clustering apparatus provided in this application is described below. The image clustering apparatus provided in this application can be implemented as various types of user terminals such as laptops, tablets, desktop computers, set-top boxes, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), or as a server. The exemplary application of the image clustering apparatus as a server will be described below.

[0054] See Figure 2 , Figure 2 This is an optional architecture diagram of the image clustering retrieval system 100 provided in the embodiments of this application. In order to support an image retrieval application, the terminal 400 connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0055] Terminal 400 submits the image to be retrieved and the search range to server 200 via network 300, and receives the most matching target image retrieved by server 200 from the image set corresponding to the search range based on the image to be retrieved, and displays the target image on graphical interface 410. Server 200 obtains the image to be retrieved and the search range, and uses the image set corresponding to the search range as the image set to be processed from the sample image library stored in database 500; it extracts semantic information from each image to be processed in the image set, and divides the image set into semantic clusters based on the semantic information of each image to be processed, obtaining n semantic subclusters; where n is a positive integer greater than or equal to 1; for each of the n semantic subclusters, it extracts image features from at least one subcluster image in each semantic subcluster; and it performs image clustering on at least one subcluster image based on the image features to obtain the corresponding semantic subcluster. One cluster center is obtained, which in turn yields n semantic subclusters. *n* cluster centers; where *i* is a positive integer greater than or equal to 1 and less than or equal to *n*; based on Cluster centers are used to cluster the image set to be processed, resulting in... The server can further extract retrieval features from the image to be retrieved, and then use these features to retrieve... The image subclass determines the target image subclass that matches the image to be retrieved; then, the target image that matches the image to be retrieved is further determined in the target image subclass, the retrieval of the image to be retrieved is completed, and the target image is returned to the terminal 400 through the network 300.

[0056] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 400 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited thereto. Terminals and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0057] See Figure 3 , Figure 3 This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application. Figure 3 The server 200 shown includes at least one processor 410, memory 450, at least one network interface 420, and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 3 The general labeled all buses as Bus System 440.

[0058] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0059] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0060] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0061] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0062] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0063] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0064] The network communication module 452 is used to reach other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0065] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with user interface 430 (e.g., a display screen, a speaker, etc.).

[0066] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0067] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 3 An image clustering device 455 stored in memory 450 is shown. It can be software in the form of programs and plug-ins, including the following software modules: classification and recognition module 4551, intra-cluster clustering module 4552 and inter-cluster clustering module 4553. These modules are logical and can therefore be arbitrarily combined or further split according to the functions they implement.

[0068] The functions of each module will be explained below.

[0069] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image clustering method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0070] The image clustering method provided in this application will be described in conjunction with exemplary applications and implementations of the server provided in the embodiments of this application.

[0071] See Figure 4 , Figure 4 This is an optional flowchart illustrating the image clustering method provided in this application embodiment, which will be combined with... Figure 4 The steps shown are explained.

[0072] S101. Obtain the semantic information of each image in the image set to be processed, and based on the semantic information of each image, divide the image set to be processed into semantic clusters to obtain n semantic sub-clusters; where n is a positive integer greater than or equal to 1.

[0073] In this embodiment, the set of images to be processed can be a collection of data from an image library used for image retrieval or image recognition, containing at least one image to be processed. The server can obtain semantic information from each image to be processed and use this semantic information as a clustering constraint to improve cluster relevance.

[0074] In some embodiments, the server can use a pre-trained embedding model in image recognition to perform image recognition on each image to be processed, thereby obtaining the embedding of each image to be processed. Since the embedding of the image to be processed comes from image recognition and contains image semantic information, the server can thus obtain the semantic information of each image to be processed.

[0075] Here, the embedded feature vector can be any embedding output by a network model with semantic information. For example, the embedded feature vector can be the embedding output by an artificial intelligence-based neural network model when performing retrieval or clustering tasks, or it can be the embedding output by any semantically rich recognition task, such as the embedding output by an ImageNet 1001 class recognition model.

[0076] In this embodiment, the server can perform semantic inference on each image to be processed based on the semantic information of each image to be processed and the preset semantic classification. From n preset semantic classifications, the server can determine the preset semantic type to which each image to be processed belongs, thereby completing the semantic cluster division of the image set to be processed into n semantic sub-clusters; where n is a positive integer greater than or equal to 1.

[0077] In some embodiments, the server may use a pre-trained classification and recognition model, for example, a 1001-class recognition model or an ImageNet 1001-class recognition model, to perform semantic clustering on the image set to be processed, generating 1001 class predictions, thereby dividing the image set to be processed into 1001 non-overlapping semantic sub-clusters.

[0078] In some embodiments, the server can use a residual network model as an initial model to train a classification and recognition model for semantic cluster partitioning. The initial model can be a neural network model containing at least convolutional layers (Conv1-Conv5) and a classification layer, such as the ResNet-101 model.

[0079] For example, when the initial model is a ResNet-101 model, during the parameter initialization stage of constructing the initial model, the parameters of the ResNet-101 pre-trained on the ImageNet dataset can be used to initialize the convolutional layers Conv1-Conv5 in the ResNet-101 model, and the classification layers such as the fully connected layer Fc_cr layer in the ResNet-101 model can be initialized using a Gaussian distribution with a variance of 0.01 and a mean of 0. All parameters in the ResNet-101 model are set to the state to be learned. In each training round, the ResNet-101 model is used to perform forward computation on an input sample image, obtaining the classification prediction result through the Fc_cr layer. The classification prediction result is compared with the true class label of the sample image, and the classification loss value of the model is calculated. Based on the classification loss value, the gradient is calculated backward to obtain the updated values ​​of all model parameters, and the ResNet-101 model is updated accordingly until the preset training objective is reached. If the preset number of iterations is reached, or the classification loss value is less than the preset classification loss threshold, the training ends, and the trained ResNet-101 model is obtained as the classification and recognition model.

[0080] In some embodiments, the network structure of the convolutional layers in the ResNet-101 model can be as shown in Table 1:

[0081]

[0082] Table 1

[0083] In some embodiments, the network structure of the classification layer in the ResNet-101 model can be as shown in Table 2. Here, N represents the number of preset semantic categories that the model needs to learn during training. (See Table 2 for details.)

[0084]

[0085] Table 2

[0086] It should be noted that in some embodiments, the server may also use initial models with different network structures and different pre-trained model weights for model training, such as the ResNet-50 model. Alternatively, in addition to the category model trained on ImageNet, the server may also use other large-scale dataset classification models, such as OpenImage, as the initial model. The specific selection depends on the actual situation, and this application embodiment does not limit the choice.

[0087] S102. Perform image clustering within each of the n semantic subclusters to obtain the corresponding semantic subclusters. We obtain a set of cluster centers for each of the n semantic subclusters; where i is a positive integer greater than or equal to 1 and less than or equal to n. It is a positive integer greater than or equal to 1.

[0088] In this embodiment, based on the n semantic subclusters divided according to semantic information, the server can use a clustering algorithm to perform image clustering on at least one image to be processed contained within each semantic subcluster, obtaining at least one cluster center for each semantic subcluster. Since the number of images to be processed contained in each semantic subcluster is different, the number of cluster centers obtained by the clustering algorithm for each semantic subcluster is different. Here, we use... This represents the number of cluster centers contained in each semantic subcluster, where i is a positive integer greater than or equal to 1 and less than or equal to n, and i represents the i-th semantic subcluster among n semantic subclusters.

[0089] Here, when the server performs image clustering on at least one image to be processed within each semantic sub-cluster, it can perform clustering based on the semantic information of the image to be processed obtained in S101, or it can extract image features that can be used for retrieval from the image to be processed, such as image retrieval services, according to the specific image processing business that the image clustering is aimed at, such as image retrieval business, and then perform clustering based on the image features. The specific choice is made according to the actual situation, and this application embodiment does not limit it.

[0090] In this embodiment of the application, after the server performs clustering processing on the images to be processed contained within each semantic sub-cluster, it can obtain the corresponding semantic sub-cluster. There are n cluster centers. The server processes n semantic subclusters using the same procedure, thus obtaining the corresponding semantic subclusters. Each cluster center is considered as a set of cluster centers.

[0091] S103. Based on the cluster center set, cluster the image set to be processed to obtain the image subclass corresponding to each cluster center in the cluster center set.

[0092] In this embodiment, the server can use the set of cluster centers obtained by intra-cluster clustering as the set of cluster centers corresponding to the entire set of images to be processed, and re-cluster the set of images to be processed. The server can recalculate the distance from each image to be processed in the set of images to be processed to each cluster center in the set of cluster centers, and assign the cluster center with the minimum distance to the target cluster center corresponding to each image to be processed. The images to be processed corresponding to the same target cluster center are divided into an image subclass, thus obtaining the image subclass corresponding to each cluster center.

[0093] In some embodiments, the server can calculate the values ​​of each image to be processed to... The distance between cluster centers is used; the cluster center corresponding to the minimum distance is taken as the target cluster center for each image to be processed; images to be processed with the same target cluster center are divided into one image subclass, thus obtaining... Image subclasses.

[0094] S104. Based on the image subclass corresponding to each cluster center, obtain the image subclass set; the image subclass set includes the image subclass corresponding to each cluster center in the cluster center set; the image subclass set is used for image retrieval.

[0095] In this embodiment of the application, the server can aggregate the image subclasses corresponding to each cluster center into an image subclass set to obtain the final clustering result.

[0096] In some embodiments, the server may save the set of image subclasses as the binning result of the image set to be processed, so as to facilitate image retrieval based on the binning result.

[0097] It is understood that, in this embodiment of the application, before performing image clustering, the entire set of images to be processed is pre-clustered using semantic information. This allows images with similar semantic content to be grouped into the same category, with semantic similarity within each semantic sub-cluster, improving the accuracy of image clustering. Thus, when performing retrieval based on the image clustering results, samples with more similar semantics can be retrieved as samples in the same or similar buckets, resulting in retrieval results that better meet expectations. Furthermore, by clustering within each semantic sub-cluster, the semantic content of each sub-cluster is obtained. By identifying cluster centers and then obtaining the complete set of cluster centers, we can reduce the instability and uncertainty caused by randomly initializing cluster centers on the entire dataset, thus improving the accuracy of image clustering. On the other hand, clustering can be performed within a limited overhead, enabling efficient clustering of large-scale data samples with limited resources, thereby improving the efficiency of image clustering. Furthermore, by re-clustering the entire set of images to be processed based on the cluster center set determined within the semantic sub-clusters, we can simultaneously improve the clustering balance within and between sub-clusters, achieving a global clustering balance effect.

[0098] In some embodiments, see Figure 5 , Figure 5 This is a schematic diagram of an optional process for the image clustering method provided in an embodiment of this application. Figure 4 The shown S101 can be implemented through S1011-S1012, which will be explained in conjunction with each step.

[0099] S1011. Using a classification and recognition model, feature extraction is performed on each image to be processed to obtain an embedded feature vector, which contains semantic information.

[0100] In this embodiment, the server can use a classification and recognition model with a neural network structure to extract an embedded feature vector containing semantic information from each image to be processed, so as to perform further semantic classification based on the feature expression data of the semantic dimension in the embedded feature vector.

[0101] Here, the embedded feature vector extracted by the classification and recognition model may also include feature expression data other than the semantic dimension, such as data used to characterize visual features such as color, shape, and contour in the image. The specific selection is made according to the actual situation, and this application embodiment does not limit it.

[0102] S1012. Based on the embedded feature vector and n preset semantic classifications, each image to be processed is classified to obtain n semantic subclusters corresponding to the image set to be processed. Each semantic subcluster corresponds to one preset semantic type among the n preset semantic classifications. The classification and recognition model is obtained by self-supervised collaborative training of the initial classification and recognition model with the labeled semantic classification results and labeled cluster centers corresponding to the sample image set. The labeled cluster centers are obtained by clustering the sample embedded feature vectors of the sample images in the sample image set.

[0103] In this embodiment of the application, after dividing the image set to be processed into semantic sub-clusters, the instability of the full data clustering can be reduced, and the clustering balance between each semantic sub-cluster can be ensured. To further ensure cluster balance within each semantic sub-cluster, for example, the ratio of cluster centers to images in the semantic sub-cluster with the semantic category "cat" is made the same as the ratio in the semantic sub-cluster with the semantic category "dog." This avoids imbalances such as too many cluster centers in semantic sub-clusters with fewer images or too few cluster centers in semantic sub-clusters with more images. During the initial classification model training process, the server can use the manually labeled semantic classification results of each sample image in the sample image set as supervision information for the semantic classification task. A clustering algorithm is then used to cluster the sample embedding feature vectors of each sample image to obtain labeled cluster centers for each sample image. A self-supervised task is then generated based on these labeled cluster centers for co-training with the semantic classification task. In each training round, the labeled semantic classification results and labeled cluster centers are used as total supervision information to co-train the initial classification model, resulting in the final classification model. Here, the sub-cluster image refers to the image to be processed within the semantic sub-cluster.

[0104] It should be noted that the server calculates labeled cluster centers based on the embedded feature vectors of the samples using a clustering algorithm. The purpose of generating a self-supervised task for co-training with the semantic classification task based on these labeled cluster centers is to enable the initial classification model to learn the correlation between image feature representations and semantic classification information when performing machine learning on the feature representations of sample images. This allows the resulting classification model to more accurately understand the semantic information feature representations, such as embeddings, in the images to be processed. The server can then use the classification model to perform semantic inference on each image to be processed based on this understanding, obtaining the semantic inference result for each image. Furthermore, based on the semantic inference result, it can more accurately calculate the semantic similarity between each image to be processed, and perform classification prediction based on the semantic similarity, ensuring that each semantic sub-cluster achieves a balanced internal clustering effect.

[0105] In this embodiment of the application, after completing the semantic inference and classification prediction for each image to be processed, the server groups the images to be processed corresponding to the same preset semantic type into a semantic sub-cluster, thereby dividing the image set to be processed into n semantic sub-clusters.

[0106] In some embodiments, see Figure 6 , Figure 6 This is an optional flowchart illustrating an image clustering method provided in an embodiment of this application, based on... Figure 5 Before S1011, S001-S007 can also be executed, which will be explained in conjunction with each step.

[0107] S001. Using the initial classification and recognition model, extract the sample embedding feature vector from each sample image in the sample image set; each sample image contains the labeled semantic classification result.

[0108] In this embodiment, the server can establish an initial classification and recognition model. Using this model, visual attributes such as color, shape, and texture are extracted from each sample image. Furthermore, based on the image features or other pixel features extracted from each sample image, the server performs image recognition on each sample image using the initial classification and recognition model, and then extracts semantic information from the image recognition results. The server uses the semantic information and image features extracted from each sample image as a sample embedding feature vector.

[0109] In some embodiments, the initial classification and recognition model may include a feature extraction layer. The server can extract the embedding feature vector from each sample image through the feature extraction layer of the initial classification and recognition model, and use it as the sample embedding feature vector.

[0110] Here, the sample image set contains multiple sample images used to train the initial classification and recognition model, and each sample image contains manually annotated semantic classification results.

[0111] Here, the sample embedding feature vector extracted by the feature extraction layer can be a feature representation of the corresponding sample image in multiple dimensions. The sample embedding feature vector can contain feature representation data of semantic information dimension and visual attribute dimension. It can also contain feature representation data of other dimensions according to the specific retrieval business or image processing business. The specific selection is based on the actual situation, and this application embodiment does not limit it.

[0112] In some embodiments, the feature extraction layer may be the pooling layer in the ResNet-101 network model; corresponding to different forms of sample image sets, the labeled semantic classification results may be 1001 classes of ImageNet labeled data, OpenImage recognition data, or general recognition data in business, etc. The specific selection is made according to the actual situation, and this application embodiment does not limit it.

[0113] S002. Based on the preset number of centers and the sample embedding feature vector, each sample image is clustered to obtain the preset number of sample cluster centers, and the sample cluster center with the smallest distance from the sample embedding feature vector is taken as the labeled cluster center corresponding to each sample image.

[0114] In this embodiment, the server can pre-set the total number of cluster centers for clustering the sample embedding feature vectors of all sample images as the preset number of centers, and then initialize the preset number of initial sample cluster centers. Based on the distance between the sample embedding feature vector and the initial sample cluster centers, the server iteratively performs clustering calculations on each sample image to obtain the preset number of sample cluster centers. The server calculates the distance from the embedding feature vector of each sample image to each sample cluster center, and takes the sample cluster center with the smallest distance to each sample image as the labeled cluster center corresponding to that sample image. The labeled cluster center can then be used as the feature representation annotation data of that sample image to generate the corresponding self-supervised task, and co-train it with the semantic classification task, associating the feature representation learned by the initial classification and recognition model from the sample embedding feature vector with semantic information.

[0115] In some embodiments, the server can set the preset number of centers based on the number of categories in the labeled semantic classification results. For labeled semantic classification results corresponding to n preset semantic classifications, the preset number of centers can be set to a positive integer multiple of n. For example, the server can set the preset number of centers to n*100, and use the k-means clustering algorithm to cluster each sample image based on the sample embedding feature vector, obtaining 100n cluster center vectors as sample cluster centers; for each sample image, the server selects the nearest sample cluster center as the labeled cluster center corresponding to that sample image based on the distance between its sample embedding feature vector and the 100n sample cluster centers. In this way, the server can use each sample image and its corresponding labeled cluster center as supervision information for the sample data-cluster label during training, and perform self-supervised training of the initial classification and recognition model's learning of feature representation.

[0116] It should be noted that, here, the server can cluster each sample image based on the feature vector data of the semantic information dimension in the sample embedding feature vector, or it can cluster each sample image based on the feature vector data of other dimensions in the sample embedding feature vector, such as the feature vector data of the visual attribute dimension. The specific choice depends on the actual situation, and this application embodiment does not limit it.

[0117] S003. In the current round of training, based on the sample embedding feature vector, semantic classification prediction is performed on each sample image to obtain the semantic classification prediction result corresponding to each sample image. Then, based on the semantic classification prediction result, the set of sample semantic subclusters for each round of training is obtained.

[0118] In this embodiment, since the sample embedding feature vector contains the semantic information of the corresponding sample image, for the current round of training in multiple iterations, the server can understand the semantic information in the sample embedding feature vector through the initial classification and recognition model, and make forward prediction of the semantic classification to which the semantic information belongs based on the semantic understanding result, so as to obtain the semantic classification prediction result corresponding to each sample image.

[0119] In some embodiments, the initial classification and recognition model may include a classification layer. For example, a fully connected layer in a neural network model may be used as the classification layer, and the classification layer may be connected to the feature extraction layer to perform semantic classification prediction on the sample embedding feature vector output by the feature extraction layer, thereby obtaining the semantic classification prediction result corresponding to each sample image.

[0120] In this embodiment, the server groups sample images with the same semantic classification prediction result into a sample semantic subcluster. Then, by classifying the semantic classification prediction results of the entire sample image set, the server obtains the sample semantic subcluster set corresponding to the current training round. Each sample semantic subcluster set contains at least one sample image with the same semantic classification prediction result. The server can perform the same process in each round of training in multiple iterations to obtain the sample semantic subcluster set corresponding to each round of training.

[0121] S004. In each sample semantic sub-cluster of the sample semantic sub-cluster set, cluster prediction is performed based on the sample embedding feature vector of each sample image to obtain the predicted cluster center corresponding to each sample image.

[0122] In this embodiment of the application, during the current round of training, for a sample semantic sub-cluster of the sample semantic sub-cluster set, the server can use the initial classification and recognition model to perform cluster prediction on the sample images contained in the sample semantic sub-cluster based on the sample embedding feature vector of each sample image in the sample semantic sub-cluster, thereby obtaining the predicted cluster center corresponding to each sample image in the sample semantic sub-cluster. The server processes each sample semantic sub-cluster in the same way to obtain the predicted cluster center corresponding to each sample image in each sample semantic sub-cluster.

[0123] In some embodiments, the initial classification and recognition model may include a feature reprojection layer. During the current training round, the feature extraction layer can embed the extracted samples into feature vectors and output them to the feature reprojection layer, which then performs clustering prediction based on the embedded feature vectors.

[0124] Here, the feature reprojection layer can map the feature representation data of the preset attribute dimension contained in the embedded feature vector of the sample to the feature space where the preset number of sample cluster centers are located, to obtain the sample projection features. Based on the sample projection features, cluster prediction is performed on the sample images contained in each sample semantic sub-cluster to obtain the predicted cluster center corresponding to each sample image. The preset attribute dimension can be the image attribute dimension determined by the image processing business to be applied based on image clustering, such as the visual attribute dimension.

[0125] S005. Compare the semantic classification prediction results with the labeled semantic classification results to obtain the classification loss.

[0126] In this embodiment, the semantic classification prediction result represents the prediction result of the initial classification and recognition model performing forward calculation on the semantic category of the sample image, and the labeled semantic classification result represents the true category of the preset semantic category to which the sample image belongs, as manually labeled. The server compares the semantic classification prediction result with the labeled semantic classification result and obtains the classification loss by calculating the error between the two.

[0127] S006. Compare the predicted cluster centers with the labeled cluster centers to obtain the equilibrium loss; the equilibrium loss characterizes the distribution difference between the predicted cluster centers and the labeled cluster centers.

[0128] In this embodiment, the predicted cluster center represents the clustering prediction result of the initial classification model on the sample embedded feature vector, and the labeled cluster center represents the labeled clustering result of the sample embedded feature vector. The server can obtain the equilibrium loss by comparing the distribution difference between the predicted cluster center and the labeled cluster center. In each round of clustering prediction, the sample image is constrained to shift towards the labeled cluster center through the equilibrium loss.

[0129] In some embodiments, the equilibrium loss can be the K-L divergence loss to maintain the consistency of the distributions of the two predicted cluster centers with the labeled cluster centers. The server can calculate the equilibrium loss using formula (1), as follows:

[0130] (1)

[0131] in, For K_L divergence loss, For each round of training, the corresponding sample images, For sample images The predicted cluster centers output by the feature reprojection layer after inputting the initial classification and recognition model; For sample images The corresponding labeled cluster centers.

[0132] S007. Use the classification loss and the equalization loss as the total loss for the current round of training. Update the model parameters of the initial classification and recognition model based on the total loss and conduct the next round of training based on the updated model parameters until the total loss is less than the preset training threshold, thus obtaining the classification and recognition model.

[0133] In this embodiment, the server obtains the total training loss for each round of training based on the classification loss corresponding to the semantic classification task and the equilibrium loss corresponding to the self-supervised task. This total supervision information is used to perform backward gradient updates and model parameter iterations on the network model parameters of the initial classification and recognition model until the preset training conditions are met, such as reaching the preset number of training iterations or the total training loss being less than the preset training loss threshold. Then, the training ends, and the classification and recognition model is obtained.

[0134] In some embodiments, the training process of the server on the initial classification and recognition model can be as follows: Figure 7As shown, the server inputs the sample image set into the feature extraction layer of the initial classification and recognition model. The feature extraction layer extracts the first feature vector data `embedding1` and the second feature vector data `embedding2` from each sample image. `embedding1` contains the semantic information of the sample image; `embedding2` is the visual attribute dimension of the embedding data, such as the outline, texture, color, and shape of the main object in the sample image. During each training round, the server performs collaborative training on the initial classification and recognition model through two branches: a supervised branch and a collaborative branch. For the supervised branch, the classification layer performs a semantic classification task based on the semantic information in `embedding1` and the preset classification label (i.e., the labeled semantic classification result) of each sample image, obtaining the semantic classification prediction result for each sample image, and thus classifying the sample image set into a set of sample semantic subclusters. For the collaborative branch, a clustering algorithm is used beforehand to cluster the `embedding2` of each sample image to obtain the labeled cluster centers corresponding to each sample image, thereby generating a self-supervised task on the collaborative branch. The self-supervised task and the supervised branch are trained collaboratively. Based on the set of sample semantic subclusters obtained from the supervised branch, a feature reprojection layer is used to perform cluster prediction on the sample images within each sample semantic subcluster based on embedding2, obtaining the cluster prediction result for each sample image, i.e., the predicted cluster center. In this way, during each training round, the server can obtain the classification loss by using the semantic classification prediction result of each sample image on the supervised branch and its preset classification label, and obtain the balance loss by using the predicted cluster center and labeled cluster center of each sample image on the collaborative branch. The classification loss and the balance loss are used as the total loss for each training round to iteratively update the network parameters of the initial classification and recognition model, so that the initial classification and recognition model continuously learns the relationship between image semantic information and feature representation, and finally obtains the trained classification and recognition model.

[0135] It is understood that in this embodiment of the application, the server performs feature reprojection of the model based on the sample semantic sub-cluster data, which can realize the gradual advancement process of cluster balance during the model training process, so that the trained classification and recognition model can achieve a balanced distribution within each semantic sub-cluster when dividing semantic clusters, thereby improving the balance of image clustering within semantic sub-clusters. Since each semantic sub-cluster is obtained by pre-clustering the image through semantic information, the final effect of cluster balance inside and outside the semantic sub-cluster, that is, global cluster balance, can be achieved.

[0136] In some embodiments, see Figure 8 , Figure 8 This is a schematic diagram of an optional process for the image clustering method provided in an embodiment of this application. Figure 4 , Figure 5 or Figure 6 The step S102 shown in the figure obtains the corresponding semantic sub-cluster. The process of identifying cluster centers can be achieved by executing S1021-S1022, which will be explained in detail below.

[0137] S1021. For each semantic subcluster, the product of the number of subcluster images in each semantic subcluster and the proportion of cluster centers is used as... The number of cluster centers; the cluster center ratio is the ratio of the total number of images to be processed in the image set to the preset total number of cluster centers; the sub-cluster image is the image to be processed contained in each semantic sub-cluster.

[0138] In this embodiment, for each of the n semantic subclusters, the server can pre-calculate the number of cluster centers corresponding to that semantic subcluster in the subsequent clustering algorithm based on the number of images to be processed in each semantic subcluster and the total number of images to be processed in the image set, combined with the preset total number of cluster centers. Number of cluster centers.

[0139] here, Representing the nth semantic subcluster A semantic sub-cluster Characterizing the first The number of cluster centers corresponding to each semantic sub-cluster, since each semantic sub-cluster contains different images to be processed, corresponds to the calculated number of cluster centers for each semantic sub-cluster. They are not the same either.

[0140] S1022, based on The number of cluster centers and the image features of each sub-cluster image are used to cluster the sub-cluster images within each semantic sub-cluster, resulting in... Cluster centers.

[0141] In this embodiment of the application, for each semantic sub-cluster, the server can perform processing within each semantic sub-cluster based on the image features of each image to be processed. The clustering calculation with a certain number of cluster centers yields the corresponding semantic sub-cluster. Cluster centers.

[0142] In this embodiment, when the server extracts an embedded feature vector containing semantic information from each image to be processed through a classification and recognition model in S101, the image features of each image to be processed can be image features extracted synchronously by the classification and recognition model and semantic information, used for image processing business related to image clustering; or, the image features can also be semantic information, that is, clustering is performed in each semantic sub-cluster through semantic information; or, the image features can also be obtained by another feature extraction module extracting features from each image to be processed. The specific selection is made according to the actual situation, and this embodiment does not limit it.

[0143] In some embodiments, the number of sub-cluster images contained in the i-th semantic sub-cluster is M, the total number of images to be processed in the image set to be processed is sum, the preset total number of cluster centers is Kcluster, and the server can calculate the number of clusters in the i-th semantic sub-cluster proportionally. The server extracts retrieval features from each sub-cluster image in the i-th semantic sub-cluster using a feature extraction module, and then uses the k-means clustering algorithm to perform M sub-cluster clustering based on the retrieval features. Clustering of centers yields There are n cluster centers. The server clusters all n semantic subclusters and obtains... , that is Cluster centers.

[0144] It is understandable that after dividing the total sum of images to be processed into n semantic subclusters, the average number of samples in a subcluster is sum / n. Clustering within a subcluster requires significantly less data than clustering the entire dataset, thus greatly reducing the amount of data required for image clustering. For large-scale semantic classification models, such as those with tens of thousands of classes, n can reach 10,000. The method described in this application can reduce the number of samples required for image clustering by four orders of magnitude, thereby enabling large-scale clustering under limited computing resources. Furthermore, when clustering within a semantic subcluster, by pre-calculating the number of cluster centers corresponding to that semantic subcluster proportionally, the instability problem of semantic clustering can be further avoided, ensuring that the number of cluster centers within a semantic subcluster is proportional to the number of images in the subcluster, improving the balance of clustering within the semantic subcluster, and thus improving the accuracy of image clustering.

[0145] In some embodiments, see Figure 9 , Figure 9 This is an optional flowchart illustrating the image clustering method provided in this application embodiment. After S103, S103_1 to S103_4 can also be executed, and will be described in conjunction with each step.

[0146] S103_1 Calculate the number of images contained in each image subclass.

[0147] In this embodiment of the application, for the set of image subclasses obtained through the image clustering process, the server can check the effect of image clustering by calculating the number of images contained in each image subclass, and check whether there are any image subclasses with too few images in the set of classified image subclasses.

[0148] S103_2. Take the image subclass with less than a preset number threshold as the subclass to be merged, and calculate the center distance from the cluster center to be merged of the subclass to at least one other cluster center in the cluster center set.

[0149] In this embodiment, if an image subclass contains too few images, it indicates that there may be an imbalance in clustering within that subclass or other similar subclasses. The server can identify image subclasses with fewer than a preset clustering data threshold as subclasses to be merged, use the cluster centers of these subclasses as the merged cluster centers, and calculate the distances from the merged cluster center to each of the other cluster centers in the cluster center set. This results in the center distances of the merged cluster center relative to each of the other cluster centers, which are then used as at least one center distance.

[0150] S103_3: Merge the subclass to be merged with the image subclass whose center distance is less than a preset distance threshold to obtain the merged image subclass, and update the image subclass set using the merged image subclass.

[0151] In this embodiment, when the center distance is less than a preset distance threshold, it indicates that there are other image subclasses that can be merged near the subclass to be merged, which has too few images. The server merges the subclass to be merged into other image subclasses whose center distance is less than the preset distance threshold, thus obtaining the merged image subclass.

[0152] Here, the server can identify at least one other cluster center whose center distance from the cluster center to be merged is less than a preset distance threshold, and merge at least one other image subclass corresponding to the at least one other cluster center with the subclass to be merged to obtain the merged image subclass.

[0153] Here, when merging at least one other image subclass with the subclass to be merged, all at least one other image subclass can be merged with the subclass to be merged, or the other image subclass that is closest to the subclass to be merged among at least one other image subclass can be merged with the subclass to be merged, or other merging strategies can be adopted according to the actual situation. This application embodiment does not limit this.

[0154] In this embodiment, the server can detect at least one image subclass to be merged from the image subclass set through the above-described checking and merging process. The server then performs clustering and merging on each of the at least one image subclass to be merged through the above-described clustering and merging process to obtain at least one merged image subclass. The server uses the at least one merged image subclass to replace the at least one image subclass to be merged and the at least one other image subclass used for merging in the original image subclass set, thereby updating the image subclass set accordingly.

[0155] In some embodiments, the server can... One subclass to be merged is identified from the image subclasses and merged with the other image subclasses, thereby reducing the set of image subclasses to a smaller number. indivual.

[0156] S103_4. Calculate the merged cluster centers of the merged image subclasses and update the cluster center set using the merged cluster centers.

[0157] In this embodiment of the application, the server can recalculate the cluster centers of the merged image set as the merged cluster centers, and use the merged cluster centers to replace the original set of cluster centers, such as... The cluster centers to be merged in each cluster center are compared with other cluster centers used for merging to update the cluster center set.

[0158] Understandably, by examining and merging the clustering results, image subclasses with unsatisfactory clustering effects can be identified from the original clustering results. These subclasses can then be optimized and merged with neighboring image subclasses, reducing the number of cluster centers and avoiding unnecessary classifications, thereby further improving the accuracy of image clustering.

[0159] In some embodiments, see Figure 10 , Figure 10 This is an optional flowchart illustrating an image clustering method provided in an embodiment of this application, based on... Figures 4-9 After S103 or S103_4, S105-S107 can be executed, which will be explained in conjunction with each step.

[0160] S105. Obtain the image to be retrieved and the retrieval features corresponding to the image to be retrieved.

[0161] In this embodiment of the application, the server can obtain the image to be retrieved and extract the image features used for retrieval from the image to be retrieved as retrieval features.

[0162] Here, the retrieval features can be image features of the corresponding attribute dimensions extracted according to the actual retrieval business needs. For example, they can be semantic embedding features or feature data of other attribute dimensions. The specific selection depends on the actual situation, and this application embodiment does not limit them.

[0163] S106. Perform cluster comparison between the search features and the set of cluster centers to determine the search cluster centers that match the search features, and take the image subclass corresponding to the search cluster center in the image subclass set as the target image subclass.

[0164] In this embodiment of the application, due to the cluster center set, such as Each cluster center is obtained by clustering the image features of the image to be processed. It represents the centroid of the feature of the subclass of the image to which the cluster center belongs. The server can then compare the retrieved features with... The features of each cluster center are compared, the matching degree between the search feature and each cluster center is calculated, and the cluster center with the highest matching degree is taken as the search cluster center that matches the search feature. Then, the image subclass corresponding to the search cluster center in the image subclass set is taken as the target image subclass.

[0165] S107. Based on the retrieval features, determine the target image that matches the image to be retrieved in the target image subclass.

[0166] In this embodiment of the application, the server can perform further feature matching on the images contained in the target image subclass based on the search features, determine the target image that matches the image to be searched, and use the target image as the search result, thereby completing the search of the image to be searched.

[0167] Understandably, due to Each cluster center is pre-classified from the entire image set corresponding to the image retrieval range using semantic information. This allows images with similar semantics to be placed in the same or similar buckets for better retrieval recall, making the retrieval results more consistent with expectations and improving the accuracy of image retrieval. Furthermore, the number of images in each image subclass obtained in this embodiment is more balanced, avoiding the situation where all images cluster into the same bucket when there is a large amount of header data. This optimizes the problem of excessive recall caused by a large number of clustering results during retrieval, which instantly increases the subsequent computational pressure by several orders of magnitude, thus improving the efficiency of image retrieval.

[0168] Below, we will combine Figure 11 This illustrates an exemplary application of the embodiments of this application in a real-world application scenario.

[0169] like Figure 11As shown, if the current clustering method is used to directly perform k-means clustering for the massive number of images stored in the image library, the semantics of the resulting clusters from cluster 1 to cluster C will be unclear. For example, if cluster 1 contains images of both "dog" and "chicken", the clustering result will be inaccurate, which will reduce the accuracy and efficiency of image retrieval when performing image retrieval based on the clustering results.

[0170] For the same image data in the image library, when using the image clustering method provided in this application embodiment, the server can, in the semantic inference and deep learning semantic cluster partitioning stage, utilize the classification and recognition model obtained by self-supervised collaborative training based on the labeled semantic classification results and labeled cluster centers, to perform semantic inference and classification prediction on the image set in the image library based on the semantic information of the images, obtaining semantic subclusters 1 to N, each semantic subcluster corresponding to a type of semantic information. The training process of the classification and recognition model is consistent with the descriptions in S001-S007, and will not be repeated here. In the intra-cluster balanced clustering stage, the server can, within each obtained semantic subcluster, according to the method in S1021-S1022, perform clustering calculations on the images contained within each semantic subcluster based on feature expression data beyond the semantic information dimension, such as the contour, texture, color, and shape of the main object, to obtain the semantic subcluster corresponding to each semantic subcluster. There are several subclasses. Among them, This value is obtained by multiplying the number of sub-cluster images in each semantic sub-cluster by the proportion of cluster centers. For example, for semantic sub-cluster 2 with the semantic information "dog," which contains multiple images with the semantic information "dog," the server can calculate the number of cluster centers corresponding to semantic sub-cluster 2 as k. Then, based on the visual feature data of the color and shape attributes contained in each image with the semantic information "dog" within semantic sub-cluster 2, clustering is performed, grouping images of "dogs" with similar colors and shapes into one sub-class, resulting in k sub-classes corresponding to semantic sub-cluster 2, i.e., sub-class 1 - sub-class k, and k cluster centers corresponding to the k sub-classes. The server performs the same intra-cluster balanced clustering process on each semantic sub-cluster to obtain the number of cluster centers corresponding to each semantic sub-cluster. Each subclass, summing all subclasses yields Each subclass. During the clustering and merging phase, the server can be based on... Each subclass corresponds to 1. Cluster centers are calculated for each image in the image database. The distance between each cluster center in the given clusters is used to determine the cluster center corresponding to the minimum distance for each image, and this clustering result is then updated for each image. The server then groups images with the same cluster center into the same bucket, resulting in... Each image is divided into buckets. For example, for k cluster centers corresponding to k subclasses, the image clustering results corresponding to the k cluster centers are updated by calculating the minimum distance from each image in the entire image library to the k cluster centers, resulting in k buckets. Here, the images contained in the k buckets may differ from the images contained in the corresponding k subclasses. The server completes the processing of all... The clustering of cluster centers is merged to obtain After dividing into individual buckets, you can The binning results corresponding to each bin are saved and used as the basis for subsequent image retrieval.

[0171] It is understood that the image clustering method provided in this application embodiment achieves semantic-based balanced clustering, thereby improving the accuracy and efficiency of image clustering.

[0172] The following continues to describe the exemplary structure of the image clustering device 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 3 As shown, the software modules stored in the image clustering device 455 of the memory 450 may include:

[0173] The classification and recognition module 4551 is used to acquire the semantic information of each image to be processed in the image set to be processed, and to divide the image set to be processed into semantic clusters based on the semantic information of each image to be processed, thereby obtaining n semantic sub-clusters; where n is a positive integer greater than or equal to 1.

[0174] Intra-cluster clustering module 4552 is used to perform image clustering within each of the n semantic sub-clusters to obtain the image clustering corresponding to each semantic sub-cluster. The set of cluster centers corresponding to the n semantic subclusters is obtained by using i as cluster centers; where i is a positive integer greater than or equal to 1 and less than or equal to n; It is a positive integer greater than or equal to 1;

[0175] Inter-cluster clustering module 4553 is used to cluster the image set to be processed based on the cluster center set to obtain an image subclass corresponding to each cluster center in the cluster center set; based on the image subclass corresponding to each cluster center, an image subclass set is obtained; the image subclass set includes the image subclass corresponding to each cluster center in the cluster center set; the image subclass set is used for image retrieval.

[0176] In some embodiments, the classification and recognition module 4551 is obtained by performing self-supervised co-training on the initial classification and recognition model using the labeled semantic classification results and labeled cluster centers corresponding to the sample image set; the labeled cluster centers are obtained by clustering the sample embedding feature vectors of the sample images in the sample image set. The classification and recognition module 4551 is used to extract features from each image to be processed to obtain an embedded feature vector; the embedded feature vector contains the semantic information; based on the embedded feature vector and n preset semantic classifications, each image to be processed is classified to obtain the n semantic subclusters corresponding to the image set to be processed, and each semantic subcluster corresponds to one preset semantic type among the n preset semantic classifications.

[0177] In some embodiments, the image clustering device 455 further includes a model training module, which is used to extract sample embedding feature vectors from each sample image in the sample image set through the initial classification and recognition model before extracting features from each image to be processed using the classification and recognition model to obtain embedded feature vectors; each sample image contains the labeled semantic classification result; based on a preset number of centers and the sample embedding feature vectors, each sample image is clustered to obtain a preset number of sample cluster centers, and the sample cluster center with the smallest distance to the sample embedding feature vector is taken as the labeled cluster center corresponding to each sample image; in the current round of training, based on the sample embedding feature vectors, each sample image is classified and predicted to obtain the semantic classification corresponding to each sample image. The prediction results are used to obtain a set of sample semantic subclusters for each round of training. In each sample semantic subcluster, clustering prediction is performed based on the sample embedding feature vector of each sample image to obtain the predicted cluster center corresponding to each sample image. The semantic classification prediction results are compared with the labeled semantic classification results to obtain the classification loss. The predicted cluster centers are compared with the labeled cluster centers to obtain the equilibrium loss. The equilibrium loss characterizes the distribution difference between the predicted cluster centers and the labeled cluster centers. The classification loss and the equilibrium loss are used as the total loss for the current round of training. The model parameters of the initial classification and recognition model are updated based on the total loss, and the next round of training is performed based on the updated model parameters until the total loss is less than a preset training threshold, thus obtaining the classification and recognition model.

[0178] In some embodiments, the intra-cluster clustering module 4552 is further configured to, for each semantic sub-cluster, use the product of the number of sub-cluster images in each semantic sub-cluster and the proportion of cluster centers as... The number of cluster centers; the cluster center ratio is the ratio of the total number of images to be processed in the image set to the preset total number of cluster centers; the sub-cluster image is the image to be processed contained in each semantic sub-cluster; based on the The number of cluster centers and the image features of each sub-cluster image are used to cluster the sub-cluster images within each semantic sub-cluster to obtain the... Cluster centers.

[0179] In some embodiments, the inter-cluster clustering module 4553 is further configured to calculate the distance from each image to be processed to each cluster center in the cluster center set; take the cluster center corresponding to the minimum distance as the target cluster center corresponding to each image to be processed; divide the images to be processed corresponding to the same target cluster center into an image subclass, thereby obtaining the image subclass corresponding to each cluster center.

[0180] In some embodiments, the inter-cluster clustering module 4553 is further configured to: after obtaining an image subclass set based on the image subclass corresponding to each cluster center, calculate the number of images contained in each image subclass; designate image subclasses with fewer than a preset number threshold as subclasses to be merged, calculate the center distance from the cluster center to be merged of the subclass to be merged to at least one other cluster center in the cluster center set; merge the subclass to be merged with the image subclasses whose center distance is less than a preset distance threshold to obtain merged image subclasses, update the image subclass set using the merged image subclasses; calculate the merged cluster center of the merged image subclasses, and update the cluster center set using the merged cluster center.

[0181] In some embodiments, the image clustering device 455 further includes a retrieval module, which is used to: after obtaining an image subclass set based on the image subclass corresponding to each cluster center, acquire the image to be retrieved and the retrieval features corresponding to the image to be retrieved; compare the retrieval features with the cluster center set to determine the retrieval cluster center that matches the retrieval features; take the image subclass corresponding to the retrieval cluster center in the image subclass set as the target image subclass; and determine the target image that matches the image to be retrieved in the target image subclass according to the retrieval features.

[0182] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0183] This application provides a computer program product or computer program that includes computer instructions stored in a readable storage medium, i.e., a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image clustering method described in this application embodiment.

[0184] This application provides a readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the method provided in this application, for example... Figure 4-6 ,as well as Figure 8-11 The method shown in the figure.

[0185] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0186] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0187] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0188] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0189] In summary, through the embodiments of this application, before performing image clustering, the entire set of images to be processed is pre-clustered using semantic information. This allows images with similar semantic content to be grouped into the same category, with semantic similarity within each semantic sub-cluster, improving the accuracy of image clustering. Thus, when performing retrieval based on the image clustering results, samples with more similar semantics can be retrieved as samples in the same or similar buckets, resulting in retrieval results that better meet expectations. Furthermore, by clustering within each semantic sub-cluster, the semantic content of each sub-cluster is obtained. Identify cluster centers, and thus obtain all clusters. Using cluster centers can reduce the instability and uncertainty caused by randomly initializing cluster centers on the entire dataset, thus improving the accuracy of image clustering. Furthermore, clustering can complete each clustering operation within limited overhead, enabling efficient clustering of large-scale data samples with limited resources, thereby improving the efficiency of image clustering. When training the classification and recognition model for semantic cluster partitioning, the server can perform feature reprojection on the sample semantic sub-clusters, achieving a gradual process of cluster balancing during model training. This ensures that the trained classification and recognition model achieves a balanced distribution within each semantic sub-cluster, improving the balance of image clustering within semantic sub-clusters. Since each semantic sub-cluster is obtained by pre-clustering the image using semantic information, it ultimately achieves a balance between internal and external semantic sub-clusters, i.e., global cluster balancing.

[0190] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An image clustering method characterized by, The method comprises: extracting an embedding feature vector containing semantic information from each of the to-be-processed images by using a classification recognition model; classifying each of the to-be-processed images based on the embedding feature vector and n preset semantic classifications, to obtain n semantic sub-clusters corresponding to the to-be-processed image set, each semantic sub-cluster corresponding to one of the n preset semantic classifications; wherein n is a positive integer greater than or equal to 1; the classification recognition model is obtained through self-supervised collaborative training as follows: comparing a semantic classification prediction result with a labeled semantic classification result to obtain a classification loss; comparing a predicted clustering center with a labeled clustering center to obtain an equilibrium loss; the equilibrium loss represents a distribution difference between the predicted clustering center and the labeled clustering center; taking the classification loss and the equilibrium loss as a total loss of the current round of training, updating model parameters of an initial classification recognition model according to the total loss, and performing the next round of training based on the updated model parameters, until the total loss is less than a preset training threshold, to obtain the classification recognition model; Image clustering is performed within each of the n semantic subclusters to obtain the corresponding semantic subclusters. The set of cluster centers corresponding to the n semantic subclusters is obtained by using i as cluster centers; where i is a positive integer greater than or equal to 1 and less than or equal to n; It is a positive integer greater than or equal to 1; performing clustering on the to-be-processed image set based on the set of clustering centers, to obtain an image subclass corresponding to each of the set of clustering centers; obtaining an image subclass set based on the image subclass corresponding to each of the set of clustering centers; the image subclass set comprises the image subclass corresponding to each of the set of clustering centers; and the image subclass set is used for image retrieval.

2. The method of claim 1, wherein, The method further comprises: extracting an embedding feature vector from each of the to-be-processed images by using a classification recognition model; the embedding feature vector contains semantic information; classifying each of the to-be-processed images based on the embedding feature vector and n preset semantic classifications, to obtain n semantic sub-clusters corresponding to the to-be-processed image set, each semantic sub-cluster corresponding to one of the n preset semantic classifications; the classification recognition model is obtained through self-supervised collaborative training of an initial classification recognition model based on labeled semantic classification results and labeled clustering centers corresponding to a sample image set; the labeled clustering centers are obtained by clustering sample embedding feature vectors of sample images in the sample image set.

3. The method of claim 2, wherein, Before the step of extracting an embedding feature vector from each of the to-be-processed images by using a classification recognition model, the method further comprises: extracting a sample embedding feature vector from each of the sample images in the sample image set by using the initial classification recognition model; each of the sample images contains the labeled semantic classification result; performing clustering on each of the sample images based on a preset center number and the sample embedding feature vector, to obtain a preset center number of sample clustering centers, and taking a sample clustering center with the smallest distance from the sample embedding feature vector as a labeled clustering center corresponding to each of the sample images; in a current round of training, performing semantic classification prediction on each of the sample images based on the sample embedding feature vector, to obtain a semantic classification prediction result corresponding to each of the sample images, and then obtaining a sample semantic sub-cluster set of each round of training based on the semantic classification prediction result. In each sample semantic sub-cluster of the sample semantic sub-cluster set, a clustering prediction is performed based on a sample embedding feature vector of each sample image, to obtain a predicted clustering center corresponding to the each sample image; The semantic classification prediction result is compared with the labeled semantic classification result, to obtain a classification loss; The predicted clustering center is compared with the labeled clustering center, to obtain an equilibrium loss; the equilibrium loss represents a distribution difference between the predicted clustering center and the labeled clustering center; The classification loss and the equilibrium loss are taken as a total loss of the current round of training, and model parameters of the initial classification and recognition model are updated according to the total loss, and the next round of training is performed based on the updated model parameters, until the total loss is less than a preset training threshold, to obtain the classification and recognition model.

4. The method according to any one of claims 1 to 3, characterized in that, The image clustering in each semantic sub-cluster of the n semantic sub-clusters obtains a corresponding cluster center of each semantic sub-cluster, comprising: For each semantic sub-cluster, a product of a number of sub-cluster images in the each semantic sub-cluster and a cluster center ratio is taken as a cluster center number; the cluster center ratio is a ratio of a total number of to-be-processed images in the to-be-processed image set to a preset total number of cluster centers; the sub-cluster image is a to-be-processed image contained in the each semantic sub-cluster; Based on the number of cluster centers and the image features of each sub-cluster image, clustering of the sub-cluster images is performed within each semantic sub-cluster to obtain the cluster centers. ​​ 5. The method of claim 4, wherein, The method further includes the following steps after the image subclass set is obtained based on the image subclass corresponding to each clustering center: A distance of each image to be processed to each clustering center in the clustering center set is calculated; A clustering center corresponding to a minimum distance is taken as a target clustering center corresponding to the each image to be processed; Images to be processed corresponding to the same target clustering center are divided into an image subclass, to obtain the image subclass corresponding to each clustering center.

6. The method of claim 5, wherein, After the image subclass set is obtained based on the image subclass corresponding to each clustering center, the method further includes the following steps: A number of images included in each image subclass is calculated; An image subclass with a number of images less than a preset number threshold is taken as a to-be-merged subclass, and at least one center distance from a to-be-merged clustering center of the to-be-merged subclass to other clustering centers in the clustering center set is calculated; The to-be-merged subclass and an image subclass with a center distance less than a preset distance threshold are merged, to obtain a merged image subclass, and the image subclass set is updated using the merged image subclass; A merged clustering center of the merged image subclass is calculated, and the clustering center set is updated using the merged clustering center.

7. The method of claim 1, wherein, After the image subclass set is obtained based on the image subclass corresponding to each clustering center, the method further includes the following steps: A to-be-searched image and a search feature corresponding to the to-be-searched image are obtained; The search feature is compared with the clustering center set, to determine a search clustering center matched with the search feature, and an image subclass corresponding to the search clustering center in the image subclass set is taken as a target image subclass; The to-be-searched image matched with the target image subclass is determined according to the search feature.

8. An image clustering apparatus characterized by comprising: The method includes the following steps: A classification and recognition module is configured to extract an embedding feature vector containing semantic information from each image to be processed; Each image to be processed is classified based on the embedding feature vector and n preset semantic classifications, to obtain n semantic sub-clusters corresponding to the image set to be processed, each semantic sub-cluster corresponding to one of the n preset semantic types; wherein n is a positive integer greater than or equal to 1. The model training module is configured to perform the following self-supervised collaborative training to obtain the classification and recognition model: comparing semantic classification prediction results with labeled semantic classification results to obtain a classification loss; comparing predicted clustering centers with labeled clustering centers to obtain an equilibrium loss; the equilibrium loss represents a distribution difference between the predicted clustering centers and the labeled clustering centers; taking the classification loss and the equilibrium loss as a total loss of current round training, updating model parameters of an initial classification and recognition model according to the total loss, and performing next round training based on the updated model parameters until the total loss is less than a preset training threshold, thereby obtaining the classification and recognition model. a cluster module in the cluster, configured to perform image clustering in each semantic sub-cluster of the n semantic sub-clusters, to obtain a corresponding cluster center of each semantic sub-cluster, and further to obtain a cluster center set corresponding to the n semantic sub-clusters; wherein i is a positive integer greater than or equal to 1 and less than or equal to n; and the is a positive integer greater than or equal to 1. The inter-cluster clustering module is configured to perform clustering on the set of images to be processed based on the set of clustering centers, to obtain an image sub-class corresponding to each clustering center in the set of clustering centers; and obtain a set of image sub-classes based on the image sub-class corresponding to each clustering center, wherein the set of image sub-classes includes the image sub-class corresponding to each clustering center in the set of clustering centers, and the set of image sub-classes is used for image retrieval.

9. The apparatus of claim 8, wherein, The model training module is further configured to extract a sample embedding feature vector from each sample image in a set of sample images by using the initial classification and recognition model, wherein each sample image contains the labeled semantic classification result; perform clustering on each sample image based on a preset center number and the sample embedding feature vector, to obtain a preset number of sample clustering centers, and take a sample clustering center with the smallest distance from the sample embedding feature vector as a labeled clustering center corresponding to each sample image. During the current round of training, perform semantic classification prediction on each sample image based on the sample embedding feature vector, to obtain a semantic classification prediction result corresponding to each sample image, and then obtain a set of sample semantic sub-clusters for each round of training based on the semantic classification prediction result; and in each sample semantic sub-cluster in the set of sample semantic sub-clusters, perform clustering prediction based on the sample embedding feature vector of each sample image, to obtain a predicted clustering center corresponding to each sample image.

10. The apparatus of claim 8, wherein, The cluster module in the cluster is further configured to, for each semantic sub-cluster, take the product of the number of sub-cluster images in the semantic sub-cluster and a cluster center ratio as a number of cluster centers; the cluster center ratio is a ratio of a total number of images to be processed in the set of images to be processed to a preset total number of cluster centers. The sub-cluster image is an image to be processed contained in each semantic sub-cluster; based on the number of cluster centers and the image features of each sub-cluster image, clustering of the sub-cluster images is performed within each semantic sub-cluster to obtain the cluster centers. The cluster centers.​ 11. An image clustering apparatus, characterized by comprising: The memory is configured to store executable instructions. The processor is configured to execute the executable instructions stored in the memory to implement the method of any one of claims 1 to 7. The memory stores executable instructions configured to be executed by the processor to implement the method of any one of claims 1 to 7.

12. A computer-readable storage medium, characterized in that, The computer instructions are executed by the processor to implement the method of any one of claims 1 to 7.

13. A computer program product comprising computer instructions, characterized in that, ​

Images