Clustering method, clustering model training method and related device

By introducing cascading codebook comparison module and hierarchical clustering of differential features in the clustering algorithm, the problem of computing resource consumption and clustering effect under the diversification of massive data and sample distribution is solved, and efficient and high-purity clustering is achieved Effect.

CN118964996AActive Publication Date: 2024-11-15BEIJING ZITIAO NETWORK TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411060484.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-15
Estimated Expiration
2044-08-02

AI Technical Summary

Technical Problem

The existing clustering algorithm consumes a lot of computing resources in scenarios with diversified massive data and sample distributions, and the clustering effect of long-tail data is poor, making it difficult to achieve efficient and high-purity clustering.

Method used

The cascading codebook comparison module based on multimodal pretrained model is used for clustering, and the N-layer codebook space is constructed through differential feature hierarchical clustering, and the cluster center is optimized to achieve high-purity aggregation.

Benefits of technology

The accuracy and generalization ability of the clustering method are improved, and high-purity aggregation can be achieved with sparse sample sizes, reducing computing resource consumption, and improving clustering effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964996A_ABST
    Figure CN118964996A_ABST
Patent Text Reader

Abstract

The invention provides a clustering method, a clustering model training method and a related device. The clustering method is applied to a clustering model, the clustering model comprises N cascaded codebook comparison modules, each level of codebook comparison module corresponds to one codebook in N layers of codebook spaces, and the clustering method comprises the following steps: inputting multi-modal representation information output by a multi-modal pre-training model into the clustering model; in each codebook comparison module of the clustering model, respectively determining a codeword closest to an input feature in a corresponding codebook, taking the codeword as a codeword corresponding to the codebook comparison module, and taking a difference feature between the input feature and the codeword as an input feature of a next-level codebook comparison module; and determining an invisible classification label corresponding to the multi-modal representation information based on the code words corresponding to the N cascaded codebook comparison modules.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a clustering method, a clustering model training method, and related devices. Background Art

[0002] In the scenario of general content understanding, the evolution of multimodal pre-training technology is getting faster and faster. The multimodal representation information output by mainstream training paradigms such as the Contrastive Language-Image Pre-training (CLIP) model is becoming more and more dense. The similarity between multiple samples can usually be characterized by the cosine distance between multimodal features. Therefore, the industry has performed content clustering of massive business data based on feature cosine similarity to meet the needs of business construction. Typical methods are traditional K-means clustering (K-means) and density clustering (Density-Based Spatial Clustering of Applications with Noise, DBSCAN) and other clustering algorithms. These methods can achieve very generalized results in scenarios with less data and uniform sample distribution. However, with the evolution of massive data and the diversification of sample distribution, the disadvantages of this type of clustering algorithm are exposed. The one-time clustering solution for the entire data consumes a lot of CPU computing resources and requires parallel operations such as pre-stratification. At the same time, a considerable portion of long-tail data will be clustered into the same category at a coarse granularity. The clustering effect of long-tail samples is less distinguishable, which is not conducive to the generalization of clustering algorithms. Therefore, how to perform efficient clustering under the premise of massive data and diversified sample distribution has become a current research hotspot. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure provide a clustering method, a clustering model training method and related devices, which can achieve very good high-purity aggregation even for a class with very sparse sample size.

[0004] The clustering method described in the embodiment of the present disclosure is applied to a clustering model, wherein the above-mentioned clustering model includes N cascaded codebook comparison modules and each level of codebook comparison module corresponds to a codebook in an N-layer codebook space, and N is an integer greater than or equal to 1; wherein the clustering method includes: inputting the multimodal representation information output by the multimodal pre-training model into the clustering model; in each codebook comparison module of the clustering model, respectively determining the codeword closest to the input feature in its corresponding codebook, using it as the codeword corresponding to the codebook comparison module, and using the difference feature between the input feature and the codeword as the input feature of the next level of codebook comparison module; and determining the invisible classification label corresponding to the multimodal representation information based on the codeword corresponding to the N cascaded codebook comparison modules.

[0005] In an embodiment of the present disclosure, in each codebook comparison module of the clustering model, respectively determining a codeword in its corresponding codebook that is closest to the input feature, using it as the codeword corresponding to the codebook comparison module, and using the difference feature between the input feature and the codeword as the input feature of the next-level codebook comparison module includes: in the first-level codebook comparison module, comparing the input multimodal representation information with multiple codewords in the first-level codebook corresponding to the first-level codebook comparison module, determining the codeword that is closest to the multimodal representation information as the codeword corresponding to the first-level codebook comparison module, performing a difference operation on the multimodal representation information and the first-level codeword to obtain a difference feature, and inputting the difference feature into the next-level codebook comparison module. A second-level codebook comparison module; in the i-th level codebook comparison module, where i is greater than 1 and less than N, the input difference feature is compared with a plurality of codewords in the i-th level codebook corresponding to the i-th level codebook comparison module, and the codeword closest to the difference feature is determined as the codeword corresponding to the i-th level codebook comparison module, and the difference feature is performed on the difference feature and the i-th level codeword to obtain the difference feature, and the difference feature is input into the next-level codebook comparison module; and in the N-th level codebook comparison module, the input difference feature is compared with a plurality of codewords in the N-th level codebook corresponding to the N-th level codebook comparison module, and the codeword closest to the difference feature is determined as the codeword corresponding to the N-th level codebook comparison module.

[0006] In an embodiment of the present disclosure, the above difference operation includes: determining the difference feature based on the following expression: j =C j -CW j ; Wherein, j represents the j-th level codebook comparison module; D j represents the difference feature determined by the j-th level codebook comparison module; C j represents the input feature of the j-th level codebook comparison module; and CW j Represents the codeword corresponding to the j-th level codebook comparison module.

[0007] In an embodiment of the present disclosure, the step of determining the invisible classification label corresponding to the multimodal representation information based on the codewords corresponding to the N cascaded codebook comparison modules includes: determining the invisible classification label corresponding to the multimodal representation information based on the following expression: Wherein, T represents the invisible classification label corresponding to the multimodal representation information; j represents the j-th level codebook comparison module; Represents the codeword CW corresponding to the j-th level codebook comparison module j is the sequence number in the j-th level codebook; and M represents the number of codewords in each codebook.

[0008] In an embodiment of the present disclosure, each of the N cascaded codebooks contains M codewords, where M is an integer greater than 1; determining the invisible classification label corresponding to the multimodal representation information based on the codewords corresponding to the N cascaded codebook comparison modules includes: selecting one codeword from each of the N cascaded codebooks for combination, to obtain a total of M*N codeword combinations; encoding the M*N codeword combinations to obtain identifiers of the M*N codeword combinations; determining the codeword combination corresponding to the multimodal representation information based on the codewords corresponding to the N cascaded codebook comparison modules; and determining the identifier of the codeword combination corresponding to the multimodal representation information as the invisible classification label corresponding to the multimodal representation information.

[0009] The embodiment of the present disclosure also discloses a clustering model training method, which is applied to the clustering model, wherein the clustering model includes N cascaded codebook comparison modules, N is an integer greater than or equal to 1; wherein the clustering model training method includes: pre-constructing N layers of codebook space; wherein the N layers of codebook space correspond one-to-one to the N cascaded codebook comparison modules; for each layer of codebook space, initializing M cluster centers; wherein M is an integer greater than 1; during the training process, for each codebook comparison module of the clustering model, respectively executing: determining the cluster center closest to the input sample feature among the M cluster centers, taking it as the candidate cluster center corresponding to the codebook comparison module, recording the distance between the input sample feature and the candidate cluster center as the candidate distance corresponding to the codebook comparison module; and recording the difference feature between the input sample feature and the candidate cluster center as the input feature of the next level codebook comparison module; summing the candidate distances corresponding to the N cascaded codebook comparison modules to obtain the similarity loss; and optimizing the M cluster centers of the N cascaded codebook comparison modules respectively with the minimum similarity loss as the first optimization target to obtain the optimized M cluster centers; performing codebook restoration and feature reconstruction based on the candidate cluster centers corresponding to the N cascaded codebook comparison modules to obtain the reconstructed features; taking the difference between the input sample feature and the reconstructed feature as the reconstruction loss; and optimizing the M cluster centers of the N cascaded codebook comparison modules respectively with the minimum reconstruction loss as the second optimization target to obtain the optimized M cluster centers; and when the training end condition is met, respectively using the M cluster centers corresponding to the N cascaded codebook comparison modules as the M codewords of the codebook corresponding to the N cascaded codebook comparison modules.

[0010] The embodiment of the present disclosure also discloses a clustering model, including: N cascaded codebook comparison modules; wherein N is an integer greater than or equal to 1; and each level of codebook comparison modules in the N cascaded codebook comparison modules corresponds to a codebook in the N-layer codebook space, respectively, and is used to compare input features with each codeword in the corresponding codebook, determine the codeword closest to the input feature, use it as the codeword corresponding to itself, and use the difference feature between the input feature and the codeword as the input feature of the next level of codebook comparison module.

[0011] An embodiment of the present disclosure also discloses a clustering device, comprising: an input interface, a clustering model as described in claim 6, and a classification label determination module; wherein the input interface is used to receive multimodal representation information output by a multimodal pre-training model, and input the received multimodal representation information into the clustering model; each codebook comparison module of the clustering model respectively determines the codeword closest to the input feature in its corresponding codebook, uses it as the codeword corresponding to the codebook comparison module, and uses the difference feature between the input feature and the codeword as the input feature of the next-level codebook comparison module; and the classification label determination module is used to determine the invisible classification label corresponding to the multimodal representation information based on the codeword corresponding to the N cascaded codebook comparison modules.

[0012] In addition, an embodiment of the present disclosure further provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above clustering method when executing the program.

[0013] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the above clustering method.

[0014] An embodiment of the present disclosure further provides a computer program product, including computer program instructions. When the computer program instructions are executed on a computer, the computer is enabled to execute the above clustering method.

[0015] The clustering method, clustering model training method and related devices described in the embodiments of the present disclosure propose a discretized clustering scheme based on deep learning, which hierarchically clusters the attributes of different dimensions of the input information-intensive multimodal representation information by introducing a residual model structure, thereby achieving the goal of fully mapping the dense multimodal representation information to a feature space with high discrimination and clear content hierarchy. The above scheme can achieve very good high-purity aggregation even for a class with very sparse sample size, thereby greatly improving the accuracy and generalization ability of the clustering method. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 The structure of the clustering model described in some embodiments of the present disclosure is shown.

[0018] Figure 2 The implementation process of the clustering method described in some embodiments of the present disclosure is shown.

[0019] Figure 3 The implementation process of the clustering model training method described in some embodiments of the present disclosure is shown.

[0020] Figure 4 The internal structure of the clustering device described in some embodiments of the present disclosure is shown.

[0021] Figure 5 A more specific schematic diagram of the hardware structure of an electronic device described in some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0022] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0023] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0024] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0025] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0026] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0028] As mentioned above, the current clustering method based on multimodal pre-training can achieve very generalized results in scenarios with less data and uniform sample distribution. However, with the evolution of massive data and the diversification of sample distribution, the one-time clustering solution for all data consumes a lot of CPU computing resources and requires parallel operations such as pre-stratification. In addition, a considerable portion of long-tail data will be clustered into the same category at a coarse granularity, and the clustering effect of long-tail samples is less distinguishable.

[0029] Specifically, in the task of content understanding, various modal carrier information that can be collected is usually utilized in the modeling stage, including video, voice, optical recognition (OCR), comments, etc., and a very rich and dense feature space can be represented in the pre-training stage. However, the output high-dimensional features are not sufficient to directly characterize the aggregation of massive samples at the content level. The subsequent optimization process usually introduces a clustering algorithm, performs intensive similarity calculations on high-dimensional features, and constructs circle relationships between data. For massive samples with highly similar content, clustering algorithms can usually achieve very good generalization effects, but for very sparse samples, they are usually clustered together with data with large content differences, affecting the downstream data confidence.

[0030] In order to solve the above technical problems, an embodiment of the present disclosure provides a clustering model, which can perform clustering based on the information-dense multimodal representation information output by the multimodal pre-training model, and output an invisible classification label corresponding to the input multimodal representation information.

[0031] Figure 1 The structure of the clustering model 100 described in some embodiments of the present disclosure is shown. Figure 1 As shown, the clustering model 100 described in the embodiment of the present disclosure includes: N cascaded codebook comparison modules; wherein N is an integer greater than or equal to 1.

[0032] In an embodiment of the present disclosure, the above-mentioned N cascaded codebook comparison modules have basically the same structure. Each level of codebook comparison modules in the above-mentioned N cascaded codebook comparison modules corresponds to a codebook in the N-layer codebook space. The N codebooks corresponding to the above-mentioned N cascaded codebook comparison modules constitute an N-layer codebook space. Among them, each layer of codebook space characterizes the attributes of the sample in one dimension. In this way, the N-layer codebook space can hierarchically cluster the input multimodal representation information from the attributes of N dimensions, thereby achieving the goal of fully mapping the dense multimodal representation information to a feature space with high discrimination and clear content hierarchy, thereby improving the accuracy and generalization ability of the clustering method.

[0033] In an embodiment of the present disclosure, each level of codebook comparison modules in the above-mentioned N cascaded codebook comparison modules can be used to compare its own input feature Z with each codeword in its corresponding codebook, determine the codeword closest to the input feature, output it as its own corresponding codeword, and use the difference feature between the input feature and the codeword as the input feature of the next level of codebook comparison module.

[0034] Specifically, for the first-level codebook comparison module, the first-level codebook comparison module compares the input multimodal representation information with multiple codewords in the first-level codebook corresponding to the first-level codebook comparison module, determines the codeword closest to the multimodal representation information as the codeword corresponding to the first-level codebook comparison module, performs a difference operation on the multimodal representation information and the first-level codeword to obtain a difference feature, and inputs the difference feature into the second-level codebook comparison module.

[0035] For the i-th level codebook comparison module, where i is greater than 1 and less than N, the i-th level codebook comparison module compares the input difference feature with multiple codewords in the i-th level codebook corresponding to the i-th level codebook comparison module, determines the codeword closest to the difference feature as the codeword corresponding to the i-th level codebook comparison module, performs a difference operation on the difference feature and the i-th level codeword to obtain the difference feature, and inputs the difference feature into the next level codebook comparison module.

[0036] As for the N-th level codebook comparison module, the N-th level codebook comparison module compares the input difference feature with multiple codewords in the N-th level codebook corresponding to the N-th level codebook comparison module, and determines the codeword closest to the difference feature as the codeword corresponding to the N-th level codebook comparison module.

[0037] For example, Figure 1The first level codebook comparison module, the second level codebook comparison module, ... and the Nth level codebook comparison module are shown in . Among them, in the first level codebook comparison module, the input feature Z is closest to codeword 1, so the first level codebook comparison module will output codeword 1, and calculate the difference feature between the input feature Z and codeword 1 as the input of the second level codebook comparison module. In the second level codebook comparison module, the input feature Z is closest to codeword 2, so the second level codebook comparison module will output codeword 2, and calculate the difference feature between the input feature Z and codeword 2 as the input of the third level codebook comparison module. .... In the Nth level codebook comparison module, the input feature Z is closest to codeword 5, so the Nth level codebook comparison module will output codeword 5. It is particularly noted that the Nth level codebook comparison module is the last level module, so there is no need to output the difference feature to the next level module.

[0038] It should be noted that, in the embodiments of the present disclosure, the above distance may refer to any one of various similarity distances, for example, Euclidean distance, cosine similarity distance, etc. The embodiments of the present disclosure do not limit the specific meaning and algorithm of the above distance.

[0039] In an embodiment of the present disclosure, the above difference operation may include: determining the difference feature based on the following expression:

[0040] D j =C j -CW j

[0041] Where, j represents the j-th level codebook comparison module; D j represents the difference feature determined by the j-th level codebook comparison module; C j represents the input feature of the j-th level codebook comparison module; and CW j Represents the codeword corresponding to the j-th level codebook comparison module.

[0042] Based on the above Figure 1 The clustering model 100 is shown, and the embodiment of the present disclosure also proposes a clustering method. Figure 2 The implementation process of the above clustering method is shown. Figure 2 As shown, the above clustering method may include the following steps:

[0043] In step 210, the multimodal representation information output by the multimodal pre-training model is input into the above clustering model.

[0044] In an embodiment of the present disclosure, the multimodal pre-training model may be a multimodal pre-training model such as a CLIP model. The embodiment of the present disclosure does not limit the specific type of the multimodal pre-training model used. As mentioned above, in the scenario of general content understanding, with the evolution of the multimodal pre-training model, the multimodal representation information output by it becomes increasingly dense. The clustering method proposed in the embodiment of the present disclosure performs clustering based on the information-intensive multimodal representation information output by the multimodal pre-training model.

[0045] In addition, the above multimodal representation information can be a fusion feature of multiple models. Furthermore, if the backend service also needs to rely on other statistical features, the required statistical values ​​can also be encoded through the encoding algorithm of numerical features, and the encoded features can be further integrated into the above multimodal representation information as the input of the above clustering model.

[0046] In step 220, in each codebook comparison module of the above clustering model, the codeword closest to the input feature in its corresponding codebook is determined, and the codeword is used as the codeword corresponding to the above codebook comparison module, and the difference feature between the above input feature and the above codeword is used as the input feature of the next level codebook comparison module.

[0047] In an embodiment of the present disclosure, the above step 220 may specifically include the following steps:

[0048] First, in the first-level codebook comparison module, the input multimodal representation information is compared with a plurality of codewords in the first-level codebook corresponding to the first-level codebook comparison module, and the codeword closest to the multimodal representation information is determined as the codeword corresponding to the first-level codebook comparison module, and a difference operation is performed between the multimodal representation information and the first-level codeword to obtain a difference feature, and the difference feature is input into the second-level codebook comparison module;

[0049] Secondly, in the i-th level codebook comparison module, where i is greater than 1 and less than N, the input difference feature is compared with a plurality of codewords in the i-th level codebook corresponding to the i-th level codebook comparison module, a codeword closest to the difference feature is determined as the codeword corresponding to the i-th level codebook comparison module, a difference operation is performed between the difference feature and the i-th level codeword to obtain a difference feature, and the difference feature is input into the next level codebook comparison module; and

[0050] Finally, in the N-th level codebook comparison module, the input difference feature is compared with multiple codewords in the N-th level codebook corresponding to the N-th level codebook comparison module, and the codeword closest to the difference feature is determined as the codeword corresponding to the N-th level codebook comparison module.

[0051] In an embodiment of the present disclosure, the above difference operation may include: determining the difference feature based on the following expression:

[0052] D j =C j -CW j

[0053] Where, j represents the j-th level codebook comparison module; D j represents the difference feature determined by the j-th level codebook comparison module; C j represents the input feature of the j-th level codebook comparison module; and CW j Represents the codeword corresponding to the j-th level codebook comparison module.

[0054] In step 230, the invisible classification label corresponding to the multimodal representation information is determined based on the codewords corresponding to the N cascaded codebook comparison modules.

[0055] In an embodiment of the present disclosure, each of the N cascaded codebooks contains M codewords, where M is an integer greater than 1. In this case, the step 230 may include the following steps:

[0056] First, select one codeword from each of the N cascaded codebooks to combine, and obtain M*N different codeword combinations in total;

[0057] Secondly, the M*N code word combinations are encoded to obtain identifications (IDs) of the M*N code word combinations;

[0058] Then, determining a codeword combination corresponding to the multimodal representation information based on the codewords corresponding to the N cascaded codebook comparison modules; and

[0059] Finally, the identifier of the codeword combination corresponding to the above multimodal representation information is determined as the invisible classification label corresponding to the multimodal representation information.

[0060] As mentioned above, the N codebooks corresponding to the N cascaded codebook comparison modules constitute an N-layer codebook space. Among them, each layer of the codebook space describes the attributes of the sample in one dimension. That is to say, by constructing the above-mentioned N-layer codebook space, the above-mentioned multimodal representation information can be classified in N dimensions respectively. In addition, each codebook in the above-mentioned N cascaded codebooks contains M codewords respectively. Among them, the M codewords in the above-mentioned one codebook represent that the above-mentioned multimodal representation information can be specifically divided into one of M categories in its corresponding dimension. Through such a setting, the final classification result will be to specifically divide the multimodal representation information into one of M*N categories from N dimensions. Therefore, after encoding the above-mentioned M*N codeword combinations and obtaining the IDs of the above-mentioned M*N codeword combinations, once the IDs of the codeword combinations corresponding to the above-mentioned multimodal representation information are determined, it can be determined which specific category of the M*N categories the above-mentioned multimodal representation information is divided into. Since the ID of the codeword combination is output in the above-mentioned method, rather than an explicit classification label, it is called an invisible classification label. The above invisible classification labels can provide a reference for the degree to which samples can be divided at the content level for a variety of downstream tasks.

[0061] In some embodiments of the present disclosure, a specific encoding method is provided. In these embodiments, the invisible classification label corresponding to the multimodal representation information can be directly determined based on the following expression:

[0062]

[0063] Wherein, T represents the invisible classification label corresponding to the multimodal representation information; j represents the j-th level codebook comparison module; represents the sequence number of the codeword corresponding to the j-th level codebook comparison module in the j-th level codebook; and M represents the number of codewords in each codebook.

[0064] It can be seen that, through the above method, the invisible classification label can be determined directly by using the sequence number of the codeword corresponding to each level of codebook comparison module in its codebook.

[0065] The above clustering models and clustering methods make full use of the quantitative mapping of multi-dimensional feature space to fully map dense multimodal representation information into a feature space with high discrimination and clear content hierarchy, and produce implicit classification labels. It can be understood that by constructing a multi-layer codebook space to perform similarity quantization of feature vectors, it can be ensured that the samples aggregated in each layer of the codebook space can be as similar as possible in different content attributes, thereby effectively mitigating the impact of sample distribution on clustering effects. Moreover, as the amount of feature information extracted by pre-training becomes more and more dense, this discretized clustering method can be unaffected by the sample size, and the feature aggregation between different samples becomes more and more refined.

[0066] Based on the above clustering model, an embodiment of the present disclosure also provides a clustering model training method. Figure 3 The implementation process of the clustering model training method described in the embodiment of the present disclosure is shown. Figure 3 As shown, the method comprises the following steps:

[0067] In step 310, an N-layer codebook space is pre-constructed.

[0068] In the embodiment of the present disclosure, the N layers of codebook space correspond one-to-one to the N cascaded codebook comparison modules, wherein each layer of codebook space may correspond to an attribute of a sample in one dimension.

[0069] In step 320, for each layer of codebook space, M cluster centers are initialized; wherein M is an integer greater than 1.

[0070] It can be understood that in the embodiments of the present disclosure, the meaning of the cluster center of each codebook is similar to the center of each classification. However, before model training, the above initialized cluster center is inaccurate. The goal of model training is to optimize and obtain suitable cluster centers, that is, to achieve the optimal clustering result of each layer of codebook.

[0071] In step 330, during the training process, the following steps are performed for each codebook comparison module of the above clustering model:

[0072] Determine the cluster center among the M cluster centers that is closest to the input sample feature;

[0073] It is used as the candidate cluster center corresponding to the above codebook comparison module;

[0074] The distance between the input sample feature and the candidate cluster center is recorded as the candidate distance corresponding to the codebook comparison module; and

[0075] The difference features between the input sample features and the candidate cluster centers are used as the input features of the next level codebook comparison module.

[0076] In the embodiments of the present disclosure, the above distance may refer to any one of various similarity distances, for example, one of the Euclidean distance, the cosine similarity distance, and the like.

[0077] In step 340, the candidate distances corresponding to the N cascaded codebook comparison modules are summed to obtain a similarity loss, and then the M cluster centers of the N cascaded codebook comparison modules are optimized respectively with the minimum similarity loss as the first optimization goal to obtain the optimized M cluster centers.

[0078] In an embodiment of the present disclosure, by summing up the candidate distances corresponding to the above-mentioned N cascaded codebook comparison modules, this is added to the training as an optimization item, and in the subsequent training process, it is expected that the constraint of this optimization item will become smaller, thereby achieving the effect of smaller distances between similar samples and larger distances between non-similar samples.

[0079] In step 350, codebook restoration and feature reconstruction are performed based on the candidate cluster centers corresponding to the N cascaded codebook comparison modules to obtain reconstructed features, and the difference between the input sample features and the reconstructed features is used as the reconstruction loss. The M cluster centers of the N cascaded codebook comparison modules are optimized respectively with the minimum reconstruction loss as the second optimization goal to obtain the optimized M cluster centers.

[0080] It should be noted that the above codebook restoration and feature reconstruction can be implemented by using existing codebook restoration and feature reconstruction methods, and the embodiments of the present disclosure do not limit the specific implementation methods of the above steps.

[0081] It can be seen that the above clustering model training method also introduces a feature mapping module after the above N cascaded codebook comparison modules to highly restore the input features, and the distance measurement between the input and output features is added to the training as an optimization item, which is expected to make the feature compression process retain as much information as possible.

[0082] In step 360, when the training end condition is met, the M cluster centers corresponding to the N cascaded codebook comparison modules are respectively used as the M codewords of the codebook corresponding to the N cascaded codebook comparison modules.

[0083] In an embodiment of the present disclosure, the above-mentioned training end condition may be one or more pre-set conditions, for example, the number of iterations exceeds a pre-set threshold, the similarity loss and / or the reconstruction loss meets the set conditions, etc.

[0084] Through the above clustering model training method, M suitable cluster centers can be determined in the above N-layer codebook space, that is, M codewords in each layer of the codebook space can be determined. Therefore, in the process of model reasoning, the codeword closest to the input feature is determined in sequence in the codebook comparison module at each level, and the multimodal representation information is classified in N dimensions in sequence, thereby achieving the goal of fully mapping the dense multimodal representation information to a feature space with high discrimination and clear content hierarchy.

[0085] The embodiment of the present disclosure also discloses a clustering device, the internal structure of which is as follows: Figure 4 As shown, it includes: an input interface 410 , the above-mentioned clustering model 420 , and a classification label determination module 430 .

[0086] The input interface is used to receive the multimodal representation information output by the multimodal pre-training model, and input the received multimodal representation information into the clustering model 420 .

[0087] The codebook comparison modules at each level of the clustering model 420 respectively determine the codeword in the corresponding codebook that is closest to the input feature, use it as the codeword corresponding to the codebook comparison module, and use the difference feature between the input feature and the codeword as the input feature of the next level of codebook comparison module.

[0088] The classification label determination module 430 is used to determine the invisible classification label corresponding to the multimodal representation information based on the codewords corresponding to the codebook comparison modules at each level.

[0089] The clustering method, clustering model training method and related devices described in the embodiments of the present disclosure propose a discretized clustering scheme based on deep learning, which hierarchically clusters the attributes of different dimensions of the input information-intensive multimodal representation information by introducing a residual model structure, thereby achieving the goal of fully mapping the dense multimodal representation information to a feature space with high discrimination and clear content hierarchy. The above scheme can achieve very good high-purity aggregation even for a class with very sparse sample size, thereby greatly improving the accuracy and generalization ability of the clustering method.

[0090] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the clustering method described in any of the above embodiments is implemented.

[0091] Figure 5 A more specific hardware structure diagram of an electronic device provided in this embodiment is shown, and the device may include: a processor 2010, a memory 2020, an input / output interface 2030, a communication interface 2040, and a bus 2050. The processor 2010, the memory 2020, the input / output interface 2030, and the communication interface 2040 are connected to each other in communication within the device through the bus 2050.

[0092] The processor 2010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0093] The memory 2020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 2020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 2020 and are called and executed by the processor 2010.

[0094] The input / output interface 2030 is used to connect input / output devices to realize information input and output. The input / output devices can be configured in the device as components, or can be externally connected to the device to provide corresponding functions. The input devices can include microphones, various sensors, etc., and the output devices can include displays, speakers, vibrators, indicator lights, etc.

[0095] The communication interface 2040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0096] The bus 2050 includes a path that transmits information between the various components of the device (eg, the processor 2010, the memory 2020, the input / output interface 2030, and the communication interface 2040).

[0097] It should be noted that, although the above device only shows the processor 2010, the memory 2020, the input / output interface 2030, the communication interface 2040, and the bus 2050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0098] The electronic device of the above embodiment is used to implement the corresponding clustering method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0099] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the clustering method described in any of the above embodiments.

[0100] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0101] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the task processing method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0102] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0103] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0104] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0105] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A clustering method, applied to a clustering model, wherein: The clustering model includes N cascaded codebook comparison modules, and each level of codebook comparison module corresponds to a codebook in an N-layer codebook space, where N is an integer greater than or equal to 1; the clustering method includes: Inputting the multimodal representation information output by the multimodal pre-training model into the clustering model; In each codebook comparison module of the clustering model, respectively determine the codeword in the corresponding codebook that is closest to the input feature, use it as the codeword corresponding to the codebook comparison module, and use the difference feature between the input feature and the codeword as the input feature of the next level codebook comparison module; and Determine an invisible classification label corresponding to the multimodal representation information based on the codewords corresponding to the N cascaded codebook comparison modules.

2. The clustering method according to claim 1, wherein: In each codebook comparison module of the clustering model, respectively determining a codeword in its corresponding codebook that is closest to the input feature, using it as the codeword corresponding to the codebook comparison module, and using the difference feature between the input feature and the codeword as the input feature of the next level codebook comparison module includes: In the first-level codebook comparison module, the input multimodal representation information is compared with a plurality of codewords in the first-level codebook corresponding to the first-level codebook comparison module, a codeword closest to the multimodal representation information is determined as the codeword corresponding to the first-level codebook comparison module, a difference operation is performed between the multimodal representation information and the first-level codeword to obtain a difference feature, and the difference feature is input into the second-level codebook comparison module; In the i-th level codebook comparison module, where i is greater than 1 and less than N, the input difference feature is compared with a plurality of codewords in the i-th level codebook corresponding to the i-th level codebook comparison module, a codeword closest to the difference feature is determined as the codeword corresponding to the i-th level codebook comparison module, a difference operation is performed on the difference feature and the i-th level codeword to obtain a difference feature, and the difference feature is input into the next level codebook comparison module; and In the N-th level codebook comparison module, the input difference feature is compared with multiple codewords in the N-th level codebook corresponding to the N-th level codebook comparison module, and the codeword closest to the difference feature is determined as the codeword corresponding to the N-th level codebook comparison module.

3. The clustering method according to claim 2, wherein: The difference operation includes: The difference feature is determined based on the following expression: D j =C j -CW j Where, j represents the j-th level codebook comparison module; D j represents the difference feature determined by the j-th level codebook comparison module; C j represents the input feature of the j-th level codebook comparison module; and CW j Represents the codeword corresponding to the j-th level codebook comparison module.

4. The clustering method according to claim 1, wherein: Determining the invisible classification label corresponding to the multimodal representation information based on the codewords corresponding to the N cascaded codebook comparison modules includes: The invisible classification label corresponding to the multimodal representation information is determined based on the following expression: Wherein, T represents the invisible classification label corresponding to the multimodal representation information; j represents the j-th level codebook comparison module; Represents the codeword CW corresponding to the j-th level codebook comparison module j is the sequence number in the j-th level codebook; and M represents the number of codewords in each codebook.

5. The clustering method according to claim 1, wherein: Each of the N cascaded codebooks contains M codewords, where M is an integer greater than 1; Determining the invisible classification label corresponding to the multimodal representation information based on the codewords corresponding to the N cascaded codebook comparison modules includes: Select one codeword from each of the N cascaded codebooks for combination, and obtain M*N codeword combinations in total; Encoding the M*N codeword combinations to obtain identifiers of the M*N codeword combinations; Determining a codeword combination corresponding to the multimodal representation information based on the codewords corresponding to the N cascaded codebook comparison modules; and The identifier of the codeword combination corresponding to the multimodal representation information is determined as the invisible classification label corresponding to the multimodal representation information.

6. A clustering model training method, applied to a clustering model, wherein: The clustering model includes N cascaded codebook comparison modules, where N is an integer greater than or equal to 1; wherein the clustering model training method includes: Pre-constructing an N-layer codebook space; wherein the N-layer codebook space corresponds one-to-one to the N cascaded codebook comparison modules; For each layer of codebook space, initialize M cluster centers; where M is an integer greater than 1; During the training process, the following steps are respectively performed for each codebook comparison module of the clustering model: determining the cluster center closest to the input sample feature among the M cluster centers, taking it as the candidate cluster center corresponding to the codebook comparison module, recording the distance between the input sample feature and the candidate cluster center as the candidate distance corresponding to the codebook comparison module; and taking the difference feature between the input sample feature and the candidate cluster center as the input feature of the next level codebook comparison module; The candidate distances corresponding to the N cascaded codebook comparison modules are summed to obtain a similarity loss; and the M cluster centers of the N cascaded codebook comparison modules are optimized respectively with the minimum similarity loss as the first optimization goal to obtain the optimized M cluster centers; Perform codebook restoration and feature reconstruction based on the candidate cluster centers corresponding to the N cascaded codebook comparison modules to obtain reconstructed features; use the difference between the input sample feature and the reconstructed feature as the reconstruction loss; and optimize the M cluster centers of the N cascaded codebook comparison modules respectively with the minimum reconstruction loss as the second optimization goal to obtain the optimized M cluster centers; and When the training end condition is met, the M cluster centers corresponding to the N cascaded codebook comparison modules are respectively used as the M codewords of the codebook corresponding to the N cascaded codebook comparison modules.

7. A clustering model comprising: N cascaded codebook comparison modules; wherein N is an integer greater than or equal to 1; and Each level of codebook comparison modules in the N cascaded codebook comparison modules corresponds to a codebook in the N-layer codebook space, and is used to compare the input features with the codewords in the corresponding codebooks, determine the codeword closest to the input features, use it as the codeword corresponding to itself, and use the difference feature between the input feature and the codeword as the input feature of the next level of codebook comparison module.

8. A clustering device, comprising: An input interface, a clustering model as claimed in claim 7, and a classification label determination module; wherein, The input interface is used to receive multimodal representation information output by the multimodal pre-training model, and input the received multimodal representation information into the clustering model; Each codebook comparison module of the clustering model determines a codeword in its corresponding codebook that is closest to the input feature, uses it as the codeword corresponding to the codebook comparison module, and uses the difference feature between the input feature and the codeword as the input feature of the next level codebook comparison module; and The classification label determination module is used to determine the invisible classification label corresponding to the multimodal representation information based on the codewords corresponding to the N cascaded codebook comparison modules.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the clustering method according to any one of claims 1 to 5 is implemented.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the clustering method according to any one of claims 1 to 5.

11. A computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the clustering method according to any one of claims 1 to 5.

Citation Information

Patent Citations

  • Method and device for identifying scene integrated by multi-feature vision codebook

    CN103366181A

  • Fast database search systems and methods

    CN110168525A

  • High-resolution remote sensing image scene multi-label classification method based on multi-packet fusion

    CN110210534A

  • System and Method for Visual Event Description and Event Analysis

    US20170103264A1

  • Database data compression method and storage device

    WO2023070424A1