Heterogeneous model-oriented multi-granularity hierarchical semantic tree construction method

Abstract

The invention discloses a heterogeneous model-oriented multi-granularity hierarchical semantic tree construction method. The method comprises the following steps of: generating an image feature vector set; forming a bottom layer of the hierarchical semantic tree according to the image and the sample label; constructing a representative semantic space according to the image feature vector and the WordNet data set based on a clustering method; using an MLP model to classify images to confirm an image center, and generating a semantic center set; the mapping relation between the image and the semantic center set noun is described as an OT problem, the mapping probability is solved with the purpose of minimizing the transmission cost, and an image pseudo tag is generated; updating the weight of the MLP model, constructing a loss function, and optimizing the MLP model; generating an optimal classification result based on the optimized MLP model, generating an image tag based on the optimal classification result by the CLIP model, and constructing an upper layer of the bottommost layer of the hierarchical semantic tree based on the image tag; and sequentially constructing other layers according to the method. The method is beneficial to maximizing the maximum potential of cooperative training of heterogeneous models.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a multi-granularity hierarchical semantic tree construction method for heterogeneous models. Background Art

[0002] Cloud-edge collaboration of large and small models has become a new trend. The existing collaborative work of large and small models ignores the role of different data granularities. However, the use of different data granularities may unleash greater potential for model collaboration. In fact, there are differences in the image recognition tasks that heterogeneous models can handle. For large models, using fine-grained labeled data for model training is conducive to more detailed distinctions and improves the ability to deeply classify images; while matching coarse-grained data will cause a waste of model capabilities. For small models, due to their own insufficient capabilities, matching fine-grained data may result in very poor accuracy performance, while relatively acceptable accuracy performance can be achieved in coarse-grained data tasks. Therefore, the mismatch between model and data granularity will lead to a waste of model capabilities or data resources. Specifically, the current methods have the following disadvantages:

[0003] 1. Ignoring the use of data granularity: Existing methods do not take into account the matching between data granularity and heterogeneous models. The matching of heterogeneous models with appropriate data granularity can give full play to the greater advantages of the collaboration between large and small models. In addition, existing data sets only have fixed granularity and do not meet the actual matching needs of heterogeneous models of different sizes. In intelligent transportation scenarios, the implementation of cloud-edge collaboration can optimize data processing through the collaboration of edge devices and cloud models. In intelligent traffic monitoring, edge devices collect traffic data in real time, match coarse-grained data with small models for rapid analysis, and fine-grained data requires large models on the cloud for in-depth analysis. The challenge in this actual scenario is how to effectively coordinate data and model capabilities of different granularities to improve overall system performance and real-time response capabilities;

[0004] 2. Single feature information: Most image clustering methods that can currently support granularity generation only explore and cluster images based on their features, and cannot distinguish between images that are visually similar but semantically different, which can easily lead to uneven construction of multi-granularity hierarchical structures and even confusion. For example, vehicle images captured by different traffic cameras in intelligent traffic monitoring are similar. Single image data cannot distinguish specific vehicles, which can easily lead to confusion between visually similar but semantically different images.

[0005] 3. It is necessary to design reasonable relevant semantics for image data to capture the deep semantics of image data and further support semantic enhancement of image clustering. In addition, simply mapping images to the nearest semantics does not improve clustering, so it is necessary to further design methods to combine images and semantics to provide guidance for image clustering, and then use semantics to enhance the accuracy of image clustering results and solve the confusion problem of visual similarity and different semantics. Summary of the invention

[0006] In view of the above problems, the present invention proposes a multi-granularity hierarchical semantic tree construction method for heterogeneous models, which can generate data granularity based on multimodal features and solve the confusion problem of different data granularity construction. In addition, by constructing a reasonable multi-granularity semantic generation tree to meet the matching requirements of heterogeneous models, the accuracy performance of large and small model training is improved, and the maximum capacity of heterogeneous models is brought into play. In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0007] A method for constructing a multi-granularity hierarchical semantic tree for heterogeneous models includes the following steps:

[0008] S1, collect images and corresponding sample labels, build a granular tree generation model including an MLP model and a CLIP model, and use the image encoder of the CLIP model to generate an image feature vector set according to the image;

[0009] S2, set the depth of the tree The corresponding leaf nodes are generated according to the images collected in step S1, and the sample labels are used as the labels of the corresponding nodes to form the first leaf node of the hierarchical semantic tree. layer, and

[0010] S3, initialize the MLP model training round Current layer And set the number of layer nodes c h ;

[0011] S4, clustering the image feature vector obtained in step S1 to obtain c h Cluster centers are formed, and the Faiss Library method is used to select several nouns closest to each cluster center based on the cluster centers and WordNet dataset to construct a representative semantic space;

[0012] S5, input the image feature vector obtained in step S1 into the MLP model for classification, based on the classified c h The image center of each image is confirmed by the class image, and a noun closest to the image center is selected from the representative semantic space obtained in step S4 based on the Faiss Library method to form a semantic center set;

[0013] S6, describing the mapping relationship between each image obtained in step S1 and each noun in the semantic center set obtained in step S5 as an optimal transport (OT) problem, solving the OT problem with the goal of minimizing the transport cost to obtain the mapping probability, and generating a pseudo label for each image based on the mapping probability;

[0014] S7, updating the weight of the MLP model according to the clustering result obtained in step S4, and constructing a loss function based on the mapping relationship between each image obtained in step S1 and each noun in the semantic center set obtained in step S5, and optimizing the MLP model with the goal of minimizing the loss function;

[0015] S8, Judgment If yes, execute Return to step S4, otherwise, execute step S9, The total number of rounds of training for the preset model;

[0016] S9, input the image feature vector obtained in step S1 into the optimized MLP model obtained in step S7 to generate the optimal classification result, the CLIP model generates image labels based on the optimal classification result, and constructs the hth layer of the hierarchical semantic tree based on the image labels, and determines whether h>1. If so, execute And return to step S3, otherwise output a multi-granularity hierarchical semantic tree.

[0017] The step S4 comprises the following steps:

[0018] S4.1, constructing a semantic space of related words based on the WordNet dataset and the sample labels of each image in step S1;

[0019] S4.2, cluster the image feature vector obtained in step S1 to obtain c h Based on the Faiss Library method, several nouns closest to each cluster center are selected from the semantic space of related words to construct a representative semantic space.

[0020] The step S5 comprises the following steps:

[0021] S5.1, input the image feature vectors obtained in step S1 into the MLP model in sequence to generate each image feature vector in c h A classification result under a class, wherein the classification result includes an image cluster and a corresponding distribution;

[0022] S5.2, according to the classification results obtained in step S5.1, select the top ε with the largest distribution probability under each image cluster. c images constitute the representative image set of each image cluster;

[0023] S5.3, based on the relationship between the image obtained in step S1 and the images in the representative image set, generate an image indicator variable set for each representative image set, and calculate the image center of each type of image based on the image indicator variable set;

[0024] S5.4, using the Faiss Library method, find the nouns closest to the image center obtained in step S5.3 from the representative semantic space obtained in step S4 to construct a semantic center set

[0025] The step S6 comprises the following steps:

[0026] S6.1, calculate the transfer cost between each image and each noun in the semantic center set based on negative cosine similarity;

[0027] S6.2, describing the mapping relationship between each image obtained in step S1 and each noun in the semantic center set obtained in step S5 as an OT problem, and introducing entropy regularization, using the Sinkhorn algorithm to solve the entropy regularized optimal transmission problem to obtain the probability of mapping each image to each semantic center;

[0028] S6.3, calculate the pseudo label of each image based on the probability obtained in step S6.2.

[0029] In step S6.2, the expression of the OT problem after introducing entropy regularization is:

[0030]

[0031]

[0032] In the formula, Π(p, q) represents the set of effective transportation plans between two distributions p and q that meet the flow constraints, that is, the set of effective mapping schemes, τ represents the transportation plan, that is, a set of mapping schemes, σ represents the scale parameter of the smooth transportation plan, H(τ) represents the entropy regularizer, τ ij Represents image x i Mapping to the semantic center set The probability of the jth noun in , Represents a dimension of n×c h The positive real matrix space, 1 n represents an all-one vector of dimension n, Indicates the dimension is c h The all-one vector of and Respectively represent the dimension n and dimension c h positive real number space, n represents the total number of images, C ij Indicates that the image x iMapping to the semantic center set The transmission cost of the j-th noun in .

[0033] In step S4.3, the pseudo-label is expressed as:

[0034] p ij =τ ij / ∑ j τ ij ;

[0035] In the formula, p ij Represents image x i Mapping to the semantic center set The normalized probability of the jth noun in ,∑ j τ ij Represents image x i Mapping to the semantic center set The probability and of each noun in .

[0036] In step S7, the formula for updating the MLP model weights is:

[0037] W = 2 μO;

[0038] Where μ represents the temperature parameter of the MLP model, and O represents the initial cluster center.

[0039] In step S7, the loss function is expressed as:

[0040]

[0041] In the formula, represents the model parameters of the MLP model, represents the image consistency loss, α and β are both trade-off parameters, represents the image-semantic consistency loss, represents the balanced regularization loss, and X represents the image dataset consisting of all images.

[0042] The image consistency loss is defined as follows:

[0043]

[0044] In the formula, rn(N k (x i )) indicates that from N k (x i ) randomly selects a sample, N k (x i ) represents the image x obtained by the FaissLibrary method i Image x iThe nearest neighbor set, k is a positive integer, is the number of elements in the nearest neighbor set, q i Image x representing the output of the MLP model i The soft cluster probability, q i , represents the image x output by the MLP model i′ The soft cluster probability of n represents the total number of images in the image dataset;

[0045] The image-semantic consistency loss is calculated as:

[0046]

[0047] In the formula, p i Indicates c h Pseudo-label p ij The label vector composed of, CE(·) represents the cross entropy loss function;

[0048] The calculation formula of the balanced regularization loss is:

[0049]

[0050] In the formula, represents the average probability that an image sample is assigned to cluster l.

[0051] Beneficial effects of the present invention:

[0052] 1. Based on the WordNet dataset, the text search scope is narrowed according to the hypernyms and synonyms of the fine-grained labels, and then the distance measurement is performed according to the initial image cluster center, and meaningful text is selected to construct a representative semantic space; then k-means clustering is performed based on image features to obtain the initial image cluster center, and the semantic space is determined based on the image cluster center. The initial clustering results can provide initial state guidance for distinguishing visual confusion in the subsequent unsupervised clustering process;

[0053] 2. Design a suitable image-text mapping relationship, combine images and semantics, that is, nouns in the semantic center set to provide guidance for image clustering, and deal with the problem of visual confusion. Propose cluster centers and semantic spaces based on image clustering, use OT methods to construct image-semantic pseudo labels, generate soft cluster mapping relationships from images to semantics, and enhance image clustering methods through consistent learning in image space and semantic space to solve the problem of visual semantic confusion;

[0054] 3. To ensure that the unsupervised clustering process meets the local smoothness assumption that similar images have similar cluster assignment results, and to prevent the generation of empty clusters. In the process of unsupervised clustering design, the clustering objective function training granularity generation model is constructed by combining image consistency learning, image semantic consistency learning, and balanced regularization to obtain a fused image-semantic relationship and a balanced clustering method to generate data of different granularities, and further obtain a multi-granularity hierarchical semantic tree through iteration. Through the designed consistency learning training, data of different granularities are generated to meet the matching requirements of heterogeneous models, improve the accuracy performance of large and small model training, and give full play to the maximum capabilities of heterogeneous models;

[0055] 4. Based on the differential performance of heterogeneous models in image recognition tasks with different data granularities, this application can divide heterogeneous models into different granularities according to their volume based on the matching requirements of the models, that is, different models correspond to the recognition capabilities of data with different granularities. By constructing a multi-granularity hierarchical semantic tree to generate data with different granularities, and then dividing image recognition tasks with different granularities, it can correspond to the division of heterogeneous models with different granularities, which helps to assign tasks to appropriate models, thereby maximizing the potential of collaborative training of heterogeneous models;

[0056] 5. It solves the problem that the current data set has only a fixed granularity and does not meet the matching requirements of heterogeneous models. In the intelligent transportation scenario, by matching data of different granularities with models of different capabilities, the overall system performance and real-time response capabilities can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 It is a schematic diagram of the process of the present invention.

[0059] Figure 2 Schematic diagram of the formation of semantic space.

[0060] Figure 3 Construct a flowchart for pseudo-labeling.

[0061] Figure 4 This is a schematic diagram for comparing this application with other solutions.

[0062] Figure 5 Schematic diagram of comparing the effectiveness of the present application on HFL on devices with different data sets and different data granularities. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] Multilayer Perceptron MLP model: a fully connected feedforward artificial neural network (ANN). An MLP consists of at least three layers of nodes: input layer, hidden layer, and output layer. In addition to the input node, each node is a neuron using a nonlinear activation function. MLP is trained using a supervised learning technique called backpropagation. Its multiple layers and nonlinear activation distinguish MLP from linear perceptrons. It can distinguish data that is not linearly separable.

[0065] The optimal transport (OT) problem is to study how to move mass from one distribution to another with the lowest "transportation cost" while satisfying certain constraints.

[0066] Sinkhorn-Knopp algorithm: It is an efficient method to solve the optimal transmission problem with entropy regularization. It solves the high computational complexity of the traditional optimal transmission problem by introducing entropy regularization, making the calculation of optimal transmission on high-dimensional data more feasible. The basic idea is to quickly converge to the optimal transmission matrix by alternately scaling the rows and columns of the matrix.

[0067] A multi-granularity hierarchical semantic tree construction method for heterogeneous models, such as Figure 1 As shown, the following steps are included:

[0068] S1, collect images and corresponding sample labels, build a granular tree generation model including an MLP model and a CLIP model, and use the image encoder of the CLIP model to generate an image feature vector set according to the image;

[0069] Collect traffic monitoring images and their corresponding sample labels. The sample labels are all fine-grained labels that can reflect the semantic information of the image. All images constitute an image dataset, which is express, Among them, x i represents the i-th image, and n represents the image dataset The total number of samples. Inputting the image dataset into the CLIP-based image encoder can obtain the image feature vector set, which is represented by U, where U = {u1, u2, ..., u i , ..., u n},u i Represents image x i The image feature vector of . The corresponding expression is u i =g(x i ), g(·) is the image encoder. The sample labels of all images form a semantic feature set, which is express, Among them, v i Represents image x i The sample labels of .

[0070] When constructing a hierarchical semantic tree, for the current layer granularity, the sample image and its label are input into the CLIP model, and the image and text are encoded using the image encoder and text encoder of the CLIP model. The image encoding of each sample is further input into the MLP model to obtain the soft cluster probability that the sample belongs to each tree node.

[0071] S2, set the depth of the tree The corresponding leaf nodes are generated according to the images collected in step S1, and the sample labels are used as the labels of the corresponding nodes to form the first leaf node of the hierarchical semantic tree. layer, and

[0072] S3, initialize the MLP model training round t = 1, the current number of layers And set the number of layer nodes c h ;

[0073] The multi-granularity hierarchical semantic tree established in this application is a multi-layer structure, and the granularity of the nodes corresponding to each layer is generated in sequence from fine to coarse. Therefore, the number of nodes in the bottom layer is greater than the number of nodes in the adjacent upper layer, and so on.

[0074] S4, such as Figure 2 As shown, the image feature vector obtained in step S1 is clustered to obtain c h Cluster centers are formed, and the FaissLibrary method is used to select several nouns closest to each cluster center based on the cluster centers and the WordNet dataset to construct a representative semantic space, including the following steps:

[0075] S4.1, constructing a semantic space of related words based on the WordNet dataset and the sample labels of each image in step S1;

[0076] Since the application is to build a multi-granularity hierarchical semantic tree, it is necessary to set meaningful text labels for the nodes obtained by clustering at each step. In this step, the goal of the application is to build a semantic space suitable for the image by introducing relevant text, which not only helps the subsequent image-semantic mapping process, but also prepares for the subsequent setting of suitable clustering node semantic labels. In the image clustering task, the application needs to cluster the images according to the object category attributes of the image, and the set of object category names is usually a subset of common nouns in English. For example, in the CIFAR10 data set, the class name is 20 commonly used English nouns ("fish", "insects", "trees", etc.).

[0077] Specifically, first, the entire list of nouns in the WordNet dataset is used to form a semantic dataset W containing more than a number of nouns. Since the sample labels of the image can effectively describe the semantic features of the image, the present application uses the synonym and hypernym operations to find nouns related to the sample labels in the semantic dataset W based on the sample labels of each image. This is a prior art and will not be described in detail in this embodiment. The set of all nouns forms a related word semantic space W. r , W r ∈W.

[0078] S4.2, cluster the image feature vector obtained in step S1 to obtain c h Based on the Faiss Library method, several nouns closest to each cluster center are selected from the semantic space of related words to construct a representative semantic space;

[0079] Considering the relevant semantic space W r The words in may be irrelevant to the given image. This application further filters the semantic space W of related words according to the image. r Specifically, first, perform K-means clustering on the image feature vectors in the image feature vector set U to obtain c n The initial cluster center O is then extracted from the relevant semantic space W based on the Faiss Library method. r In the above example, we select γ closest nouns for each cluster center to form the final representative semantic space T, where T = {t1, t2, ..., t m}, T∈W r , m represents the number of nouns in the semantic space T, m = c h *γ,t m It represents the mth noun in the semantic space T.

[0080] S5, input the image feature vector obtained in step S1 into the MLP model for classification, based on the classified c nThe image center of each image is confirmed by the class image. Based on the Faiss Library method, a noun closest to the image center is selected from the representative semantic space obtained in step S4 to form a semantic center set, including the following steps:

[0081] S5.1, input the image feature vectors obtained in step S1 into the MLP model in sequence to generate each image feature vector in c h A classification result under a class, wherein the classification result includes an image cluster and a corresponding distribution;

[0082] S5.2, according to the classification results obtained in step S5.1, select the top ε with the largest distribution probability under each image cluster. c images constitute the representative image set of each image cluster;

[0083] S5.3, based on the relationship between the image obtained in step S1 and the images in the representative image set, generate an image indicator variable set for each representative image set, and calculate the image center of each type of image based on the image indicator variable set;

[0084] The image feature vectors in the image feature vector set are sequentially input into the MLP. The MLP model generates a soft cluster assignment result based on the image feature vector. The soft cluster assignment result includes the cluster to which the image belongs, i.e., the class, and the distribution under the cluster. The total number of clusters is c h . Filter out the top ε of each class of images c The image feature vectors with larger distribution probabilities form the corresponding representative image set, and the image indicator variable set of each representative image set is generated according to the relationship between the images in the image feature vector set and the images in the representative image set. The selection results can be stored by the binary matrix Z.

[0085] According to the relationship between the images in the image data set and the images in the representative image set, the image indicator vector set of each representative image set is generated respectively, which means to judge whether each image in the image data set belongs to the elements of each representative image set in turn, and generate the image indicator variable set based on the judgment result. The representative image set of the lth category of images adopts Indicates that, and l≤c h , represents the image set The image indicator variable set is adopted express, When the image x i Belongs to the representative image set When the element in il =1, otherwise, z il =0.

[0086] The image center is calculated as:

[0087]

[0088] In the formula, represents the image center set v c That is, the image center of the lth image in the set of all image centers, z il Represents image x i With representative image set The image indicator variable, ||z l ||1 means z l The L1 norm of z l Represents an image dataset All images in the set belong to the representative image set The total number of elements in .

[0089] S5.4, using the Faiss Library method, find the nouns closest to the image center obtained in step S5.3 from the representative semantic space obtained in step S4 to construct a semantic center set

[0090] Since the multimodal pre-trained model CLIP builds a bridge between images and text, the present application can effectively link images with semantics. In the prior art, the image feature vector set U obtained based on the CLIP model and the semantic data set W are directly mapped to simply obtain the semantic center result, but this may lead to semantic ambiguity, that is, the image may be mapped to semantics that do not match the meaning of the image expression, and ultimately generate meaningless semantic centers. Therefore, the present application proposes to generate c h Representative semantic centers Based on the center mapping strategy, Compared with directly mapping images to text, it is more meaningful to map the image center to semantics and then obtain the semantic center.

[0091] Specifically, the Faiss Library method is used to find the noun closest to each image center from the representative semantic space T in turn. All nouns form a semantic center set. Therefore, the number of elements in the semantic center set is c. h .

[0092] S6, such as Figure 3 As shown, the mapping relationship between each image obtained in step S1 and each noun in the semantic center set obtained in step S5 is described as an optimal transport (OT) problem. The OT problem is solved with the goal of minimizing the transport cost to obtain the mapping probability. A pseudo label for each image is generated based on the mapping probability, including the following steps:

[0093] S6.1, calculate the negative cosine similarity between each image and the semantic center set The transmission cost between each noun in is calculated as follows:

[0094]

[0095] In the formula, C ij Indicates that the image x i Mapping to the semantic center set The transmission cost of the jth noun in the image x is i The image feature vector u i Semantic Center The negative cosine similarity between the normalized embeddings of gives, c h Representing the semantic center set The number of semantic centers in , and 1≤j≤c h ,

[0096] S6.2, describing the mapping relationship between each image obtained in step S1 and each noun in the semantic center set obtained in step S5 as an OT problem, and introducing entropy regularization, using the Sinkhorn algorithm to solve the entropy regularized optimal transmission problem to obtain the probability of mapping each image to each semantic center;

[0097] The pseudo-label is obtained based on the relationship between the image and the semantic center, which can be expressed as an optimal transport (OT) problem. The expression of the OT problem after introducing entropy regularization is:

[0098]

[0099] In the formula, Π(p, q) represents the set of effective transportation plans between two distributions p and q that meet the flow constraints, that is, the set of effective mapping schemes, τ represents the transportation plan, that is, a set of mapping schemes, σ represents the scale parameter of the smooth transportation plan, H(τ) represents the entropy regularizer, τ ij Represents image x i Mapping to the semantic center set The probability of the jth noun in . Represents a dimension of n×c h The positive real matrix space, 1 n represents an all-one vector of dimension n, Indicates the dimension is c h The all-one vector of q and p represent the image set and semantic center set respectively, and Respectively represent the dimension n and dimension c h The space of positive real numbers.

[0100] The entropy regularizer H(τ) is calculated as:

[0101] H(τ)=-∑ i，j τ ij log τ ij ; (4)

[0102] This application uses the Sinkhorn-Knopp algorithm to effectively solve equation (3), and then obtains the image soft cluster assignment pseudo label that combines the image and semantic relationship to guide the self-supervised clustering process.

[0103] S6.3, calculating the pseudo label of each image according to the probability obtained in step S6.2;

[0104] Once the transport plan τ is obtained, the application can generate a transport plan for the image x by normalizing the transport plan of the image to 1. i Define pseudo labels. The expression of pseudo labels is:

[0105] p ij =τ ij / ∑ j τ ij ; (5)

[0106] In the formula, p ij Represents image x i Mapping to the semantic center set The normalized probability of the jth noun in , and ∑ j τ ij Represents image x i Mapping to the semantic center set The probability and of each noun in . The semantically enhanced pseudo-labels are used to guide the subsequent MLP clustering self-supervised learning process.

[0107] S7, updating the weight of the MLP model according to the clustering result obtained in step S4, and constructing a loss function based on the mapping relationship between each image obtained in step S1 and each noun in the semantic center set obtained in step S5, and optimizing the MLP model with the goal of minimizing the loss function;

[0108] The formula for updating the MLP model weights is:

[0109] W = 2 μO; (6)

[0110] Where μ represents the temperature parameter of the MLP model, and O represents the initial cluster center, i.e., c h The matrix composed of the initial cluster centers.

[0111] The expression of the loss function is:

[0112]

[0113] In the formula, represents the model parameters of the MLP model, represents the image consistency loss, α and β are both trade-off parameters, represents the image-semantic consistency loss, represents the balanced regularization loss.

[0114] In the learning model MLP In the process, the present application initializes the multi-layer perceptron MLP model based on the initial cluster center in step S2 before training. First, the local smoothness assumption is introduced for consistency learning. The specific content of the local smoothness assumption is: if two images x i and x i′ are located in local neighborhoods of each other in the low-dimensional manifold, then they will have similar soft clustering assignments, i.e., q i and q i′ Based on the local smoothness assumption, this method can enhance the consistency between neighbors in the image space to improve the model Perform learning and training. The image consistency loss is defined as follows:

[0115]

[0116] In the formula, rn(N k (x i )) indicates that from N k (x i ) randomly selects a sample, N k (x i ) represents the image x obtained by the FaissLibrary method i Image x i The nearest neighbor set, k is a positive integer, is the number of elements in the nearest neighbor set, q i Image x representing the output of the MLP model i The soft cluster probability, q i′ Image x representing the output of the MLP model i′ The soft cluster probability.

[0117] In the process of self-supervised learning, this application defines the calculation formula of image-semantic consistency loss as follows:

[0118]

[0119] In the formula, p i Indicates c h Pseudo-label p ij The label vector is composed of , CE(·) represents the cross entropy loss function. The image-semantic consistency loss combines both images and semantics to provide guidance for image clustering, so as to solve the problem of visual semantic confusion caused by consistency learning based only on images.

[0120] In order to prevent the model training from producing empty cluster results, this application adds balanced clustering regularization and introduces the commonly used negative entropy loss. The calculation formula of the balanced regularization loss defined in this application is as follows:

[0121]

[0122] In the formula, It represents the average probability that an image sample is assigned to cluster l, measures the representativeness of the cluster, and reflects the average cluster assignment. Based on the average cluster, a balanced regularization loss is further constructed to ensure that each cluster is properly assigned during the clustering process through optimization, thereby avoiding the generation of empty clusters or extremely unbalanced cluster structures.

[0123] Average probability The calculation formula is:

[0124]

[0125] In the formula, q il Represents image x i The probability of mapping to the lth cluster, It represents the sum of the probabilities that all image samples in the image dataset belong to the lth cluster.

[0126] S8, Judgment If yes, execute Return to step S4, otherwise, execute step S9, The total number of rounds of training for the preset model;

[0127] S9, input the image feature vector obtained in step S1 into the optimized MLP model to generate the optimal classification result, the CLIP model generates image labels based on the optimal classification result, and constructs the hth layer of the hierarchical semantic tree based on the image labels, and determines whether h>1. If so, execute And return to step S3, otherwise output a multi-granularity hierarchical semantic tree;

[0128] This application uses this centralized construction method of multi-granularity semantic trees to generate data granularity to meet the matching requirements of models of different sizes. Compared with the five clustering methods for granularity generation in the prior art: SIC (Structural Information Constraints), SCAN (Structural Clustering Algorithm for Networks), NNM (Nearest Neighbor Matching), and CC (Contrastive Clustering), the technical advantages of this application are obtained, indicating that by utilizing multi-dimensional information fusion and cross-model interaction, the challenge of incomplete generation is solved, and the needs of heterogeneous models and data granularity matching are met. This application matches the single-layer granularity of the semantic tree generated by the baseline to a specific model for verification. For example, this method verifies the performance of the first layer of the semantic tree on AlexNet (50 nodes on CIFAR100 and 100 nodes on TinyImageNet). This method evaluates the effectiveness of the generated granularity by the accuracy of each layer. In addition, this method redesigns the precision density based on the matching relationship between the model scale and the data granularity (calculated as the accuracy divided by the number of nodes in the current layer). At the same time, this method calculates the sum of the precision densities of all layers as the evaluation index of the entire MGTree.

[0129] like Figure 4 As shown in the figure, on CIFAR100 and TinyImageNet, the granular generation model shows significant accuracy over the baselines of all layers, 14.98% and 17.43% respectively. At the same time, the accuracy density of this method is also significantly improved, 12.47% on CIFAR100 and 11.49% on TinyImageNet. This method uses OT-based pseudo-label construction, which can provide more multimodal mapping information to generate pseudo-labels and enhance the semantic guidance of multimodal consistency learning.

[0130] In the HFL (Horizontal Federated Learning) framework, we performed federated fine-tuning on the large-scale pre-trained model CLIP to verify the effectiveness of MGTree. First, we set different devices with coarse data granularity and fine data granularity, and considered the data distribution between devices for IID and Non-IID. Next, we compared the accuracy of the fine-tuned CLIP in three scenarios: local training, FL with devices of the same data granularity, and FL guided across devices of different granularities. Figure 5As shown, on CIFAR100, devices with different data granularities guide each other in HFL, and fine-tuning CLIP achieves higher accuracy. On TinyImageNet, the NonIID data distribution makes better use of multi-granularity data. This suggests that for federated fine-tuning of large models, HFL with different data granularities using MGTree can effectively improve collaborative accuracy performance.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a multi-granularity hierarchical semantic tree for heterogeneous models, characterized in that: The steps include: S1, collect images and corresponding sample labels, build a granular tree generation model including an MLP model and a CLIP model, and use the image encoder of the CLIP model to generate an image feature vector set according to the image; S2, set the depth of the tree The corresponding leaf nodes are generated according to the images collected in step S1, and the sample labels are used as the labels of the corresponding nodes to form the first leaf node of the hierarchical semantic tree. layer, and S3, initialize the MLP model training round Current layer And set the number of layer nodes c h ; S4, clustering the image feature vector obtained in step S1 to obtain c h Cluster centers are formed, and the Faiss Library method is used to select several nouns closest to each cluster center based on the cluster centers and WordNet dataset to construct a representative semantic space; S5, input the image feature vector obtained in step S1 into the MLP model for classification, based on the classified c h The image center of each image is confirmed by the class image, and a noun closest to the image center is selected from the representative semantic space obtained in step S4 based on the Faiss Library method to form a semantic center set; S6, describing the mapping relationship between each image obtained in step S1 and each noun in the semantic center set obtained in step S5 as an optimal transport (OT) problem, solving the OT problem with the goal of minimizing the transport cost to obtain the mapping probability, and generating a pseudo label for each image based on the mapping probability; S7, updating the weight of the MLP model according to the clustering result obtained in step S4, and constructing a loss function based on the mapping relationship between each image obtained in step S1 and each noun in the semantic center set obtained in step S5, and optimizing the MLP model with the goal of minimizing the loss function; S8, Judgment If yes, execute Return to step S4, otherwise, execute step S9, The total number of rounds of training for the preset model; S9, input the image feature vector obtained in step S1 into the optimized MLP model obtained in step S7 to generate the optimal classification result, the CLIP model generates image labels based on the optimal classification result, and constructs the hth layer of the hierarchical semantic tree based on the image labels, and determines whether h>1. If so, execute And return to step S3, otherwise output a multi-granularity hierarchical semantic tree.

2. The method for constructing a multi-granularity hierarchical semantic tree for heterogeneous models according to claim 1, characterized in that: The step S4 comprises the following steps: S4.1, constructing a semantic space of related words based on the WordNet dataset and the sample labels of each image in step S1; S4.2, cluster the image feature vector obtained in step S1 to obtain c h Based on the Faiss Library method, several nouns closest to each cluster center are selected from the semantic space of related words to construct a representative semantic space.

3. The method for constructing a multi-granularity hierarchical semantic tree for heterogeneous models according to claim 1, characterized in that: The step S5 comprises the following steps: S5.1, input the image feature vectors obtained in step S1 into the MLP model in sequence to generate each image feature vector in c h A classification result under a class, wherein the classification result includes an image cluster and a corresponding distribution; S5.2, according to the classification results obtained in step S5.1, select the top ε with the largest distribution probability under each image cluster. c images constitute the representative image set of each image cluster; S5.3, based on the relationship between the image obtained in step S1 and the images in the representative image set, generate an image indicator variable set for each representative image set, and calculate the image center of each type of image based on the image indicator variable set; S5.4, using the Faiss Library method, find the nouns closest to the image center obtained in step S5.3 from the representative semantic space obtained in step S4 to construct a semantic center set 4. The method for constructing a multi-granularity hierarchical semantic tree for heterogeneous models according to claim 1, characterized in that: The step S6 comprises the following steps: S6.1, calculate the transfer cost between each image and each noun in the semantic center set based on negative cosine similarity; S6.2, describing the mapping relationship between each image obtained in step S1 and each noun in the semantic center set obtained in step S5 as an OT problem, and introducing entropy regularization, using the Sinkhorn algorithm to solve the entropy regularized optimal transmission problem to obtain the probability of mapping each image to each semantic center; S6.3, calculate the pseudo label of each image based on the probability obtained in step S6.

2.

5. The method for constructing a multi-granularity hierarchical semantic tree for heterogeneous models according to claim 4, characterized in that: In step S6.2, the expression of the OT problem after introducing entropy regularization is: In the formula, Π(p, q) represents the set of effective transportation plans between two distributions p and q that meet the flow constraints, that is, the set of effective mapping schemes, τ represents the transportation plan, that is, a set of mapping schemes, σ represents the scale parameter of the smooth transportation plan, H(τ) represents the entropy regularizer, τ ij Represents image x i Mapping to the semantic center set The probability of the jth noun in , Represents a dimension of n×c h The positive real matrix space, 1 n represents an all-one vector of dimension n, Indicates the dimension is c h The all-one vector of and Respectively represent the dimension n and dimension c h positive real number space, n represents the total number of images, C ij Indicates that the image x i Mapping to the semantic center set The transmission cost of the j-th noun in .

6. The method for constructing a multi-granularity hierarchical semantic tree for heterogeneous models according to claim 4, characterized in that: In step S4.3, the pseudo-label is expressed as: p ij =t ij / ∑ j t ij ; In the formula, p ij Represents image x i Mapping to the semantic center set The normalized probability of the jth noun in ,∑ j τ ij Represents image x i Mapping to the semantic center set The probability and of each noun in .

7. The method for constructing a multi-granularity hierarchical semantic tree for heterogeneous models according to claim 1, characterized in that: In step S7, the formula for updating the MLP model weights is: W = 2 μO; Where μ represents the temperature parameter of the MLP model, and O represents the initial cluster center obtained by clustering the image feature vector.

8. The method for constructing a multi-granularity hierarchical semantic tree for heterogeneous models according to claim 1, characterized in that: In step S7, the loss function is expressed as: In the formula, represents the model parameters of the MLP model, represents the image consistency loss, α and β are both trade-off parameters, represents the image-semantic consistency loss, represents the balanced regularization loss, and χ represents the image dataset consisting of all images.

9. The method for constructing a multi-granularity hierarchical semantic tree for heterogeneous models according to claim 8, characterized in that: In step S7, the image consistency loss is defined as follows: In the formula, rn(N k (x i )) indicates that from N k (x i ) randomly selects a sample, N k (x i ) represents the image x obtained by the Faiss Library method i Image x i The nearest neighbor set, k is a positive integer, is the number of elements in the nearest neighbor set, q i Image x representing the output of the MLP model i The soft cluster probability, q i′ Image x representing the output of the MLP model i′ The soft cluster probability of n represents the total number of images in the image dataset; The image-semantic consistency loss is calculated as: In the formula, p i Indicates c h Pseudo-label p ij The label vector composed of, CE(·) represents the cross entropy loss function; The calculation formula of the balanced regularization loss is: In the formula, represents the average probability that an image sample is assigned to cluster l.

Images