Distributed large model training-oriented multi-granularity semantic tree construction system and method thereof

Abstract

The invention discloses a distributed large model training-oriented multi-granularity semantic tree construction system and a distributed large model training-oriented multi-granularity semantic tree construction method. According to an edge-cloud information fusion space, samples and text tags of a cloud server are embedded and collected into public data set information of edge equipment # imgabs0 # for multi-modal data fusion; the model information interaction space is used for embedding samples and text labels of a cloud server and obtaining a global initial cluster center Ol belonging to Rd * l by adopting a hypergraph clustering algorithm; the cloud server constructs a semantic space # imgabs1 # based on the initial cluster center Ol, and the cloud server distributes the initial cluster center Ol and a text embedding # imgabs2 # of the semantic space to each edge device to construct an edge-cloud cooperative interaction space; the cloud server generates a multi-granularity semantic tree; according to the edge-cloud collaborative interaction space construction global granularity generation model, appropriate data granularity is cooperatively generated in a construction system, and the matching requirement of a heterogeneous model is met.

Description

Technical Field

[0001] The present invention belongs to large model collaborative processing technology, and in particular to a multi-granularity semantic tree construction system and method for distributed large model training. Background Art

[0002] With the rapid development of artificial intelligence technology, large models such as GPT4 and BERT continue to emerge, driving major breakthroughs in various fields. However, large models face major challenges in energy consumption and edge deployment, which limit the application and development of large models. To address this issue, it has become necessary to perform collaborative heterogeneous distributed training for large and small models in building systems. In order to fully exploit the potential of this collaborative training, some recent studies have been proposed, mainly focusing on models and training, including lightweight models, enhanced quantization performance, and personalization. However, an important factor that has received less attention is the different data granularity.

[0003] In distributed training, data presents multiple granularities, that is, samples are similar but have different labels, which also leads to diversity in training data. Most studies use the same data granularity, ignoring the potential benefits of utilizing different data granularities. In fact, models of different sizes have different performances in data of different granularities. Smaller models have lower accuracy on fine-grained data due to their insufficient ability to handle complex tasks, but show higher model and data utilization in coarse-grained data. On the contrary, larger models cannot fully utilize their potential on coarse-grained data, which is a waste of their own model capabilities. These findings highlight the necessity of matching appropriate data granularity to different model scales, especially in the complex scenarios of distributed training in the era of large models. Therefore, exploring how to make full use of data granularity may become a new breakthrough in promoting accurate and efficient heterogeneous distributed training of large and small models.

[0004] The existing technology has disadvantages:

[0005] 1. The fixed granularity of current datasets cannot meet the matching requirements of various models ranging from billion parameters to lightweight ones. In order to maximize the benefits of distributed training and achieve high model utilization, it is necessary to generate appropriate data granularity for different model scales. In addition, due to the decentralized nature of the construction system, large-scale data challenges, and privacy issues of edge sensitive data, centralized data granularity generation methods are impractical.

[0006] 2. Due to the lack of comprehensive information, it is difficult to obtain all the knowledge to generate transformation relationships of different data granularities, resulting in incomplete generation. In a distributed architecture, a single device may only contain part of the fine-grained data, which limits the construction of comprehensive data granularity and hinders the granularity expansion of new data. In addition, using unimodal features for generation may confuse the multi-granularity hierarchy because it may be difficult to distinguish visually similar but semantically different data using only image features. Therefore, making full use of the multimodal features of comprehensive data is crucial to generating complete and robust data granularity.

[0007] 3. The data granularity generation process based on clustering is significantly affected by the initial state. However, due to the chaotic complexity of multidimensional data information, it is challenging to obtain an effective initial state. It is difficult to integrate heterogeneous data from all edges, and the data itself is large-scale and high-dimensional, which poses a challenge to traditional clustering methods. The chaotic and complex information from the edges to the data makes it difficult to achieve a stable and comprehensive initial spatial clustering. Summary of the invention

[0008] In response to the difficulties in the prior art, the present invention provides a multi-granularity semantic tree construction system and method for distributed large model training. The present invention collaboratively generates appropriate data granularity in the construction system to meet the matching requirements of heterogeneous models. At the same time, the present invention significantly enhances the guidance ability of large models to small models by adopting a distributed mechanism for data and model interaction, thereby achieving efficient and accurate distributed training.

[0009] In order to solve the problems of the prior art, the present invention adopts the following technical solutions to implement

[0010] A multi-granularity semantic tree construction system for distributed large model training, the construction system comprising a cloud server and an edge device; the construction system comprising an edge-cloud information fusion space, a model information interaction space, an edge-cloud collaborative interaction space, a multi-granularity semantic tree and a global granularity generation model; wherein:

[0011] The edge-cloud information fusion space embeds the samples and text labels of the cloud server into and Collect edge devices in the form of Multimodal data fusion based on public data set information;

[0012] The model information interaction space embeds the samples and text labels of the cloud server into and use

[0013] Hypergraph clustering algorithm obtains the global initial cluster center O l ∈R d×k ;

[0014] The cloud server is based on the initial cluster center O l Filtering words in WordNet to build semantic space

[0015] The cloud server sets the initial cluster center O l and text embedding in semantic space Distribute to each edge device to build edge-cloud collaborative interaction space;

[0016] The cloud server sets the initial cluster center O l Send it to each edge device to initialize the granularity generation model of each edge device;

[0017] The edge-cloud collaborative interaction space embeds text in the semantic space It is sent to each edge device to guide the subsequent local learning and training of edge devices to build a global granularity generation model, namely:

[0018]

[0019] Among them: it maps labels from the lth level granularity to the l+1th level; represents the sample embedding on edge device n, represents the l-th level text label embedding of samples on edge device n, is the l+1th level text label of the sample on edge device n; φ n That is, the local granularity generation model parameters for building a multi-granularity semantic tree deployed on the edge device n.

[0020] Furthermore, the construction system consists of N edge devices and 1 cloud server, which are respectively denoted as and C;

[0021] The multi-granularity semantic tree uses distributed layer-by-layer generation of different data granularities, and constructs an L-layer tree structure corresponding to models of different scales; that is, Among them: The number of tree nodes in the lth layer is recorded as L l ;

[0022] The edge device is Each edge device, i.e. As a local sample data set, where: M n ∈N + The text label set corresponding to the number of samples at the l-th level granularity is recorded as The edge device uses a sample encoder g(·) to generate a sample embedding The edge device has a Use the text encoder z(·) to get the corresponding text embedding in: For the edge The cloud server collects a public data set Where: contains representative samples from each edge device; M C ∈N + express The number of samples in ; As a collection of sample embeddings, As text embeddings of level l granularity labels, where: Represents the text label corresponding to the sample at the lth level of the multi-granularity semantic tree.

[0023] Furthermore, the model information interaction space embeds the samples and text labels of the cloud server into

[0024] and The hypergraph clustering algorithm is used to obtain the global initial cluster center O l ∈R d×k Process; including:

[0025] Based on sample embedding By changing V v Its nearest top-k I Neighbor vertices are connected to obtain sample hyperedges Right now:,

[0026] Text Hyperedge By embedding V based on the similarity of text embedding at the current input layer l data granularity v Its top-k T The nearest neighbor vertices are connected; that is:

[0027] According to the following formula, ε I and ε T Normalize the weights of

[0028] The degree is:

[0029]

[0030] Among them, ∈∈[0,1] balances the sample and text hyperedge weights;

[0031] The normalized Laplace matrix of the hypergraph is constructed according to the following formula:

[0032]

[0033] Among them: and The diagonal matrices representing vertex degree and hyperedge degree respectively; W = diag(w(E 1 ),…,w(E |ε| )) represents the diagonal matrix of hyperedge weights;

[0034] Cluster from layer l to layer l+1, use spectral clustering on Λ; and calculate the eigenvector L corresponding to Λ l+1 smallest non-zero eigenvalue;

[0035] These eigenvectors form the spectral embedding matrix

[0036] A low-dimensional encoding that represents the structure of a hypergraph;

[0037] right Apply K-means clustering to obtain L in the lth layer l+1 The initial clusters are denoted as

[0038] Furthermore, the cloud server sets the initial cluster center O l Sent to each edge device to initialize the granularity generation model of each edge device; including:

[0039] Construct the l+1th layer of labels based on the lth layer of labels, using the text labels currently input to the lth layer Get sample descriptors;

[0040] This set is expanded by searching for synonyms and hypernyms in WordNet, creating an initial semantic space.

[0041] Through the initial cluster center O l Based on the Faiss library, Among them: Based on Select the closest γ nouns for each cluster center, that is:

[0042]

[0043] Among them: sim(·,·) is the similarity measure based on Euclidean distance; z(t) represents Semantic embedding of

[0044] The final semantic space of the lth layer is formed based on the union of these nearest neighbor words, that is, Where: T is the total number of noun phrases selected from WordNet;

[0045] For each noun Construct a descriptive sentence template "a photo of a ”, obtain semantic embedding Furthermore, the edge-cloud collaborative interaction space embeds text into It is sent to each edge device to guide the subsequent local learning and training of edge devices to build a global granularity generation model process; including:

[0046] Jointly balance the regularized consistency loss to train local multi-granularity semantic trees;

[0047] The FL framework is used to jointly train the upper layer of the multi-granularity semantic tree;

[0048] The edge device optimizes the parameters in its granular generation model locally according to the following loss function and transmits it to the cloud server;

[0049]

[0050] Where: η is the learning rate. At the end of each global iteration e, the edge device will Upload to the cloud server;

[0051] The cloud server aggregates these parameters using the FedAvg method and builds a global granular generation model through E rounds of iterations, and updates the model parameters;

[0052]

[0053] in: is the number of data on edge device n, is the total amount of data on all edge devices;

[0054] The global granularity generation model takes the first layer of data as input and finally maps the lth layer of data to the l+1th layer of labels, that is,

[0055] Realize heterogeneous edge device model interaction.

[0056] Furthermore, the edge-cloud collaborative interaction space trains a local multi-granularity semantic tree process by jointly balancing the regularized consistency loss, including:

[0057] The parameter Φ in the model is generated by the granularity n Will embed u n,i Mapping to soft cluster assignment probabilities

[0058] Right now

[0059] According to the relationship between sample embedding and semantic embedding, the pseudo labels of the sample set are generated by normalization:

[0060]

[0061] in: The cth element of is calculated as τ i,c / ∑ c τ i,c , indicating u n,i arrive Soft clustering probabilities of mapping;

[0062] The semantic enhancement loss function is obtained according to the pseudo-label according to the following formula;

[0063]

[0064] Where: CE(·) is the cross entropy loss function; M n represents the number of local samples of edge device n,

[0065] Represents the embedding u n,i To the semantic center The pseudo-labels formed by mapping, Represents the embedding u n,i Soft cluster allocation result mapped by local granularity generator;

[0066] According to the semantic enhancement loss function, the balanced regularization loss function is obtained according to the following formula;

[0067]

[0068] Among them: α is a trade-off parameter.

[0069] Beneficial Effects

[0070] The present invention solves the problems existing in the prior art:

[0071] 1. For the distributed training of large and small models, the present invention collaboratively generates appropriate data granularity in the construction system to meet the matching requirements of heterogeneous models. By adopting a distributed mechanism for data and model interaction, the guidance ability of large models to small models is significantly enhanced, and efficient and accurate distributed training is achieved.

[0072] 2. In order to overcome the problem of incomplete granularity generation, the present invention proposes a multi-dimensional information fusion strategy that combines image features and text labels from the edge and the cloud. This strategy enables the present invention to comprehensively capture the deep semantics of the image and provides rich guidance for the granularity generation model. In addition, the present invention adopts the Optimal Transport (OT) theory to establish a soft pseudo-label mapping between images and semantics, and designs semantic-enhanced consistency learning based on multimodal fusion. By utilizing the Fedrated Learning (FL) method, the present invention realizes efficient collaborative model interaction across the edge and the cloud, and effectively integrates data information.

[0073] 3. To address the challenge of uneven granularity generation caused by the chaotic complexity of multimodal data, this paper introduces a hypergraph-based multimodal clustering method. The cloud collects and processes multimodal features of public data from the edge and uses Hypergraph Clustering (HGC) to capture complex high-order relationships within the data. This makes the generation of stable and comprehensive initial cluster centers a robust guide for dynamic learning of granularity generation models on the edge.

[0074] The present invention matches the single-layer granularity of the semantic tree generated by the baseline to a specific model for verification. For example, the present invention verifies the performance of the first layer of the semantic tree (50 nodes on CIFAR100 and 100 nodes on TinyImageNet) on AlexNet.

[0075] like Figure 5 As shown, on CIFAR100 and TinyImageNet, the granular generation model shows significant accuracy over the baselines of all layers, 14.98% and 17.43%, respectively. At the same time, Cultivator's accuracy density has also been significantly improved, by 12.47% on CIFAR100 and 11.49% on TinyImageNet. Compared with the centralized baseline, Cultivator (centralized) generates more appropriate data granularity for different layers and datasets by enhancing multimodal information fusion and semantic learning. However, Cultivator (centralized) uses all data and more model parameters for training, which is prone to overfitting.

[0076] When generating the first layer of granularity, the differences between classes may be blurred, resulting in incomplete data and insufficient fitting. As higher layers of granularity are generated, that is, the number of classes decreases, the classes become easier to distinguish. In this case, due to overfitting, Cultivator (centralized) loses important general patterns that Cultivator can better capture with partial data. This shows that Cultivator generates data granularity suitable for the model, which is conducive to efficient and accurate distributed training. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 The present invention has a construction system architecture with data and model interaction;

[0078] Figure 2 is the initial HGC method of the present invention;

[0079] Figure 3 It is the semantic space determination process of the present invention;

[0080] Figure 4 It is the design of the multimodal consistency learning loss of the present invention;

[0081] Figure 5 The effectiveness of the generated granularity is compared between the proposed method and the baseline solution. DETAILED DESCRIPTION

[0082] The following is combined with Figure 1 to Figure 4 The present invention is described as follows:

[0083] 1. System Overview

[0084] The goal of this invention is to achieve distributed construction of multi-granularity semantic trees in the construction system to adapt to heterogeneous edges. The construction system consists of N edge devices and 1 cloud server, denoted as C. The present invention proposes a distributed construction method of a multi-granularity semantic tree called Cultivator. Cultivator generates different data granularities layer by layer, corresponding to models of different scales, and constructs a hierarchical tree structure called MGTree; that is, a multi-granularity semantic tree.

[0085] 1) MGTree in the system: The specific architecture of the semantic tree built by Cultivator is as follows Figure 1 The present invention needs to generate a multi-granularity semantic tree, called MGtree with L layers, that is, The number of tree nodes in the lth layer is denoted as L l Without losing generality, the present invention assumes that the finest granularity of semantic labels constitutes layer 0 of the MGtree. As the number of layers increases, the granularity of semantic labels decreases.

[0086] 2) Side: For For each edge device, the present invention defines As a local sample data set, M n ∈N + is the number of samples. The text label set corresponding to the l-th level granularity is recorded as The present invention uses a sample encoder g(·) to generate sample embedding For each label The present invention constructs a descriptive sentence (e.g., "a Then the text encoder z(·) is used to get the corresponding text embedding in For the edge The present invention defines a local granularity generation model. It maps labels from the lth layer granularity to the l+1th layer. The global granularity generation model obtained by aggregating the parameters of the local granularity generation model interprets and processes the labels of the current layer to generate labels with a higher semantic information layer. The generated labels will be used by upper-layer applications or other edge devices. At the same time, there is a model designed specifically for sample-related tasks (such as image recognition tasks), with model parameters θ n express.

[0087] 3) Cloud Server: The cloud server collects a public dataset It contains representative samples from each edge device. To protect privacy, edge devices only transmit samples and text embeddings. Let M C ∈N + express The present invention defines As a collection of sample embeddings, as the text embedding of the l-th level granularity label, where Indicates the corresponding text label in MGtree.

[0088] 2. Interaction process in the construction system

[0089] The distributed generation of multi-granularity semantic trees in the construction system involves data and model interactions, such as Figure 1 Here, the present invention takes the construction of the data granularity from the first layer to the second layer of MGTree as an example:

[0090] 1) Data interaction: The data interaction in the method of the present invention includes the following two main parts:

[0091] ① Multimodal data fusion: cloud server embeds samples and text labels and Collection in the form of

[0092] Edge Devices public dataset information to create an edge-cloud information fusion space.

[0093] ② Information interaction for guiding models: In the edge cloud information fusion space, the present invention adopts a clustering algorithm

[0094] Get the global initial sample center Where d is the embedding dimension and k is the number of clusters. Then the cloud server constructs an appropriate semantic space To capture the deep semantics of the sample. and text embedding in semantic space Distribute to edge devices and establish an edge-cloud collaborative interaction space. This approach ensures that edge devices initialize their granular generative models with collaborative multimodal information (Cultivato r ), and optimize under the guidance of unified semantics.

[0095] 2) Model interaction: Model interaction is achieved through the following main components:

[0096] ③Edge-cloud collaborative learning: In the edge-cloud collaborative interaction space, the present invention adopts the FL framework for joint training

[0097] The second layer of granular generation model. Let E denote the total number of iterations of global training. For each iteration e∈{1,…,E}, edge devices locally optimize their granular generation model parameters Φ n

[0098] And transmit it to the cloud server, which aggregates these parameters using the FedAvg strategy. After E rounds of iterations, the present invention obtains a global granularity generation model Φ. The granularity generation model integrates the granularity mining capabilities of distributed edge devices and maps the first layer of data to the second layer of labels, i.e.

[0099] 3. Multimodal Clustering Based on Hypergraph

[0100] When computational efficiency is a priority and the feature structure of the data is very simple, traditional K-means is a viable alternative to HGC. However, in real-world scenarios, many datasets are heterogeneous and high-dimensional, making it difficult to process chaotic and complex information. In order to handle complex multimodal relationships in large-scale datasets, the present invention uses HGC on a cloud server, such as Figure 2 As shown. The present invention defines a weighted hypergraph in is a set of vertices, where each vertex represents a common sample. is a hyperedge set, satisfying Super Edge E e The relevant vertices for ∈ε are The union of all vertices associated with a hyperedge is equivalent to the set except, is a weight function that assigns positive real weights to hyperedges The correlation matrix Defined by the function h:

[0101]

[0102] Where: h v,e is the element in the vth row and eth column of the matrix H. The present invention defines the hyperedge E e The degree is This formula enables the present invention to model complex high-order relationships between samples, thereby promoting more effective clustering in multimodal space.

[0103] The present invention uses Gaussian kernel function to define the hyperedge E e Hyperedge weight of:

[0104]

[0105] Where: V i ,V j ∈E e Represents the hyperedge E e All the different pairs of vertices in σ e YesE e The median of all pairwise distances between vertices in . is a distance metric for computing vertex similarity. This formula ensures that hyperedges connecting closely related vertices receive higher weights, thereby emphasizing strong local relationships in the multimodal feature space.

[0106] Based on the hypergraph introduced above The present invention constructs two types of homogeneous hyperedges: sample hyperedge set ε I and the text hyperedge set ε T To model the multimodal relationship between samples and their semantics. For each vertex The present invention is structured as follows:

[0107] Sample-based embedding By taking V v Its nearest top-k I Neighbor vertices are connected to obtain sample hyperedges formal,

[0108] Text hyperedge By embedding V based on the similarity of text embedding at the current input layer l data granularity v Its top-k T The nearest neighbor vertices are connected. The form is

[0109] The present invention respectively I and ε T Normalize the weights of . Given the association matrix H, define The degree is:

[0110]

[0111] Among them, ∈∈[0,1] balances the sample and text hyperedge weights.

[0112] make and Diagonal matrices representing vertex degree and hyperedge degree, respectively.

[0113] W = diag(w(E 1 ),…,w(E |ε| )) represents the diagonal matrix of hyperedge weights. The present invention can define the normalized Laplace matrix of the hypergraph as:

[0114]

[0115] In order to cluster from the lth layer to the l+1th layer, the present invention uses spectral clustering on Λ. Specifically, the present invention calculates the feature vector L corresponding to Λ l+1 The smallest non-zero eigenvalues. These eigenvectors form the spectral embedding matrix Represents the low-dimensional code of the hypergraph structure. Then the present invention Apply K-means clustering to obtain L in the lth layer l+1 The initial clusters are denoted as

[0116] 1) Semantic space determination: The cloud server constructs a semantic space in the edge-cloud information fusion space to facilitate sample-semantic mapping and restrict the semantic labels of upper tree nodes. The method of the present invention uses a vocabulary database containing a wide range of word domains as a semantic dataset, such as WordNet, a vocabulary database containing more than 82,000 nouns. Figure 3 As shown, in order to select the most relevant nouns in a given data set, the present invention proposes a two-step filtering process:

[0117] In order to construct the l+1th layer of labels based on the lth layer of labels, the present invention uses the text labels currently input to the lth layer As an effective sample descriptor.

[0118] Search for synonyms and hypernyms in the word list to expand this set and create an initial semantic space

[0119] The present invention is based on the initial cluster center O l Further improved on the basis of Exploit Faiss

[0120] The present invention is based on Select the closest γ nouns for each cluster center, that is:

[0121]

[0122] where sim(·,·) is a similarity measure based on Euclidean distance. z(t) represents The present invention then forms the final semantic space of the lth layer based on the union of these nearest neighbor words, i.e. Where T is the total number of noun phrases selected from WordNet.

[0123] For each noun The present invention constructs a descriptive sentence template "a photo of a ”, and obtained its semantic embedding 2) Multimodal consistency learning: In the edge-cloud collaborative learning process, in order to mine higher-level data granularity, the edge uses multimodal consistency learning to locally train the granularity generation model, such as Figure 4 In addition, the global granularity generation model parameter Φ is generated through edge-cloud collaborative training. For edge device n, with parameter Φ n The granularity generator embeds the sample into u n,i Mapping to soft cluster assignment probabilities Right now At this stage, only the network parameters Φ are updated n , while the CLIP encoders g(·) and z(·) remain frozen.

[0124] a. Balanced Regularized Consistency Learning: For better clustering, the present invention introduces the manifold hypothesis, that is, if two samples are located in the local neighborhood of each other's low-dimensional manifold, they will have similar soft cluster assignments. Based on the manifold hypothesis, the present invention defines the sample consistency loss as:

[0125]

[0126] where <·,·> represents the dot product, and Nκ(x n,i ) is the sample x n,i The sample set of κ nearest neighbors, that is

[0127] In addition, in order to avoid the granular generation model assigning all samples to a single cluster, the present invention adds a balanced regularization term and introduces the commonly used negative entropy loss. The present invention defines the sample consistency loss with the balanced regularization term as:

[0128]

[0129] in is the balanced regularization term. in Represents sample x n,i The probability of being assigned to cluster c. β is a trade-off parameter for balancing regularization. Sample consistency loss Indicates the consistency between adjacent samples in the sample dimension. However, some of these samples may not belong to the same semantic cluster, which may lead to clustering results that are visually similar but semantically different, thus causing confusion.

[0130] b. Semantic Enhanced Consistency Learning: In order to address the limitations of sample-based consistency learning, this paper proposes a semantic space based on construction This method uses the relationship between sample embedding and semantic embedding to generate meaningful semantically enhanced pseudo labels to supplement and guide the clustering self-supervised learning process.

[0131] Given a sample set middle The soft cluster allocation result of the present invention first selects the top-ξ samples for each cluster and stores the selection results as a binary value matrix If you select a sample As a cluster c∈{1, ···, L l+1}, then B n The element B in row i and column c n,i,c Equal to 1; otherwise B n,i,c = 0. Sample center set Calculated as:

[0132] To obtain the semantic center set The present invention converts the sample center Map it to The nearest semantic embedding in

[0133]

[0134] The present invention will Embedded with samples The mapping between them is formulated as an OT problem to generate pseudo labels.

[0135] set up is the cost matrix. The present invention defines the transmission cost as Each semantic center in The negative cosine similarity between samples of

[0136] Then the present invention solves the entropy regularized OT problem:

[0137]

[0138] Where ∏(a,b) is a set of effective transportation plans that satisfy the flow constraints, H(τ) is the entropy regularizer, and v>

[0139] 0 is the regularization parameter. 1 k represents a k-dimensional vector of 1, where + is edge constraint, generally uniform distribution.

[0140] After obtaining the plan τ, normalize its rows to obtain the sample set x n Pseudo-labels The cth element of is calculated as τ i,c / ∑ c τ i,c , indicating u n,i arrive The soft clustering probability of the mapping. For the entropy regularized OT problem defined above, the present invention uses the Sinkhorn-Knopp algorithm for efficient solution. Based on pseudo labels The present invention defines semantic enhancement in self-supervised learning

[0141] loss:

[0142]

[0143] Where CE(·) is the cross entropy loss function. The proposed loss is used to align samples and their semantics.

[0144] Therefore, the present invention defines the network from the perspective of balancing the regularization consistency loss and the semantic enhancement loss.

[0145] for:

[0146]

[0147] Where α is a trade-off parameter. Before self-supervised training, the present invention uses the initial HGC results to provide effective guidance for the initial state of unsupervised clustering, ensuring that the self-supervised process follows the correct trajectory and accelerating model convergence. The weight of the initial cultivation network in the present invention is W l =2ψO l , the bias is Where ψ is the temperature parameter, O l is the initial HGC result of layer l.

[0148] Edge-cloud collaborative learning: Each edge device trains a local model based on the multimodal consistency learning designed above. In order to integrate edge data and build a comprehensive multi-granularity semantic tree, the present invention uses a FL framework in the granularity generation model. The process is as follows: In each global iteration e∈{1,…,

[0149] In E}, each edge device n uses local data and Training parameters The goal of edge device n is to update the parameters by optimizing the consistency loss function

[0150]

[0151] Where η is the learning rate and g∈{1,…,G} is the number of local iterations. At the end of each global iteration e, the edge device sets its local model parameters Upload to the cloud. The cloud uses the FedAvg policy

[0152] These local parameters are roughly aggregated to update the global model parameters:

[0153]

[0154] in is the number of data on edge device n, is the total number of data on all edges. Then the updated global parameter φ e+1 Distribute it back to the edge as the initial state for the next local training. This iterative process continues until the global model parameters converge. Each global iteration integrates distributed edge device data, optimizes the global model, and gradually builds a robust MGTree.

[0155] Example:

[0156] The present invention intends to use this technology to achieve the matching requirements of generating data granularity to meet models of different sizes. The present invention takes this distributed construction method of multi-granularity semantic trees, Cultivator, as an example, and compares it with five centralized clustering methods: SIC, SCAN, NNM, CC and Cultivator (centralized), to obtain the technical advantages of this solution. It shows that by utilizing multi-dimensional information fusion and cross-model interaction and hypergraph-based clustering, Cultivator solves the challenges of incomplete and uneven generation.

[0157] Although the present invention has been described above, the present invention is not limited to the above-mentioned specific embodiments. The above-mentioned specific embodiments are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can make many modifications without departing from the purpose of the present invention, which are all within the protection of the present invention.

Claims

1. A multi-granularity semantic tree construction system for distributed large model training, the construction system comprising a cloud server and an edge device; characterized in that: The construction system includes edge-cloud information fusion space, model information interaction space, edge-cloud collaborative interaction space, multi-granularity semantic tree and global granularity generation model; wherein: The edge-cloud information fusion space embeds the samples and text labels of the cloud server into and Collect edge devices in the form of Multimodal data fusion based on public data set information; The model information interaction space embeds the samples and text labels of the cloud server into and The hypergraph clustering algorithm is used to obtain the global initial cluster center O l ∈R d×k ; The cloud server is based on the initial cluster center O l Filtering words in WordNet to build semantic space The cloud server sets the initial cluster center O l and text embedding in semantic space Distribute to each edge device to build edge-cloud collaborative interaction space; The cloud server sets the initial cluster center O l Send it to each edge device to initialize the granularity generation model of each edge device; The edge-cloud collaborative interaction space embeds text in the semantic space It is sent to each edge device to guide the subsequent local learning and training of edge devices to build a global granularity generation model, namely: Among them: it maps labels from the lth level granularity to the l+1th level; represents the sample embedding on edge device n, represents the l-th level text label embedding of samples on edge device n, is the l+1th level text label of the sample on edge device n; φ n That is, the local granularity generation model parameters for building a multi-granularity semantic tree deployed on the edge device n.

2. The multi-granularity semantic tree construction system for distributed large model training according to claim 1 is characterized by: The construction system consists of N edge devices and 1 cloud server, which are respectively and C; The multi-granularity semantic tree uses distributed layer-by-layer generation of different data granularities, and constructs an L-layer tree structure corresponding to models of different scales; that is, Among them: The number of tree nodes in the lth layer is recorded as L l ; The edge device is Each edge device, i.e. As a local sample data set, where: M n ∈N + The text label set corresponding to the number of samples at the l-th level granularity is recorded as The edge device uses a sample encoder g(·) to generate a sample embedding The edge device has a Use the text encoder z(·) to get the corresponding text embedding in: For the edge The cloud server collects a public data set Where: contains representative samples from each edge device; M C ∈N + express The number of samples in ; As a collection of sample embeddings, As text embeddings of level l granularity labels, where: Represents the text label corresponding to the sample at the lth level of the multi-granularity semantic tree.

3. The multi-granularity semantic tree construction system for distributed large model training according to claim 1 is characterized in that: The model information interaction space embeds the samples and text labels of the cloud server into and The hypergraph clustering algorithm is used to obtain the global initial cluster center O l ∈R d×k Process; including: Based on sample embedding By changing V v Its nearest top-k I Neighbor vertices are connected to obtain sample hyperedges Right now:, Text Hyperedge By embedding V based on the similarity of text embedding at the current input layer l data granularity v Its top-k T The nearest neighbor vertices are connected; that is: According to the following formula, ε I and ε T Normalize the weights of The degree is: Among them, ∈∈[0, 1] balances the sample and text hyperedge weights; The normalized Laplacian matrix of the hypergraph is constructed according to the following formula: Among them: and The diagonal matrices representing vertex degree and hyperedge degree respectively; W = diag(w(E1),…,w(E |ε| )) represents the diagonal matrix of hyperedge weights; Cluster from layer l to layer l+1, use spectral clustering on ∧; and calculate the eigenvector L corresponding to ∧ l+1 smallest non-zero eigenvalue; These eigenvectors form the spectral embedding matrix A low-dimensional encoding that represents the structure of a hypergraph; right Apply K-means clustering to obtain L in the lth layer l+1 The initial clusters are denoted as 4. The multi-granularity semantic tree construction system for distributed large model training according to claim 1 is characterized in that: The cloud server sets the initial cluster center O l Sent to each edge device to initialize the granularity generation model of each edge device; including: Construct the l+1th layer of labels based on the lth layer of labels, using the text labels currently input to the lth layer Get sample descriptor; This set is expanded by searching for synonyms and hypernyms in WordNet, creating an initial semantic space. Through the initial cluster center O l Based on the Faiss library, Among them: Based on Select the closest γ nouns for each cluster center, that is: Among them: sim(·,·) is the similarity measure based on Euclidean distance; z(t) represents Semantic embedding of The final semantic space of the lth layer is formed based on the union of these nearest neighbor words, that is, Where: T is the total number of noun phrases selected from WordNet; For each noun Construct a descriptive sentence template "a photo of a ”, obtain semantic embedding 5. The multi-granularity semantic tree construction system for distributed large model training according to claim 1 is characterized in that: The edge-cloud collaborative interaction space embeds text into It is sent to each edge device to guide the subsequent local learning and training of edge devices to build a global granularity generation model process; including: Jointly balance the regularized consistency loss to train local multi-granularity semantic trees; The FL framework is used to jointly train the upper layer of the multi-granularity semantic tree; The edge device optimizes the parameters in its granular generation model locally according to the following loss function and transmits it to the cloud server; Where: η is the learning rate. At the end of each global iteration e, the edge device will Upload to the cloud server; The cloud server aggregates these parameters using the FedAvg method and builds a global granular generation model through E rounds of iterations, and updates the model parameters. in: is the number of data on edge device n, is the total amount of data on all edge devices; The global granularity generation model takes the first layer of data as input and finally maps the lth layer of data to the l+1th layer of labels, that is, Realize heterogeneous edge device model interaction.

6. The multi-granularity semantic tree construction system for distributed large model training according to claim 5 is characterized by: The edge-cloud collaborative interaction space trains a local multi-granularity semantic tree process by jointly balancing the regularized consistency loss, including: The parameter Φ in the model is generated by the granularity n Will embed u n，i Mapping to soft cluster assignment probabilities Right now According to the relationship between sample embedding and semantic embedding, the pseudo labels of the sample set are generated by normalization: in: The cth element of is calculated as τ i，c / ∑ c τ i，c , indicating u n，i arrive Soft clustering probabilities of mapping; The semantic enhancement loss function is obtained according to the pseudo-label according to the following formula; Where: CE(·) is the cross entropy loss function; M n represents the number of local samples of edge device n, Represents the embedding u n，i To the semantic center The pseudo-labels formed by mapping, Represents the embedding u n，i Soft cluster allocation result mapped by local granularity generator; According to the semantic enhancement loss function, the balanced regularization loss function is obtained according to the following formula; Among them: α is a trade-off parameter.

7. A method for constructing a system using a multi-granularity semantic tree for distributed large model training, characterized in that: The method is based on the system implementation of any one of claims 1-6 to build system data and model interaction.

Images