Distributed model distribution method and distribution network system for cloud-assisted transparent compression

Abstract

The application relates to a cloud-assisted transparent compression distributed model distribution method and a distribution network system. The method comprises the following steps: preparing an AI model on the cloud side, generating a file of the AI model, and publishing the file of the AI model on the cloud side; the cloud side cooperates to calculate a difference part of the AI model which needs to be updated; the cloud side performs transparent compression on the difference part of the AI model; the cloud side cooperates to transmit the compressed difference part of the AI model to the terminal side through a P2P multi-path transmission mode; and the terminal side decompresses the received compressed difference part of the AI model and then assembles the same part locally to form a complete updated model. The method can realize rapid updating and upgrading of the terminal side model, adapt to the network and energy consumption constraints of the terminal side, and optimize the transmission process through transparent compression and multi-path transmission cooperation of the cloud side.

Description

Technical Field

[0001] This application relates to the field of model distribution technology, and in particular to a cloud-assisted transparent compression distributed model distribution method and distribution network system. Background Technology

[0002] Intelligent inference on edge nodes is becoming the next "blue ocean" in intelligent computing. Utilizing the limited resources of edge nodes for local intelligent inference can reduce response latency. After cloud-side models are upgraded or fine-tuned, they need to be quickly distributed to edge node devices.

[0003] Currently, model distribution is mainly achieved through direct push from the cloud side, which has the advantages of simplicity and convenience. However, as the scale of model parameters continues to grow, the waiting time and bandwidth overhead for direct transmission of model files also increase. Furthermore, each time model parameters are upgraded or updated, there is redundancy between the cloud-side and edge-side models, and redundancy also exists between edge-side models. Therefore, transmitting redundant parts wastes cloud-side bandwidth resources and does not fully utilize redundant data between edge-side models to accelerate transmission. Simultaneously, the unstable network or low power consumption constraints on the edge side limit the model distribution efficiency of edge-side nodes, necessitating acceleration of edge-side node model distribution efficiency. Summary of the Invention

[0004] Therefore, it is necessary to provide a distributed model distribution method and distribution network system with cloud-assisted transparent compression to address the aforementioned technical problems.

[0005] A distributed model distribution method with cloud-assisted transparent compression, the method comprising:

[0006] The cloud-side nodes prepare the AI ​​model, generate the AI ​​model file, and then publish the AI ​​model file.

[0007] The differences in computation between cloud-side nodes and edge-side nodes require updating the AI ​​model.

[0008] The nodes on the cloud side transparently compress the differences in the AI ​​model.

[0009] The cloud-side nodes transmit the differences in the compressed AI model to the edge-side nodes via P2P multipath transmission.

[0010] The edge node decompresses the differences in the received compressed AI model and then assembles them with the same parts locally to form a complete updated model.

[0011] In one embodiment, the cloud-side node prepares the AI ​​model and generates the AI ​​model file. The cloud side then places the AI ​​model file in the cloud. Prior to this step, a system initialization step is also included, specifically:

[0012] The cloud-side server and the client-side connect to the network, preparing the local computing, storage, and network environment, and starting the corresponding model publisher, model compressor, model forwarder, and model subscriber processes; among which:

[0013] The model publisher is used to prepare and publish AI models and is currently in the sending state.

[0014] The model compressor is used to calculate model differences and compress them. It is in a forwarding state and decouples the model publisher and model receiver through a proxy.

[0015] The model forwarder is used to optimize transmission latency through multipath transmission and is in a forwarding state.

[0016] Model subscribers are used to receive, decompress, and assemble models, and are in receive, sleep, and forward states.

[0017] In one embodiment, the cloud-side nodes and the edge-side nodes compute the differences in the AI ​​model that need to be updated, including:

[0018] The publisher of the model and the subscriber of each model calculate the difference between the AI ​​model in the cloud and the local model by comparing the tensor differences of the model. If there is a difference between the bucket arrays of the remote and local models, the tensor needs to be transmitted; otherwise, the tensor is skipped.

[0019] In one embodiment, the difference portion of the AI ​​model includes several tensors to be transmitted.

[0020] The cloud-side nodes transparently compress the differences in the AI ​​model, including:

[0021] The cloud-based model compressor uses transparent compression to compress each tensor to be transmitted. Transparent compression involves using a set of bucket vectors to concatenate the tensor to the bucket vectors, then using sparse encoding on the bucket vectors to include only the top-k elements. At the same time, quantization is used to represent each non-zero value with low-bit quantization. The compressed model is then stored in the file system via pickle serialization.

[0022] In one embodiment, the cloud-side node transmits the differences in the compressed AI model to the edge-side node via P2P multipath transmission, including:

[0023] The cloud side initiates the BitTorrent transmission protocol, and the nodes on the edge side join the BitTorrent protocol, transmitting the differences in the compressed AI model to the edge nodes through the collaboration of the edge nodes.

[0024] In one embodiment, the edge node decompresses the differing portions of the received compressed AI model and then assembles them with the local identical portions to form a complete updated model, including:

[0025] The model subscribers of the nodes on the edge perform reverse decompression of the sparsed and quantized bucket vectors to recover each tensor, and combine it with the local unupdated part to form the dictionary of the AI ​​model, which is then stored in a local file through pickle serialization.

[0026] In one embodiment, the AI ​​model is organized as a dictionary, including metadata for each level of the model and the corresponding parameter tensors.

[0027] A cloud-assisted transparent compression distributed model distribution network system includes: model publishers, model compressors, model forwarders, and model subscribers.

[0028] The model publisher is a node on the cloud side, and the node is in a sending state, used to publish the model.

[0029] The model compressor consists of dedicated cloud-side sites, with nodes in a forwarding state used for model compression.

[0030] The model forwarder consists of nodes on the end side, providing multipath coverage network routing. The nodes are in a forwarding state and are used to forward the model.

[0031] Model subscribers consist of nodes on the endpoint. The states of the nodes include receiving, sleeping, and forwarding, which are used to upgrade and update the model.

[0032] During model distribution, the model publisher prepares the AI ​​model, generates the AI ​​model file, and publishes the AI ​​model file; the model publisher and each model subscriber collaboratively calculate the differences in the AI ​​model that need to be updated, and the model compressor transparently compresses the differences in the AI ​​model; the cloud-side node transmits the compressed differences in the AI ​​model to the edge-side node via P2P multipath transmission; the edge-side node decompresses the received compressed differences in the AI ​​model and then assembles them with the local identical parts to form a complete updated model.

[0033] In one embodiment, the sleep state is only responsible for metadata communication and not for model distribution, while the forwarding state can be used in conjunction with sending and receiving.

[0034] In one embodiment, the cloud-side node transmits the differences in the compressed AI model to the edge-side node via P2P multipath transmission, including:

[0035] The cloud side initiates the BitTorrent transmission protocol, and the nodes on the edge side join the BitTorrent protocol, transmitting the differences in the compressed AI model to the edge nodes through the collaboration of the edge nodes.

[0036] The aforementioned cloud-assisted transparent compression distributed model distribution method and distribution network system include the following steps: cloud-side preparation of an AI model, generation of an AI model file, and cloud-side publication of the AI ​​model file; cloud-side collaborative computation of the differences in the AI ​​model requiring updates; cloud-side transparent compression of the differences in the AI ​​model; cloud-side collaborative transmission of the compressed differences in the AI ​​model to the endpoint via P2P multipath transmission; and endpoint decompression of the received compressed differences in the AI ​​model, followed by assembly with the local identical parts to form a complete updated model. This method enables rapid updates and upgrades of the endpoint model, adapts to network and energy consumption constraints on the endpoint, and optimizes the transmission process through cloud-based transparent compression and multipath transmission. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating a distributed model distribution method with cloud-assisted transparent compression in one embodiment.

[0038] Figure 2 This is a flowchart illustrating a distributed model of cloud-assisted transparent compression in one embodiment. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0040] In one embodiment, such as Figure 2 As shown, a distributed model distribution method with cloud-assisted transparent compression is provided, which includes the following steps:

[0041] Step 100: The cloud-side node prepares the AI ​​model, generates the AI ​​model file, and publishes the AI ​​model file.

[0042] Specifically, the AI ​​model is organized as a dictionary, which includes metadata for each level of the model and the corresponding parameter tensors.

[0043] Nodes: Composed of participating processes or nodes, responsible for upgrading and updating the model, and participating in the distribution of the model.

[0044] Node states include sending, receiving, sleeping, and forwarding states. The sleeping state is only responsible for metadata communication and not for model distribution. The forwarding state can be used in conjunction with sending and receiving states.

[0045] Step 102: The nodes on the cloud side and the nodes on the edge side calculate the differences in the AI ​​model that need to be updated.

[0046] Step 104: The cloud-side nodes transparently compress the differences in the AI ​​model.

[0047] Step 106: The cloud-side node transmits the differences in the compressed AI model to the edge-side node via P2P multipath transmission.

[0048] Step 108: The edge node decompresses the difference portion of the received compressed AI model and then assembles it with the local identical portion to form a complete updated model.

[0049] The aforementioned cloud-assisted transparent compression distributed model distribution method includes: preparing an AI model on the cloud side, generating an AI model file, and publishing the AI ​​model file on the cloud side; collaboratively computing the differences in the AI ​​model that need updating on the cloud side; transparently compressing the differences in the AI ​​model on the cloud side; collaboratively transmitting the compressed differences in the AI ​​model to the endpoint via P2P multipath transmission on the cloud side; and decompressing the received compressed differences in the AI ​​model on the endpoint, then assembling it with the local identical parts to form a complete updated model. This method enables rapid updates and upgrades of the endpoint model, adapts to network and energy consumption constraints on the endpoint, and optimizes the transmission process through cloud-based transparent compression and multipath transmission.

[0050] In one embodiment, a system initialization step is included before step 100, specifically including:

[0051] The cloud-side server and the client-side connect to the network, prepare the local computing, storage, and network environment, and start the corresponding model publisher, model compressor, model forwarder, and model subscriber processes. Among them: the model publisher is used to prepare and publish AI models and is in the sending state; the model compressor is used to calculate model differences and compress them and is in the forwarding state, decoupling the model publisher and model receiver through a proxy; the model forwarder is used to optimize transmission latency through multi-path transmission and is in the forwarding state; the model subscriber is used to receive, decompress, and assemble models and is in the receiving, sleeping, and forwarding states.

[0052] In one embodiment, step 102 includes: the publisher of the model and the subscriber of each model calculate the difference between the AI ​​models in the cloud and the local model by comparing the tensor differences of the models (for the two tensors to be compared, each initializes a bucket array of length m (hyperparameter, default set to 1024), maps each element of the tensor to a bucket of the array using the same hash function (the key is the position index of the element, ensuring that elements at the same position are mapped to the same bucket), the bucket uses the summation (SUM) operator to aggregate the elements inserted into the bucket, and then swaps the two bucket arrays). If the bucket arrays of the remote and the local models differ, the tensor needs to be transmitted; otherwise, the tensor is skipped.

[0053] In one embodiment, the difference portion of the AI ​​model includes several tensors to be transmitted; step 104 includes: the cloud-side model compressor compresses each tensor to be transmitted using a transparent compression method. The transparent compression method refers to using a set of bucket vectors to concatenate the tensors to the bucket vectors, and then using a sparse encoding method on the bucket vectors to include only the top-k elements (top-k is a hyperparameter, which by default retains the first 5% of the elements). At the same time, quantization is used to represent each non-zero value through low-bit quantization; the compressed model is then stored in the file system via pickle serialization.

[0054] In one embodiment, step 106 includes: the cloud side starts the BitTorrent transmission protocol, the node on the end side joins the BitTorrent protocol, and the difference part of the compressed AI model is transmitted to the node on the end side through the cooperation of the node on the end side.

[0055] In one embodiment, step 108 includes: the model subscriber of the node on the end side decompresses the sparsed and quantized bucket vectors in reverse to recover each tensor, and combines it with the local unupdated part to form the dictionary of the AI ​​model, which is then stored in a local file via pickle serialization.

[0056] In one embodiment, the AI ​​model is organized as a dictionary, including metadata for each level of the model and the corresponding parameter tensors.

[0057] In one specific embodiment, the steps of the distributed model distribution method with cloud-assisted transparent compression include:

[0058] a) System initialization: Cloud-side servers and edge nodes connect to the network, prepare local computing, storage, and network environments, and start the necessary model publisher, model compressor, model forwarder, and model subscriber processes;

[0059] b) Model preparation: The AI ​​model is prepared on the cloud side, the AI ​​model file is generated, and the AI ​​model is released on the cloud side.

[0060] c) Model differences: Cloud-based collaborative computing requires updating the differences in the model; only the differences are transmitted.

[0061] d) Model compression: The cloud side transparently compresses the differences in the AI ​​model to be transmitted, and the publisher does not need to worry about the compression.

[0062] e) Model transmission: Cloud-based collaborative model transmission optimizes transmission latency through P2P multipath transmission;

[0063] f) Decompression and assembly: The end-side receives the differences in the model, decompresses it, and then assembles it with the same parts locally into a complete updated model.

[0064] In another specific embodiment, taking the distribution of the llama3 model as an example, the steps of the model distribution method are described in detail.

[0065] Step S1, Model Preparation: Prepare the llama3 model, generate the model file, and place the model file in the cloud.

[0066] 1) The model publisher prepares and publishes the AI ​​model, which is located on the cloud side and is in the sending state;

[0067] 2) The model compressor calculates and compresses model differences. It is located on the cloud side and is in a forwarding state. It decouples the model publisher and the model receiver through a proxy and passes the compressed model difference data to the model forwarder.

[0068] 3) Model forwarder, located on the cloud side or the edge side, optimizes transmission latency through multi-path transmission and is in forwarding state. It uses the BitTorrent protocol by default, but can also use other P2P model file transmission protocols.

[0069] 4) Model subscribers, located on the end-side nodes, are responsible for receiving, decompressing, and assembling models. They are in receiving, sleeping, and forwarding states and use the BitTorrent protocol to download model files by default.

[0070] Step S2, Model Difference: The publisher of the model and each subscriber of the model calculate the difference between the cloud and local models. The calculation adopts the tensor difference comparison method. For the two tensors to be compared, each initializes a bucket array of length m (hyperparameter, default setting is 1024). Each element of the tensor is mapped to a bucket in the array using the same hash function (the key is the position index of the element, ensuring that elements at the same position are mapped to the same bucket). The elements inserted into the bucket are aggregated using the summation (SUM) operator. Then the two bucket arrays are swapped. If the average relative difference of the buckets with the same index in the remote and local bucket arrays (average relative difference = average(|bucket(A,i)-bucket(B,i)|max(bucket(A,i),bucket(B,i)), for i ∈[1,m])) exceeds 10% (hyperparameter, which can be adjusted according to the application needs. The closer it is to zero, the more sensitive it is to changes in difference), a significant difference has occurred. Then the tensor needs to be transmitted. Otherwise, the tensor is skipped to reduce data communication volume and improve distribution efficiency.

[0071] The steps for calculating the differences between tensors between two nodes include:

[0072] Step S2.1: Input the length m of the bucket array, where m is an integer and the default value is 1024; set the difference judgment threshold, where threshold is a floating-point number and the default value is 0.1, i.e. 10%.

[0073] Step S2.2: Calculate the local bucket array for each node. Specifically: Initialize the local bucket array to 0; traverse the local tensor elements and map them to buckets: for each element index and corresponding value, calculate the hash value and accumulate the element value into the corresponding bucket. The formula for calculating the hash value is:

[0074] bucket_index = hash(index) % m

[0075] Here, hash() is a hash function that maps elements at the same position to the same bucket;

[0076] Step S2.3: Nodes exchange bucket arrays. Specifically: Each node sends its own local_buckets to the other node; and receives the bucket array from the other node, denoted as remote_buckets.

[0077] Step S2.4: Calculate the average relative difference for each node. Specifically: calculate the relative difference for each bucket and initialize the relative difference array to a 0 array; for each bucket index i (i ∈ [0, m-1]), perform the following operations: obtain the values ​​of the local and remote buckets, calculate the maximum value based on the local and remote bucket values, if the maximum value is greater than 0, calculate the relative difference; otherwise, set the corresponding value in the relative difference array to 0; calculate the average relative difference based on the relative difference array.

[0078] Step S2.5: Determine if the difference exceeds the threshold. Specifically, compare the average relative difference with a preset threshold. If the average relative difference is greater than the preset threshold, the difference is considered significant, and True is returned; otherwise, False is returned.

[0079] Step S2.6: Return the difference judgment result and output a Boolean value indicating whether the difference between the local and remote tensors significantly exceeds the threshold.

[0080] Step S3, Model Compression: For each tensor to be transmitted, transparent compression is applied. Specifically, a set of bucket vectors is used to concatenate the tensor to the bucket vectors. Then, sparse encoding is applied to the bucket vectors, including only the top-k elements. Simultaneously, quantization is used to represent each non-zero value with low-bit quantization, further reducing storage overhead. The compressed model is then serialized to the file system using pickle.

[0081] The specific steps for model compression include:

[0082] Step S3.1: Obtain the dictionary representation of model differences, diff(dict); initialize the compressed difference dictionary, compressed_diff, to empty;

[0083] Step S3.2: Traverse each tensor in the dictionary representation of model differences, flatten the tensor into a one-dimensional vector, then perform sparse encoding, retaining the top-k elements, quantize the sparse encoding result, and store the quantized data in the compressed difference dictionary. Specifically:

[0084] Step S3.2.1: Flatten the tensor into a one-dimensional vector:

[0085] bucket_vector = tensor.flatten(tensor)

[0086] Where bucket_vector is a one-dimensional vector, and tensor.flatten() is a function that flattens a tensor into a one-dimensional vector in row-major order;

[0087] Step S3.2.2: Sparse coding, retaining the top-k elements. Specifically, obtain the indices of the k elements with the largest absolute values, obtain the values ​​of the elements corresponding to these indices, and construct sparse coded data based on the indices of the k elements with the largest absolute values ​​and the values ​​of the corresponding elements;

[0088] indices = np.argpartition(np.abs(bucket_vector), -k)[-k:]

[0089] values ​​= bucket_vector[indices]

[0090] sparse_encoded = {'indices': indices, 'values': values}

[0091] Here, sparse_encoded is the sparse encoded data, values ​​are the indices of the k elements with the largest absolute values ​​and their corresponding values, np.abs() is a function that calculates the absolute value of each element bitwise and returns the result, np.argpartition() is a function that sorts the input array and returns the k largest sorted indices, and indices are the indices of the k elements with the largest absolute values.

[0092] Step S3.2.3: Quantize the sparse coding result. Specifically, this includes:

[0093] Step S3.2.3.1: Obtain the maximum absolute value of the values ​​in the sparsely encoded data:

[0094] max_val = np.max(np.abs(sparse_encoded['values']))

[0095] Among them, max_val is the maximum absolute value, np.max() is the function to select the maximum value in the array, and sparse_encoded['values'] is the value in the sparsely encoded data.

[0096] Step S3.2.3.2: Calculate the scaling ratio based on the maximum absolute value max_val and the number of bits quantized:

[0097] scale = (2 bits - 1) / max_val

[0098] Where scale is the scaling ratio and bits is the number of bits for quantization. Preferably, when bits=8, the values ​​in the sparsely encoded data are scaled to the range of 0-255.

[0099] Step S3.2.3.3: Determine the quantization value based on the scaling ratio:

[0100] quantized_values ​​= np.round(sparse_encoded['values'] * scale).astype(np.int8)

[0101] Here, quantized_values ​​are the quantized values, np.round() is the function for rounding to the nearest integer, astype() is the function for returning an array of encoded integers based on the input integer encoding pattern, and np.int8 is the 8-bit signed integer encoding pattern.

[0102] Step S3.2.3.4: Based on the index, quantized_values, and scaling factor in the sparsely encoded data, construct the quantized data as follows:

[0103] quantized = {'indices': sparse_encoded['indices'], 'values': quantized_values, 'scale': scale}

[0104] Here, quantized refers to the quantified data, which includes 'indices', 'values', and 'scale'.

[0105] Step S3.2.4: Store the quantized data in a compressed difference dictionary.

[0106] compressed_diff[layer] = quantized

[0107] Where compressed_diff is the compressed difference dictionary data to be serialized; layer is the hierarchical index of the model.

[0108] Step S3.3: Serialization compression difference dictionary:

[0109] bytes =pickle_serialize(compressed_diff)

[0110] Where bytes is the serialized data, and pickle_serialize() is a function that serializes the compressed difference dictionary data to be serialized according to the pickle pattern.

[0111] Step S4, Model Transmission: The cloud side initiates the BitTorrent transmission protocol, and the end side joins the BitTorrent protocol. The compressed model to be transmitted is accelerated through the collaboration of the end side.

[0112] The specific steps for transmitting compressed model differences using the BitTorrent protocol include:

[0113] Step S4.1: Obtain the compressed model difference data compressed_diff (bytes), and set the list of paths (list) for saving the data on the edge device (model_diff_path (string): the path to save the model difference file, for example, ". / model_diff.bin", torrent_file_path (string): the path to save the torrent file, for example, ". / model_diff.torrent").

[0114] Step S4.2: Save the compressed model difference data (compressed_diff) to a file;

[0115] Step S4.3: Create a seed file, encode the seed file, and save it to a file. Specifically, this includes: creating a file storage object, adding the model difference file to the file storage object, creating a seed object based on the file storage object, adding a tracker (called Tracker in BitTorrent) to the seed object, calculating the hash value of the seed object, generating the seed file, and finally encoding and saving the seed file to a file.

[0116] Step S4.4: Start the seed. Specifically, create a session object, set the listening port range, obtain the seed file information, add the seed file information to the created session, and specify the save path.

[0117] Step S4.5: The cloud waits for the end-device to connect and download. Keep the cloud program running, monitor the upload status, and ensure that the model difference file continues to be provided as a seed for download.

[0118] Step S4.6: Add the client device to the BitTorrent protocol. Specifically, set the save path column on the client device, create a session object and set the listening port, read the torrent file and add it to the session, and then wait for the download to complete.

[0119] Step S5, Decompression and Assembly: The endpoint decompresses the sparsed and quantized bucket vectors in reverse, recovering each tensor, and combines it with the locally unupdated portion to form the AI ​​model's dictionary, which is then serialized to a local file using pickle. The specific steps for decompressing and assembling the model include:

[0120] Step S5.1: Obtain the compressed model difference data and the dictionary representation of the local model;

[0121] Step S5.2: Use the pickle_deserialize() function to deserialize the compressed model difference data;

[0122] Step S5.3: Traverse the model difference data of each layer, and dequantize the quantized data of each layer of the compressed model difference data after deserialization:

[0123] values ​​= quantized['values'].astype(np.float32) / scale

[0124] Then construct sparse coded data:

[0125] dec_sparse_encoded = {'indices': quantized['indices'], 'values':values}

[0126] Where dec_sparse_encoded is the sparse encoding dictionary, indices are the indices of the sparse elements, and values ​​are the value vectors of the sparse elements.

[0127] Step S5.4: Decode the sparse coded data constructed in step S5.3. Specifically, this includes:

[0128] Step S5.4.1: Obtain sparse encoded data dec_sparse_encoded (dict), which contains 'indices' and 'values', and set the size of the decoded tensor.

[0129] Step S5.4.2: Initialize bucket_vector1 as an all-zero vector, and fill the vector with sparse encoded data:

[0130] bucket_vector1[dec_sparse_encoded['indices']] = dec_sparse_encoded['values']

[0131] Where bucket_vector1 is the decoded vector.

[0132] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0133] In one embodiment, a cloud-assisted transparent compression distributed model distribution network system is provided, comprising: model publishers, model compressors, model forwarders, and model subscribers, wherein:

[0134] The model publisher is a node on the cloud side, and the node is in a sending state, used to publish the model.

[0135] The model compressor consists of dedicated cloud-side sites, with nodes in a forwarding state used for model compression.

[0136] The model forwarder consists of nodes on the end side, providing multipath coverage network routing. The nodes are in a forwarding state and are used to forward the model.

[0137] Model subscribers consist of nodes on the endpoint. The states of the nodes include receiving, sleeping, and forwarding, which are used to upgrade and update the model.

[0138] During model distribution, the model publisher prepares the AI ​​model, generates the AI ​​model file, and publishes the AI ​​model file; the model publisher and each model subscriber collaboratively calculate the differences in the AI ​​model that need to be updated, and the model compressor transparently compresses the differences in the AI ​​model; the cloud-side node transmits the compressed differences in the AI ​​model to the edge-side node via P2P multipath transmission; the edge-side node decompresses the received compressed differences in the AI ​​model and then assembles them with the local identical parts to form a complete updated model.

[0139] In one embodiment, the sleep state is only responsible for metadata communication and not for model distribution, while the forwarding state can be used in conjunction with sending and receiving.

[0140] In one embodiment, the cloud-side node transmits the differences in the compressed AI model to the edge-side node via P2P multipath transmission, including:

[0141] The cloud side initiates the BitTorrent transmission protocol, and the nodes on the edge side join the BitTorrent protocol, transmitting the differences in the compressed AI model to the edge nodes through the collaboration of the edge nodes.

[0142] Specific limitations regarding the cloud-assisted transparent compression distributed model distribution network system can be found in the limitations of the cloud-assisted transparent compression distributed model distribution method described above, and will not be repeated here. Each module in the aforementioned cloud-assisted transparent compression distributed model distribution network system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0143] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A distributed model distribution method with cloud-assisted transparent compression, characterized in that, The method includes: The cloud-side nodes prepare the AI ​​model, generate the AI ​​model file, and then publish the AI ​​model file. The differences between cloud-side nodes and edge-side nodes need to be updated in the AI ​​model computation; specifically, this includes: initializing a bucket array for the tensors of the AI ​​models in the cloud and on-premises, mapping each element of the tensor to a bucket using the same hash function and aggregating them using an accumulation operator, exchanging bucket arrays between nodes, and if the average relative difference between buckets with the same index in the cloud and on-premises bucket arrays exceeds a preset threshold, then the tensor needs to be transmitted; otherwise, the tensor is skipped; the differences in the AI ​​model include several tensors to be transmitted. The cloud-side nodes transparently compress the differences in the AI ​​model; specifically, the cloud-side model compressor transparently compresses each tensor to be transmitted; the transparent compression refers to concatenating the tensor into a bucket vector, using sparsity coding to retain the top-k elements of the bucket vector, and performing low-bit quantization on each non-zero value. The cloud-side nodes transmit the differences in the compressed AI model to the edge-side nodes via P2P multipath transmission. The edge node decompresses the differences in the received compressed AI model and then assembles them with the same parts locally to form a complete updated model.

2. The distributed model distribution method with cloud-assisted transparent compression according to claim 1, characterized in that, The cloud-side nodes prepare the AI ​​model and generate the AI ​​model file. The cloud-side nodes then place the AI ​​model file on the cloud. Before this process, a system initialization step is also included, specifically: The cloud-side server and the client-side connect to the network, preparing the local computing, storage, and network environment, and starting the corresponding model publisher, model compressor, model forwarder, and model subscriber processes; among which: The model publisher is used to prepare and publish AI models and is currently in the sending state; The model compressor is used to calculate model differences and compress them. It is in a forwarding state and decouples the model publisher and model receiver through a proxy. The model forwarder is used to optimize transmission latency through multipath transmission and is in a forwarding state; The model subscriber is used to receive, decompress, and assemble models, and is in a receiving, sleeping, and forwarding state.

3. The distributed model distribution method with cloud-assisted transparent compression according to claim 1, characterized in that, The cloud-side nodes transmit the differences in the compressed AI model to the edge-side nodes via P2P multipath transmission, including: The cloud side initiates the BitTorrent transmission protocol, and the nodes on the edge side join the BitTorrent protocol, transmitting the differences in the compressed AI model to the edge nodes through the collaboration of the edge nodes.

4. The distributed model distribution method with cloud-assisted transparent compression according to claim 1, characterized in that, The edge nodes decompress the differences in the received compressed AI model and then assemble them with the local identical parts to form a complete updated model, including: The model subscribers of the nodes on the edge perform reverse decompression of the sparsed and quantized bucket vectors to recover each tensor, and combine it with the local unupdated part to form the dictionary of the AI ​​model, which is then stored in a local file through pickle serialization.

5. The distributed model distribution method with cloud-assisted transparent compression according to claim 1, characterized in that, AI models are organized as dictionaries, including metadata for each level of the model and the corresponding parameter tensors.

6. A distributed model distribution network system with cloud-assisted transparent compression, characterized in that, The system includes: model publishers, model compressors, model forwarders, and model subscribers; The model publisher is a node on the cloud side, and the node is in a sending state, used to publish the model; The model compressor consists of dedicated cloud-side sites, with nodes in a forwarding state used for model compression. The model forwarder consists of nodes on the end side, providing multipath coverage network routing. The nodes are in a forwarding state and are used to forward the model. The model subscriber consists of nodes on the end side. The node states include receiving, sleeping, and forwarding, which are used to upgrade and update the model. During model distribution, the model publisher prepares the AI ​​model, generates the AI ​​model file, and publishes the AI ​​model file; the model publisher and each model subscriber collaboratively calculate the differences in the AI ​​model that need to be updated, and the model compressor transparently compresses the differences in the AI ​​model; the cloud-side node transmits the compressed differences in the AI ​​model to the edge-side node via P2P multipath transmission; the edge-side node decompresses the received compressed differences in the AI ​​model and then assembles them with the local identical parts to form a complete updated model; The process involves the publisher of the model and the subscribers of each model collaboratively calculating the differences in the AI ​​model that need to be updated. This includes: cloud-side nodes and edge-side nodes calculating the differences in the AI ​​model that need to be updated by comparing the tensor differences of the models; specifically, this includes: initializing a bucket array for the tensors of the AI ​​models in the cloud and on-premises, mapping each element of the tensor to a bucket using the same hash function and aggregating them using an accumulation operator, exchanging bucket arrays between nodes, and if the average relative difference between buckets with the same index in the bucket arrays of the cloud and on-premises exceeds a preset threshold, then the tensor needs to be transmitted; otherwise, the tensor is skipped. The differences in the AI ​​model include several tensors to be transmitted. The model compressor performs transparent compression on the differences in the AI ​​model, including: the model compressor performs transparent compression on each tensor to be transmitted; the transparent compression includes: the cloud-side model compressor concatenates the tensors to be transmitted into a bucket vector, uses sparsity coding to retain the top-k elements of the bucket vector, and performs low-bit quantization on each non-zero value.

7. The distributed model distribution network system with cloud-assisted transparent compression according to claim 6, characterized in that, The sleep state is only responsible for metadata communication and not for model distribution. The forwarding state can be used in conjunction with sending and receiving.

8. The distributed model distribution network system with cloud-assisted transparent compression according to claim 6, characterized in that, The cloud-side nodes transmit the differences in the compressed AI model to the edge-side nodes via P2P multipath transmission, including: The cloud side initiates the BitTorrent transmission protocol, and the nodes on the edge side join the BitTorrent protocol, transmitting the differences in the compressed AI model to the edge nodes through the collaboration of the edge nodes.

Images