Model file transmission method, device, computer equipment and storage medium

Abstract

The present application relates to a model file transmission method, device, computer equipment and storage medium. The method includes: obtaining an edge cloud model file to be distributed; compressing and segmenting the edge cloud model file in sequence to obtain at least two sub-files and a set of download addresses of the sub-files; generating node download policy data of the sub-files according to the node information and download address set of the edge node; sending the node download policy data to each node at the edge, triggering each node to download the sub-file based on the node download policy data, and combining each sub-file to complete the transmission of the edge cloud model file. The use of this method can improve the transmission efficiency of the model file.

Description

Technical Field

[0001] The present application relates to the field of edge cloud technology, and in particular to a model file transmission method, device, computer equipment and storage medium. Background Art

[0002] Container technology has a series of advantages such as second-level delivery, efficient deployment, elastic scaling, and resource scheduling. Edge cloud deploys edge environment with limited resources and runs AI offline most of the time. In addition to the container image environment, the AI ​​operation process also requires model files. Generally, model files are obtained through model training, which are generally above 1G (may exceed 10G) and have low security requirements. In the edge cloud scenario, the network line between the edge cloud center and each edge terminal is often a VPN or dedicated line (generally 2M-6M, mostly 2-4M), so it faces the scenario of model file distribution in the edge cloud.

[0003] However, the existing model file transmission is to pack the model file into the container image, distribute the image to the edge warehouse, directly distribute the model file (such as through a dedicated line or a self-built public network download server) or use the traditional CDN mode (Content Delivery Network), resulting in low transmission efficiency of the model file. Summary of the invention

[0004] Based on this, it is necessary to provide a model file transmission method, device, computer equipment and storage medium that can improve the model file transmission efficiency in response to the above technical problems.

[0005] A model file transmission method, the method comprising:

[0006] Get the edge cloud model file to be distributed;

[0007] The edge cloud model file is compressed and split in sequence to obtain at least two sub-files and a set of download addresses of the sub-files;

[0008] Generate node download strategy data of the sub-file according to the node information and download address set of the edge node;

[0009] The node download strategy data is sent to each node at the edge end, triggering each node to download sub-files based on the node download strategy data, and combining each sub-file to complete the transmission of the edge cloud model file.

[0010] In one embodiment, the edge cloud model file is compressed and split in sequence to obtain at least two sub-files and a set of download addresses of the sub-files, including:

[0011] Get the file properties of the edge cloud model file;

[0012] When the file attribute is an incremental update attribute, the edge cloud model file is differentially processed to obtain the corresponding difference file;

[0013] Compressing and splitting the difference file in sequence to obtain at least two sub-files;

[0014] The sub-files are uploaded to a source server of a content distribution network, and a download address of each sub-file is generated to obtain a download address set.

[0015] In one embodiment, generating node download strategy data of a sub-file according to node information and a download address set of an edge node includes:

[0016] Determine the number of addresses in the download address set, the number of nodes in the node information, and the priority of the node identification of each node;

[0017] According to the number of addresses and the number of nodes, download addresses are allocated to each node in turn from the download address set according to the priority until the download addresses in the download address set are allocated, thereby obtaining the node download strategy data of the sub-file.

[0018] In one embodiment, according to the number of addresses and the number of nodes, download addresses are allocated to each node in sequence from the download address set according to the priority until all download addresses in the download address set are allocated, including:

[0019] When the number of addresses is greater than the number of nodes, download addresses are allocated to each node in batches according to the number of nodes based on the priority of the node identifier until all download addresses in the download address set are allocated.

[0020] In one embodiment, the node download strategy data is sent to each node at the edge end, each node is triggered to download a sub-file based on the download strategy data, and each sub-file is combined to complete the transmission of the edge cloud model file, including:

[0021] Based on the priority of each node identifier, each node is triggered to combine and decompress the downloaded sub-files to obtain the target edge model file;

[0022] The distribution instruction is triggered by the target node, and the target edge model file is sent to the file storage of each corresponding node based on the distribution instruction to complete the transmission of the edge cloud model file.

[0023] In one embodiment, a distribution instruction is triggered by a target node, and a target edge model file is sent to a file storage of each corresponding node based on the distribution instruction to complete the transmission of the edge cloud model file, including:

[0024] Get the file properties of the target edge model file;

[0025] When it is detected that the file attribute is a difference attribute, the target edge model file is subjected to inverse difference processing to obtain a merged file;

[0026] The distribution instruction is triggered by the target node, and the merged file is sent to the file storage of each corresponding node based on the distribution instruction to complete the transmission of the edge cloud model file.

[0027] A model file transmission device, the device comprising:

[0028] A file acquisition module, used to acquire edge cloud model files to be distributed;

[0029] A file processing module, used to compress and split the edge cloud model file in sequence to obtain at least two sub-files and a set of download addresses of the sub-files;

[0030] A strategy generation module, used to generate node download strategy data of a sub-file according to the node information and download address set of the edge node;

[0031] A file download module, used to send node download policy data to each node at the edge end, triggering each node to download a sub-file based on the node download policy data;

[0032] The first transmission module is used to combine the sub-files to complete the transmission of the edge cloud model file.

[0033] A model file transmission method, the method comprising:

[0034] Receive node download strategy data sent by the server; the node download strategy data is generated according to the node information of each edge terminal and the download address set of the sub-file; the sub-file is obtained by sequentially compressing and splitting the edge cloud model file to be distributed;

[0035] Based on the node download strategy data, the corresponding sub-files are downloaded by each node, and the sub-files are combined to complete the transmission of the edge cloud model file.

[0036] In one embodiment, based on the node download strategy data, each node downloads the corresponding sub-file, and each sub-file is combined to complete the transmission of the edge cloud model file, including:

[0037] Based on the node download strategy data, the corresponding sub-files are downloaded through each node pair;

[0038] According to the priority of the node identifier of each node, the sub-files are combined and decompressed to obtain the target edge model file;

[0039] According to the distribution instruction triggered by the target node, the target edge model file is sent to the file storage of the corresponding node to complete the transmission of the edge cloud model file.

[0040] A model file transmission device, the device comprising:

[0041] The data receiving module is used to receive the node download strategy data sent by the server; the node download strategy data is generated according to the node information of each edge terminal and the download address set of the sub-file; the sub-file is obtained by sequentially compressing and splitting the edge cloud model file to be distributed;

[0042] The second transmission module is used to download the corresponding sub-files through each node based on the node download strategy data, and combine the sub-files to complete the transmission of the edge cloud model file.

[0043] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0044] Get the edge cloud model file to be distributed;

[0045] The edge cloud model file is compressed and split in sequence to obtain at least two sub-files and a set of download addresses of the sub-files;

[0046] Generate node download strategy data of the sub-file according to the node information and download address set of the edge node;

[0047] The node download strategy data is sent to each node at the edge end, triggering each node to download sub-files based on the node download strategy data, and combining each sub-file to complete the transmission of the edge cloud model file.

[0048] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0049] Get the edge cloud model file to be distributed;

[0050] The edge cloud model file is compressed and split in sequence to obtain at least two sub-files and a set of download addresses of the sub-files;

[0051] Generate node download strategy data of the sub-file according to the node information and download address set of the edge node;

[0052] The node download strategy data is sent to each node at the edge end, triggering each node to download sub-files based on the node download strategy data, and combining each sub-file to complete the transmission of the edge cloud model file.

[0053] The above-mentioned model file transmission method, device, computer equipment and storage medium obtain the edge cloud model file to be distributed; compress and split it in sequence to obtain at least two sub-files and a set of download addresses of the sub-files; generate node download policy data of the sub-files according to the node information of the edge end and the set of download addresses; send the node download policy data to each node of the edge end, download the sub-files by each node and combine the downloaded sub-files to complete the transmission of the edge cloud model file; that is, by compressing and splitting the model file, the capacity of each transmitted file can be reduced, and the corresponding download policy data can be generated according to the changes in the node information of the edge end, thereby reducing the time consumed by file transmission and improving the efficiency of file transmission.

[0054] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0055] Receive node download strategy data sent by the server; the node download strategy data is generated according to the node information of each edge terminal and the download address set of the sub-file; the sub-file is obtained by sequentially compressing and splitting the edge cloud model file to be distributed;

[0056] Based on the node download strategy data, the corresponding sub-files are downloaded by each node, and the sub-files are combined to complete the transmission of the edge cloud model file.

[0057] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0058] Receive node download strategy data sent by the server; the node download strategy data is generated according to the node information of each edge terminal and the download address set of the sub-file; the sub-file is obtained by sequentially compressing and splitting the edge cloud model file to be distributed;

[0059] Based on the node download strategy data, the corresponding sub-files are downloaded by each node, and the sub-files are combined to complete the transmission of the edge cloud model file.

[0060] In the above-mentioned model file transmission method, device, computer equipment and storage medium method, the node at each edge end receives the node download strategy data sent by the server end, downloads each sub-file through the download address of each sub-file based on the node strategy download data, and combines the downloaded sub-files to obtain the edge model file to be distributed to complete the transmission of the edge model file, that is, by compressing and dividing the edge model file into sub-files, the single download capacity of the file is reduced, the time spent on file downloading and transmission is reduced, and the efficiency of file transmission is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A diagram showing an application environment of a model file transmission method in an embodiment;

[0062] Figure 2 A schematic diagram of a flow chart of a model file transmission method in one embodiment;

[0063] Figure 3 A schematic diagram of a flow chart of steps of a model file transmission method in an embodiment;

[0064] Figure 4 A schematic diagram of a flow chart of a model file transmission method in another embodiment;

[0065] Figure 5 A schematic diagram of a flow chart of a model file transmission method in another embodiment;

[0066] Figure 6 It is an application schematic diagram of a model file transmission method in an embodiment;

[0067] Figure 7 A structural block diagram of a model file transmission method device in an embodiment;

[0068] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0070] The model file transmission method provided in this application can be applied to Figure 1 In the application environment shown. Among them, the edge terminal 102 communicates with the server 104 through the network. The server 104 obtains the edge cloud model file to be distributed; compresses and splits the edge cloud model file in turn to obtain at least two sub-files and a set of download addresses for the sub-files; generates node download policy data for the sub-files based on the node information and download address set of the edge node; sends the node download policy data to each node of the edge terminal 102, triggers each node to download the sub-file based on the node download policy data, and combines each sub-file to complete the transmission of the edge cloud model file. Among them, the edge terminal 102 can be, but is not limited to, a terminal device distributed on the edge side of the network to provide real-time data processing; the server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0071] In one embodiment, Figure 2As shown, a model file method is provided, which is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0072] Step 202: Obtain the edge cloud model file to be distributed.

[0073] Among them, the edge cloud model file can be but is not limited to trained model files in various application fields (for example, logistics field, power grid field, and image processing field, etc.), such as image recognition model files, logistics order anomaly recognition model files, transformer anomaly detection model files, etc.

[0074] Specifically, a distribution instruction of a model file carrying a model file identifier is received, and the edge cloud model file to be distributed is obtained from the model file storage server according to the model file identifier; wherein the edge cloud model file to be distributed is pre-trained, that is, the user uploads the pre-trained edge cloud model file to the server through the interface, and stores it in the corresponding model file storage server. When the server obtains the edge cloud model file to be distributed, an asynchronous file distribution instruction is generated, and the edge cloud model file to be distributed is asynchronously processed.

[0075] Step 204: compress and split the edge cloud model file in sequence to obtain at least two sub-files and a set of download addresses of the sub-files.

[0076] Specifically, at the server backend, the edge cloud model file is compressed by executing a file compression instruction (for example, a gzip command) to obtain a compressed edge cloud model file; the compressed edge cloud model file is segmented according to the obtained file segmentation instruction to obtain at least two sub-files and a file list corresponding to the sub-files, and the sub-file list is uploaded to the CDN source server to obtain a set of download addresses of the sub-files; wherein the size of the sub-file is less than or equal to 100M.

[0077] Step 206: Generate node download policy data for the sub-file according to the node information of the edge node and the download address set.

[0078] Among them, the node information includes the number of edge nodes and the priority of the node identification. The node identification priority can be pre-set and can be the order of the node identification; the node identifications are 1, 2, 3, 4, and 5 respectively, and the priority of the node identification is the natural order of 1, 2, 3, 4, and 5.

[0079] Specifically, the number of nodes and the priority of the node identifier of the edge node, as well as the number of addresses in the download address set are obtained, and the number of addresses is determined according to the number of sub-files obtained by segmentation; the relationship between the number of nodes and the number of addresses is judged, and when the number of nodes is greater than or equal to the number of download addresses, a download address is allocated to each node from the download address set in turn according to the priority of each node identifier, until the download addresses in the download address set are allocated; when the number of addresses is greater than the number of nodes, a download address is allocated to each node in batches according to the number of nodes based on the priority of the node identifier, until the download addresses in the download address set are allocated, and the node download policy data is obtained.

[0080] Step 208, sending the node download policy data to each node at the edge, triggering each node to download sub-files based on the node download policy data, and combining each sub-file to complete the transmission of the edge cloud model file.

[0081] Specifically, the generated node download strategy data, sub-file assembly order and file md5 are sent to each node of the designated edge end through VPN or dedicated line, triggering each node to download the corresponding sub-file based on the download address in the node download strategy data at the same time; based on the priority of the node identification, the downloaded sub-files are combined to obtain the edge cloud model file to be distributed; that is, the same edge node combination combines the sub-files downloaded by each node into a local area network to obtain the edge cloud model file to be distributed, and completes the transmission of the edge cloud file through the node's intranet.

[0082] Optionally, in one embodiment, after the downloaded sub-files are combined to obtain the edge cloud model file to be distributed, the edge cloud model file is verified by the message digest algorithm (MD5). When the verification passes, the edge network model file is distributed through the intranet to complete the transmission of the edge cloud file.

[0083] In the above-mentioned model file transmission method, the edge cloud model file to be distributed is obtained; compression and segmentation processing are performed in sequence to obtain at least two sub-files and a set of download addresses of the sub-files; node download strategy data of the sub-files is generated according to the node information of the edge end and the download address set; the node download strategy data is sent to each node of the edge end, and the transmission of the edge cloud model file is completed by downloading the sub-files by each node and combining the downloaded sub-files; that is, by compressing and segmenting the model file, the capacity of each transmitted file is reduced, and the corresponding download strategy data can be generated according to the changes in the node information of the edge end, thereby reducing the time consumed in file transmission and improving the efficiency of file transmission.

[0084] In one embodiment, Figure 3 As shown, a model file transmission step is provided, which is applied to Figure 1 The following is an example of a server in the example:

[0085] Step 302: Obtain the edge cloud model file to be distributed and the corresponding file attributes.

[0086] Step 304 , determining whether the file attribute is an incremental update attribute, if so, executing step 306 , otherwise executing step 310 .

[0087] Among them, the incremental update attribute means that the edge cloud model file refers to the update of the uploaded model file, for example, an upgraded edge cloud model file.

[0088] Step 306, performing differential processing on the edge cloud model file to obtain a corresponding difference file.

[0089] Specifically, when the file attribute of the edge cloud model file is an incremental update attribute, the backend stores the latest version of the model file in the model server, and performs differential processing through the open source technology BSDiff (execute command bsdiff oldfile newfilepatchfile) to generate a patch difference file.

[0090] Step 308: compress and split the difference file in sequence to obtain at least two sub-files.

[0091] Specifically, the difference file is compressed at the server backend by executing a file compression instruction (for example, a gzip command) to obtain a compressed difference file; the compressed difference file is split according to the obtained file splitting instruction to obtain at least two sub-files and a file list corresponding to the sub-files.

[0092] Step 310: compress and split the edge cloud model file in sequence to obtain at least two sub-files.

[0093] Step 312: Upload the obtained sub-files to the source server of the content distribution network, generate download addresses for each sub-file, and obtain a set of download addresses.

[0094] Step 314: determine the number of addresses in the download address set, the number of nodes in the node information, and the priority of the node identifier of each node.

[0095] Step 316, according to the number of addresses and the number of nodes, download addresses are allocated to each node in sequence from the download address set according to the priority, until all download addresses in the download address set are allocated, thereby obtaining node download strategy data of the sub-file.

[0096] Specifically, the number of nodes and the priority of the node identifier of the edge node, as well as the number of addresses in the download address set, are obtained, and a node download strategy with the shortest downloading time is generated according to the number of nodes at different edge ends, that is, when the number of nodes is greater than or equal to the number of download addresses, a download address is allocated to each node from the download address set in turn according to the priority of each node identifier, until all the download addresses in the download address set are allocated; when the number of addresses is greater than the number of nodes, a download address is allocated to each node in batches according to the number of nodes based on the priority of the node identifier, until all the download addresses in the download address set are allocated, and the node download strategy data is obtained.

[0097] Step 318, based on the priority of each node identifier, trigger each node to combine and decompress the downloaded sub-files to obtain the target edge model file.

[0098] Optionally, before triggering the node to download the sub-file based on the node download policy data based on the priority of each node identifier, a file verification is performed on each sub-file on each node, wherein the file md5 value of each sub-file is verified n times (n is a positive integer, for example, 3). When the verification passes, each node is triggered to download the corresponding sub-file through the corresponding download address; by verifying each sub-file, the integrity of each sub-file and the data security are ensured.

[0099] Step 320, obtaining the file attributes of the target edge model file.

[0100] Step 322 , determining whether the file attribute of the target edge model file is a difference attribute, if so, executing step 324 , otherwise, executing step 328 .

[0101] Step 324, performing inverse difference processing on the target edge model file to obtain a merged file.

[0102] Step 326, triggering the distribution instruction through the target node, sending the merged file to the file storage of each corresponding node based on the distribution instruction, and completing the transmission of the edge cloud model file.

[0103] Among them, the target node can be but is not limited to a node with the highest priority, for example, a node with a node identifier of 1.

[0104] Specifically, when it is detected that the combined target edge model file is a difference file, that is, patch file processing, the open source technology BSDiff reverse process is used to execute the command: bspatch oldfile newfile2patchfile), and the target edge model file is reversely differentially processed to obtain the merged file patchfile. The newfile2 file is verified by the message digest algorithm (MD5). When the verification passes, the edge network model file is distributed through the intranet and patchfile is deleted to complete the transmission of the edge cloud file.

[0105] Step 328, triggering the distribution instruction through the target node, sending the target edge model file to the file storage of each corresponding node based on the distribution instruction, and completing the transmission of the edge cloud model file.

[0106] In the above-mentioned model file transmission step, by obtaining the edge cloud model file to be distributed and the corresponding file attributes, different processing operations are performed on the edge cloud model file to be distributed according to the file attributes, and the file to be distributed is compressed and split into at least two sub-files, that is, the edge model file is processed by BSDiff technology to reduce the file size that needs to be distributed each time, and improve the internal distribution speed of CDN; according to the number of nodes on the edge and the set of download addresses, a download strategy with the least download time is generated, that is, the edge model file to be distributed is distributed through the node intranet, and the model file is not transmitted through a dedicated line, so as to avoid the impact on the dedicated line network during downloading, improve the distribution success rate and reduce the distribution time, thereby improving the transmission efficiency of the model file.

[0107] In another embodiment, Figure 4 As shown, a model file method is provided, which is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0108] Step 402: Obtain the edge cloud model file to be distributed.

[0109] Step 404, obtaining file attributes of the edge cloud model file.

[0110] Step 406, when the file attribute is an incremental update attribute, differential processing is performed on the edge cloud model file to obtain a corresponding difference file.

[0111] Step 408: compress and split the difference file in sequence to obtain at least two sub-files.

[0112] Step 410: upload the sub-files to the source server of the content distribution network, generate download addresses for each sub-file, and obtain a set of download addresses.

[0113] Step 412: Determine the number of addresses in the download address set, the number of nodes in the node information, and the priority of the node identifier of each node.

[0114] Step 414, according to the number of addresses and the number of nodes, download addresses are allocated to each node in sequence from the download address set according to the priority, until all download addresses in the download address set are allocated, thereby obtaining node download strategy data of the sub-file.

[0115] Step 416, based on the priority of each node identifier, trigger each node to combine and decompress the downloaded sub-files to obtain the target edge model file.

[0116] Specifically, each sub-file is verified on each node, wherein the file md5 of each sub-file is verified n times (n is a positive integer, for example, 3). When the verification passes, based on the priority of each node identifier, each node is triggered to combine and decompress the downloaded sub-files to obtain the target edge model file. That is, by verifying each sub-file, the accuracy and data security of each sub-file are ensured.

[0117] Step 418: When it is detected that the file attribute is a difference attribute, an inverse difference process is performed on the target edge model file to obtain a merged file.

[0118] Optionally, when it is detected that the file attribute is not a difference attribute, the target edge model file is distributed through the node intranet to complete the transmission of the edge cloud model file.

[0119] Step 420, triggering the distribution instruction through the target node, sending the merged file to the file storage of each corresponding node based on the distribution instruction, and completing the transmission of the edge cloud model file.

[0120] Optionally, in one embodiment, the server receives a periodic update instruction, and periodically updates the sub-files in the source server according to the periodic update instruction, thereby improving the utilization rate of the source server.

[0121] In the above-mentioned model file transmission method, the edge cloud model file to be distributed is differentially processed by incrementally updating the attributes to obtain the corresponding difference file, and the difference file is compressed and split in turn to obtain at least two sub-files. The edge model file is processed using BSDiff technology to reduce the file size that needs to be distributed each time and improve the internal distribution speed of CDN; according to the number of nodes on the edge and the set of download addresses, a download strategy with the least download time is generated, that is, the edge model file to be distributed is distributed through the node intranet, and the model file is not transmitted through a dedicated line, so as to avoid the impact on the dedicated line network during downloading, improve the distribution success rate and reduce the distribution time, thereby improving the transmission efficiency of the model file.

[0122] In another embodiment, Figure 5 As shown, a model file transmission method is provided, which is applied to Figure 1 The edge end in FIG. 1 is taken as an example to illustrate, including the following steps:

[0123] Step 502, receiving the node download strategy data sent by the server; the node download strategy data is generated based on the node information of each edge terminal and the download address set of the sub-file; the sub-file is obtained by compressing and splitting the edge cloud model file to be distributed in sequence.

[0124] Specifically, the node download strategy data, sub-file assembly sequence and file md5 sent by the server are received through VPN or dedicated line.

[0125] Among them, the node download policy data is generated by the server obtaining the number of nodes and the priority of the node identification of the edge node, as well as the number of addresses in the download address set, that is, when the number of nodes is greater than or equal to the number of download addresses, the download address is allocated to each node from the download address set in turn according to the priority of each node identification, until the download addresses in the download address set are allocated; when the number of addresses is greater than the number of nodes, the download address is allocated to each node in batches according to the number of nodes based on the priority of the node identification, until the download addresses in the download address set are allocated, and the node download policy data is obtained.

[0126] Step 504: Based on the node download strategy data, the corresponding sub-files are downloaded through each node, and each sub-file is combined to complete the transmission of the edge cloud model file.

[0127] Specifically, based on the node download strategy data, the corresponding sub-files are downloaded through each node; the sub-files are combined and decompressed according to the priority of the node identifier of each node to obtain the target edge model file; the file attributes of the target edge model file are obtained; the file attributes of the target edge model file are detected, and if the file attribute is detected to be a difference attribute, the target edge model file is inversely differentially processed to obtain a merged file; the distribution instruction is triggered through the target node, and the merged file is sent to each corresponding node to complete the transmission of the edge cloud model file; if the file attribute is detected not to be a difference attribute, the distribution instruction is triggered through the target node, and the target edge model file is sent to the file storage of the corresponding node to complete the transmission of the edge cloud model file.

[0128] Optionally, in one embodiment, before downloading sub-files based on the node download policy data, each edge node performs a file verification on the file md5 of each sub-file. When the verification passes, each node is triggered to download the corresponding sub-file through the corresponding download address; by verifying each sub-file, the accuracy and data security of each sub-file are ensured.

[0129] In the above-mentioned model file transmission method, each edge node receives the node download strategy data sent by the server, downloads each sub-file through the download address of each sub-file based on the node strategy download data, and combines the downloaded sub-files to obtain the edge model file to be distributed, and completes the transmission of the edge model file through the node intranet, that is, by compressing and dividing the edge model file into sub-files, the single download capacity of the file is reduced, the time spent on file download and transmission is reduced, and the efficiency of file transmission is improved.

[0130] The following example shows an application scenario of model file transmission: Figure 6 As shown, the server includes a backend server (center), a model storage server (center) and a CDN source server (center); the edge cloud model file input by the user is saved in the backend server; a corresponding model folder is created in the model storage server; a distribution instruction of a model file carrying a model file identifier is received, and the edge cloud model file to be distributed is obtained from the model file storage server according to the model file identifier, that is, a model is selected according to a model file list uploaded by the user, and an edge cloud model to be distributed is obtained, and the processing result of the distribution task saved in the backend server is fed back to the user, and saved in the file to be distributed in the model storage server; the file attribute of the edge cloud model file to be distributed is obtained, and it is judged whether it is If it is not a newly uploaded file, when it is a newly uploaded file, the edge cloud model file is compressed (e.g., patch file compression) and split in sequence to obtain at least two sub-files; if it is not a newly uploaded file, the edge cloud model file is differentially processed to obtain a corresponding difference file, and the difference file is compressed and split in sequence (e.g., BSDiff generates a patch file and then compresses the patch file) to obtain at least two sub-files; the obtained sub-files are uploaded to the CDN source server of the content distribution network, and the download address of each sub-file is generated to obtain a download address set, and the node download strategy data of the sub-file is generated according to the node information of the edge node and the download address set.

[0131] The node download strategy data is sent to the node at the edge end in an asynchronous manner through a dedicated network line. When the first node at the edge end with an order of 1 (i.e., the first node at the edge end) and other nodes receive the node download strategy data, a file check is performed on the sub-file. When the check passes, each node is triggered to download the corresponding sub-file through the corresponding download address, and the downloaded sub-files are combined and decompressed to obtain the target edge model file; when the check fails, the check is performed again, and the number of checks can be limited to but not limited to 3; the file attribute of the target edge model file is obtained, and when the file attribute of the target edge model file is a difference attribute, the target edge model file is inversely differentially processed to obtain a merged file and the merged file is checked. When the check passes, a distribution instruction is triggered through the target node, and the merged file is sent to each corresponding node through the node intranet and saved in a folder to complete the transmission of the edge cloud model file. When the check fails, a prompt is generated to prompt the operator to process; when the file attribute of the target edge model file is not a difference attribute, a distribution instruction is triggered through the target node, and the target edge model file is sent to each corresponding node through the node intranet to complete the transmission of the edge cloud model file.

[0132] That is, by obtaining the edge cloud model file to be distributed and the corresponding file attributes, different processing operations are performed on the edge cloud model file to be distributed according to the file attributes, and the file to be distributed is compressed and split into at least two sub-files, that is, the BSDiff technology is used to process the edge model file, so as to reduce the file size that needs to be distributed each time and improve the internal distribution speed of CDN; according to the number of nodes on the edge and the set of download addresses, a download strategy with the least download time is generated, that is, the edge model file to be distributed is distributed through the node intranet, and the model file is not transmitted through a dedicated line, so as to avoid the impact on the dedicated line network during download, improve the distribution success rate and reduce the distribution time, thereby improving the transmission efficiency of the model file.

[0133] It should be understood that although Figure 2-5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-5 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0134] In one embodiment, Figure 7As shown, a model file transmission device is provided, including: a file acquisition module 702, a file processing module 704, a strategy generation module 706, a file download module 708 and a first transmission module 710, wherein:

[0135] The file acquisition module 702 is used to acquire the edge cloud model file to be distributed.

[0136] The file processing module 704 is used to compress and split the edge cloud model file in sequence to obtain at least two sub-files and a set of download addresses of the sub-files.

[0137] The policy generation module 706 is used to generate node download policy data of the sub-file according to the node information of the edge node and the download address set.

[0138] The file download module 708 is used to send the node download policy data to each node at the edge end, triggering each node to download the sub-file based on the node download policy data.

[0139] The first transmission module 710 is used to combine the sub-files to complete the transmission of the edge cloud model file.

[0140] In the above-mentioned model file transmission device, the edge cloud model file to be distributed is obtained; compression and segmentation processing are performed in sequence to obtain at least two sub-files and a set of download addresses of the sub-files; node download strategy data of the sub-files is generated according to the node information of the edge end and the download address set; the node download strategy data is sent to each node of the edge end, and the transmission of the edge cloud model file is completed by downloading the sub-files by each node and combining the downloaded sub-files; that is, by compressing and segmenting the model file, the capacity of each transmitted file is reduced, and the corresponding download strategy data can be generated according to the changes in the node information of the edge end, thereby reducing the time consumed for file transmission and improving the efficiency of file transmission.

[0141] In another embodiment, a model file transmission device includes, in addition to the file acquisition module 702, the file processing module 704, the policy generation module 706, the file download module 708 and the first transmission module 710, the following:

[0142] The acquisition module 702 is also used to obtain the file attributes of the edge cloud model file.

[0143] The differential processing module is used to perform differential processing on the edge cloud model file when the file attribute is an incremental update attribute to obtain a corresponding difference file.

[0144] The file processing module 704 is also used to compress and split the difference file in sequence to obtain at least two sub-files; upload the sub-files to the source server of the content distribution network, generate download addresses for each sub-file, and obtain a download address set.

[0145] The first determination module is used to determine the number of addresses in the download address set, the number of nodes in the node information, and the priority of the node identifier of each node.

[0146] The strategy generation module 706 is also used to allocate download addresses to each node in sequence from the download address set according to the number of addresses and the number of nodes and according to the priority until all the download addresses in the download address set are allocated, thereby obtaining node download strategy data of the sub-file.

[0147] The strategy generation module 706 is also used to allocate download addresses to each node in batches according to the number of nodes based on the priority of the node identifier when the number of addresses is greater than the number of nodes, until all download addresses in the download address set are allocated.

[0148] The trigger module is used to trigger each node to combine and decompress the downloaded sub-files based on the priority of each node identifier to obtain the target edge model file.

[0149] The first transmission module 710 is also used to trigger a distribution instruction through a target node, and send the target edge model file to the file storage of the corresponding node based on the distribution instruction to complete the transmission of the edge cloud model file.

[0150] The acquisition module 702 is also used to acquire the file attributes of the target edge model file.

[0151] The inverse difference processing module is used to perform inverse difference processing on the target edge model file to obtain a merged file when it is detected that the file attribute is a difference attribute.

[0152] The first transmission module 710 is also used to trigger a distribution instruction through a target node, and send the merged file to the file storage of each corresponding node based on the distribution instruction to complete the transmission of the edge cloud model file.

[0153] In one embodiment, by obtaining the edge cloud model file to be distributed and the corresponding file attributes, different processing operations are performed on the edge cloud model file to be distributed according to the file attributes; if the file attribute of the edge cloud model file is an incremental update attribute, the edge cloud model file is differentially processed to obtain the corresponding difference file, and the difference file is compressed and split in turn to obtain at least two sub-files; if the file attribute of the edge cloud model file is not an incremental update attribute, the edge cloud model file is compressed and split in turn to obtain at least two sub-files, that is, the file to be distributed is compressed and split to generate at least two sub-files, thereby reducing the file size that needs to be distributed each time and improving the internal distribution speed of the CDN; a download strategy with the least download time is generated according to the number of nodes at the edge and the download address set, and the edge model file to be distributed is distributed through the node intranet, and the model file is not transmitted through a dedicated line, so as to avoid the impact on the dedicated line network during download, improve the distribution success rate and reduce the distribution time, thereby improving the transmission efficiency of the model file.

[0154] In another embodiment, a model file transmission device is provided, comprising: a data receiving module and a second transmission module, wherein:

[0155] The data receiving module is used to receive the node download strategy data sent by the server; the node download strategy data is generated based on the node information of each edge terminal and the download address set of the sub-file; the sub-file is obtained by compressing and splitting the edge cloud model file to be distributed in turn.

[0156] The second transmission module is used to download the corresponding sub-files through each node based on the node download strategy data, and combine the sub-files to complete the transmission of the edge cloud model file.

[0157] In one embodiment, a download module is also included, which is used to download corresponding sub-files through each node pair based on the node download policy data.

[0158] In one embodiment, it further includes a combination and decompression processing module, which is used to combine and decompress the sub-files according to the priority of the node identifier of each node to obtain a target edge model file;

[0159] In one embodiment, the second transmission module is also used to send the target edge model file to the file storage of the corresponding node according to the distribution instruction triggered by the target node, thereby completing the transmission of the edge cloud model file.

[0160] The above-mentioned nodes at each edge end receive the node download strategy data sent by the server end, download each sub-file through the download address of each sub-file based on the node strategy download data, and combine the downloaded sub-files to obtain the edge model file to be distributed to complete the transmission of the edge model file, that is, by compressing and dividing the edge model file into sub-files, the single download capacity of the file is reduced, the time spent on file downloading and transmission is reduced, and the efficiency of file transmission is improved.

[0161] For the specific definition of the model file transmission device, please refer to the definition of the model file transmission method above, which will not be repeated here. Each module in the above-mentioned model file transmission device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0162] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store edge cloud model file data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a model file transmission method is implemented.

[0163] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0164] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0165] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0166] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0167] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0168] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A model file transmission method, characterized in that: The method comprises: Get the edge cloud model file to be distributed; Compressing and splitting the edge cloud model file in sequence to obtain at least two sub-files and a set of download addresses of the sub-files; Determine the number of addresses in the download address set and node information of the edge end, wherein the node information includes the priority of the node identifier of each node of the edge end and the number of nodes of the edge end; When the number of addresses is greater than the number of nodes, based on the priority of the node identifier, download addresses are sequentially allocated to each of the nodes in batches according to the number of nodes until the download addresses in the download address set are completely allocated, thereby obtaining the node download strategy data for the sub-file; when the number of nodes is greater than or equal to the number of addresses, download addresses are sequentially allocated to each of the nodes from the download address set according to the priority of the node identifier, until the download addresses in the download address set are completely allocated, thereby obtaining the node download strategy data for the sub-file; Sending the node download strategy data to each node on the edge end, triggering each node to download the sub-file based on the node download strategy data, and combining each sub-file to complete the transmission of the edge cloud model file; The step of sending the node download strategy data to each node at the edge end, triggering each node to download the sub-file based on the download strategy data, and combining each sub-file to complete the transmission of the edge cloud model file includes: Based on the priority of each node identifier, trigger each node to combine and decompress the downloaded sub-files to obtain a target edge model file; The distribution instruction is triggered by the target node, and the target edge model file is sent to the file storage of each corresponding node based on the distribution instruction to complete the transmission of the edge cloud model file, wherein the target node is the node with the highest priority.

2. The method according to claim 1, characterized in that The edge cloud model file is compressed and split in sequence to obtain at least two sub-files and a set of download addresses of the sub-files, including: Obtaining file attributes of the edge cloud model file; When the file attribute is an incremental update attribute, performing differential processing on the edge cloud model file to obtain a corresponding difference file; Compressing and splitting the difference file in sequence to obtain at least two sub-files; The sub-files are uploaded to a source server of a content distribution network, and a download address of each sub-file is generated to obtain a download address set.

3. The method according to claim 1, characterized in that The method of triggering a distribution instruction through a target node and sending the target edge model file to a file storage device of each corresponding node based on the distribution instruction to complete the transmission of the edge cloud model file includes: Obtaining file attributes of the target edge model file; When it is detected that the file attribute is a difference attribute, performing inverse difference processing on the target edge model file to obtain a merged file; The distribution instruction is triggered by the target node, and the merged file is sent to the file storage of each corresponding node based on the distribution instruction to complete the transmission of the edge cloud model file.

4. A model file transmission method, characterized in that: The method comprises: Receive node download strategy data sent by the server; the node download strategy data is obtained by sequentially allocating download addresses to each node in batches according to the number of nodes based on the priority of the node identifier of each node at the edge end when the number of addresses in the download address set of the sub-file is greater than the number of nodes at the edge end, and sequentially allocating download addresses to each node from the download address set according to the priority of the node identifier when the number of nodes is greater than or equal to the number of addresses, until the download addresses in the download address set are completely allocated; the sub-file is obtained by sequentially compressing and splitting the edge cloud model file to be distributed; Based on the node download strategy data, the corresponding sub-files are downloaded by each node, and each sub-file is combined to complete the transmission of the edge cloud model file; The downloading of the corresponding sub-files by each node based on the node download strategy data and combining the sub-files to complete the transmission of the edge cloud model file include: Combining and decompressing the sub-files according to the priority of the node identifier of each node to obtain a target edge model file; According to the distribution instruction triggered by the target node, the target edge model file is sent to the file storage of the corresponding node to complete the transmission of the edge cloud model file, wherein the target node is the node with the highest priority.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images