Large file distribution method and device, equipment, medium and product

Abstract

The invention provides a large file distribution method and device, equipment, a medium and a product, and is applied to the technical field of computers. The method comprises the following steps: acquiring the total number of file fragments of a to-be-distributed large file, and network performance parameters, resource performance parameters and scheduling performance parameters of each container node; determining the performance weight of each container node based on the network performance parameter, the resource performance parameter and the scheduling performance parameter of each container node; and determining the number of the file fragments distributed by each container node based on the performance weight of each container node and the total number of the file fragments. A large file is fragmented to obtain a plurality of small-magnitude file fragments convenient to distribute, the performance weight of each container node is calculated based on the current multi-dimension performance parameters of each container node, and the number of the file fragments borne and distributed by each container node is dynamically determined according to the performance weight. Therefore, the optimal distribution efficiency of the large file in a multi-container node scene is achieved. Therefore, the large file distribution efficiency can be improved.

Description

Technical Field

[0001] This disclosure belongs to the field of computer technology, specifically relating to a method, apparatus, device, medium, and product for distributing large files. Background Technology

[0002] With the development of cloud computing and cloud-native technologies, high-speed and secure distribution of large files to containers in a Kubernetes (K8S) environment can be used to meet needs such as data backup and recovery, data sharing between microservices, CI / CD pipeline integration, and high-performance computing and data analysis. Large file transfers in a K8S environment can significantly improve transfer efficiency and ensure data security.

[0003] In general, large file distribution typically relies on centralized storage and transmission methods. This approach becomes highly complex when dealing with large-scale, distributed storage and transmission needs, hindering rapid file distribution and thus reducing efficiency. Summary of the Invention

[0004] This disclosure is made in view of the above-mentioned problems. This disclosure provides a method, apparatus, device, medium, and product for distributing large files, which can improve the efficiency of large file distribution.

[0005] According to one aspect of this disclosure, a method for distributing large files is provided, comprising:

[0006] Obtain the total number of file fragments of the large file to be distributed, the network performance parameters, resource performance parameters, and scheduling performance parameters of each container node;

[0007] The performance weight of each container node is determined based on the network performance parameters, resource performance parameters, and scheduling performance parameters of each container node.

[0008] The number of file fragments distributed by each container node is determined based on the performance weight of each container node and the total number of file fragments.

[0009] Optionally, obtaining network performance parameters includes:

[0010] The network bandwidth parameters, network stability parameters, and network latency parameters of each container node are obtained. The network bandwidth parameters are related to the historical average bandwidth, historical uplink bandwidth, historical downlink bandwidth, and historical peak bandwidth. The network stability parameters are related to the network packet loss rate parameters and network latency jitter parameters. The network latency parameters are related to the average network latency from the source container node to the target container node and the average network latency from the target container node to the other container nodes.

[0011] The network performance parameters are determined based on the network bandwidth parameters, the network stability parameters, the network latency parameters, and the preset adjustment parameters corresponding to each parameter.

[0012] Optionally, obtaining resource performance parameters includes:

[0013] The CPU performance parameters, memory performance parameters, disk performance parameters, and graphics processor performance parameters of each container node are obtained. The CPU performance parameters are related to CPU utilization, CPU average load, number of CPU cores, and CPU cache size. The memory performance parameters are related to memory utilization, memory bandwidth, and memory access latency. The disk performance parameters are related to disk size, disk data read speed, and disk data write speed. The graphics processor performance parameters are related to the number of graphics processors.

[0014] The resource performance parameters are determined based on the CPU performance parameters, memory performance parameters, disk performance parameters, and graphics processor performance parameters.

[0015] Optionally, obtaining the scheduling performance parameters includes:

[0016] Get the number of container nodes in the worker nodes of the target container node;

[0017] The scheduling performance parameters are determined based on the number of container nodes.

[0018] Optionally, determining the number of file shards distributed by each container node based on the performance weight of each container node and the total number of file shards includes:

[0019] The performance weights of each container node are summed to obtain the total performance weights;

[0020] The number of file fragments distributed by each container node is determined based on the sum of the performance weights, the performance weights of each container node, and the total number of file fragments.

[0021] Optionally, after determining the number of file shards distributed by each container node based on the performance weight of each container node and the total number of file shards, the method further includes:

[0022] When the target container node receives the first task and the network resources of the target container node are insufficient, a network performance adjustment factor is determined based on the bandwidth parameter and peak bandwidth parameter occupied by the first task in the target container node, and a resource performance adjustment factor is determined based on the number of CPUs occupied by the first task in the target container node and the total number of CPUs in the target container node, wherein the priority of the first task is greater than the priority of the distribution task of the large file to be distributed.

[0023] Based on the network performance adjustment factor, the resource performance adjustment factor, and the performance weight, the current performance weight of the target container node is determined.

[0024] Based on the current performance weight and the total number of currently undistributed file shards, the number of current file shards distributed by each container node is determined.

[0025] Optionally, after determining the number of file shards distributed by each container node based on the performance weight of each container node and the total number of file shards, the method further includes:

[0026] If the worker node to which the target container node belongs restarts, obtain the newly generated container node and determine the performance weight of the new container node;

[0027] Based on the performance weight of the new container node, the performance weight of the container node with undistributed file shards, and the total number of undistributed file shards, the number of file shards distributed by the container node for each undistributed file shard and the number of file shards of the new container node are determined.

[0028] According to another aspect of this disclosure, a large file distribution apparatus is provided, comprising:

[0029] The acquisition module is used to obtain the total number of file fragments of the large file to be distributed, the network performance parameters, resource performance parameters and scheduling performance parameters of each container node;

[0030] The first determining module is used to determine the performance weight of each container node based on the network performance parameters, resource performance parameters and scheduling performance parameters of each container node.

[0031] The second determining module is used to determine the number of file fragments distributed by each container node based on the performance weight of each container node and the total number of file fragments.

[0032] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor executing the computer program to implement the above-described large file distribution method.

[0033] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, the program being executed by a processor to implement the above-described large file distribution method.

[0034] According to another aspect of this disclosure, a computer program product is provided, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the above-described large file distribution method.

[0035] This disclosure obtains the total number of file fragments of a large file to be distributed, the network performance parameters, resource performance parameters, and scheduling performance parameters of each container node; based on the network performance parameters, resource performance parameters, and scheduling performance parameters of each container node, the performance weight of each container node is determined; based on the performance weight of each container node and the total number of file fragments, the number of file fragments to be distributed by each container node is determined. By fragmenting the large file, multiple smaller file fragments that are easy to distribute are obtained, and the performance weight of each container node is calculated based on the current performance parameters of each container node. The number of file fragments to be distributed by each container node is dynamically determined according to the performance weight, avoiding the problem that centralized storage and transmission methods cannot effectively manage and distribute large files in real time, thereby achieving the best distribution efficiency of large files in multi-container node scenarios. Therefore, the efficiency of large file distribution can be improved.

[0036] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0037] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0038] Figure 1 A flowchart of a large file distribution method provided in this disclosure.

[0039] Figure 2 The flowchart for node authentication and verification provided in this disclosure.

[0040] Figure 3 This is a schematic diagram of a large file distribution device provided in this disclosure.

[0041] Figure 4 This is a hardware block diagram of an electronic device provided in this disclosure.

[0042] Figure 5 This is a schematic diagram of a computer program product provided in this disclosure. Detailed Implementation

[0043] To enable those skilled in the art to better understand the technical solution of this application, the application scenario of this application will be described first below.

[0044] With the development of cloud computing and cloud-native technologies, high-speed and secure distribution of large files to containers in a Kubernetes (K8S) environment can be used to meet needs such as data backup and recovery, data sharing between microservices, CI / CD pipeline integration, and high-performance computing and data analysis. Large file transfers in a K8S environment can significantly improve transfer efficiency and ensure data security.

[0045] Currently, large file distribution typically relies on centralized storage and transmission methods. This approach is complex for managing large files and cannot quickly achieve file distribution when facing large-scale, distributed storage and transmission needs. For example, during the container image building phase, large files are copied into the image build context using COPY or ADD instructions in the Dockerfile (the text file for building the image) of Docker (the application container engine), directly embedding the required large files into the container image. After image building is complete, the large files become part of the container image and are stored in the container image repository. When the container starts, since the image already contains these large files, no additional network transmission steps are needed, and the container can directly access these files. However, while this method, because the files are directly packaged into the image, ensures that each container instance uses the exact same file version, and deployment is simple without adding external dependencies, it requires the large files to be fixed and explicit. If the content of the large files needs to be updated, or if the content of the large files is dynamically generated, this approach requires rebuilding the image and redeploying the container, impacting business continuity and the efficiency of large file transmission. Alternatively, large files can be effectively managed and accessed by dynamically mounting pre-allocated and managed persistent storage volumes into the container. This approach allows file storage to be independent of the container's lifecycle, supporting data persistence and efficient access. Container applications can access files in the mounted storage volume as if they were local file systems. Any changes to these files are synchronized to the volume, ensuring data consistency and durability. However, this method requires predefined mounted volumes, and for applications with large files that have very high requirements for data read / write speeds, managing local volumes is very complex, has low security, and is inefficient for file distribution in multi-Kubernetes cluster scenarios. Therefore, it reduces the efficiency of large file distribution.

[0046] To address the aforementioned technical issues, this disclosure provides a method, apparatus, device, medium, and product for large file distribution. In this disclosure, large files are fragmented into multiple smaller file fragments that are easier to distribute. A performance weight for each container node is calculated based on multiple dimensions of its current performance parameters. The number of file fragments each container node is responsible for distributing is dynamically determined based on these performance weights. This avoids the problem of centralized storage and transmission methods failing to effectively manage and distribute large files in real time, thereby achieving optimal distribution efficiency for large files across multiple container nodes.

[0047] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0048] Figure 1 A flowchart illustrating a large file distribution method provided in this disclosure. Figure 1 As shown, the method includes:

[0049] S101: Obtain the total number of file fragments of the large file to be distributed, the network performance parameters, resource performance parameters, and scheduling performance parameters of each container node.

[0050] Specifically, the file to be distributed can be fragmented based on its file type to obtain the total number of file fragments for the large file to be distributed. Performance metrics of each container node are monitored, and network performance parameters, resource performance parameters, and scheduling performance parameters for each container node are calculated and determined. Among these, network performance parameters analyze network quality aspects such as bandwidth, latency, and stability; resource performance parameters comprehensively analyze the read, write, and computational performance of the container nodes; and scheduling performance parameters comprehensively analyze the affinity metrics of the container nodes.

[0051] Furthermore, file types include configuration files, binary files, log files, media files, and script files. Different sharding methods can be used for different file types to obtain the total number of file shards. Specifically, a structured sharding method is used for structured data such as configuration files, sharding according to hierarchy. When file content changes, the shards for the changed content are dynamically calculated, allowing the reuse of unchanged shard content. For example, for JSON objects, based on the characteristics of various Kubernetes elements, various resources such as nodes, namespaces, and apps are identified through data structures. Various resource objects can be identified by their kind. Data is then split based on some major common API resource identifiers, API objects, declarative configurations, tags and selectors, dependencies, etc. For binary files, this embodiment uses 5MB as the single shard size for sharding, performing sharding of native large file bytes. This ensures maximum reuse of sharded content, distinguishes between basic and variable files, and reduces interference from additional factors in sharding. For log files, this embodiment also uses 5MB as the segment size for cutting; for media files, such as video files and audio files, this embodiment uses 5MB as the segment size for cutting; for script files, this embodiment uses 1MB as the individual segment size for cutting.

[0052] S102: Determine the performance weight of each container node based on its network performance parameters, resource performance parameters, and scheduling performance parameters.

[0053] Specifically, the performance weight of each container node can be dynamically calculated in real time based on its network performance parameters, resource performance parameters, and scheduling performance parameters. This determines the number of shards each container node is responsible for. In other words, the number of shards distributed is determined based on the current network quality and performance of the container node. Container nodes with better quality can take on more file shards to improve the efficiency of large file distribution.

[0054] S103: Determine the number of file shards distributed to each container node based on the performance weight of each container node and the total number of file shards.

[0055] Specifically, the performance weights of each container node can be summed to obtain a total performance weight. Then, based on the total performance weight, the performance weight of each container node, and the total number of file fragments, the number of file fragments distributed to each container node is determined. This allows large files to be dynamically allocated to distributed containers, optimizing file fragment transmission by using different fragmentation schemes for different types of files. Furthermore, by calculating network performance parameters, resource performance parameters, and scheduling performance parameters of nodes in the network, the performance weights of container nodes in the network are dynamically calculated. This dynamic calculation of the number of fragments for each container node achieves optimal performance for real-time dynamic large file distribution.

[0056] Traditional file distribution mechanisms, such as FTP or HTTP downloads, are inefficient in such distributed, large-scale environments, susceptible to network fluctuations, and struggle to guarantee data security. This disclosure optimizes the efficiency and security of large file distribution in distributed container environments, making it particularly suitable for situations requiring the synchronization of large amounts of data across multiple geographically dispersed data centers or cloud environments. Typical scenarios include large-scale software update distribution, synchronization of large media content, and dataset sharing in big data analytics. This disclosure leverages the advantages of distributed networks, accelerating file transmission through P2P technology, ensuring that each node participates in file distribution, thereby significantly improving overall transmission speed and optimizing network bandwidth usage. A dynamic container node selection mechanism ensures that the optimal transmission path is chosen at any given time, thus enhancing the stability and efficiency of the distribution process.

[0057] This disclosure obtains the total number of file fragments of a large file to be distributed, the network performance parameters, resource performance parameters, and scheduling performance parameters of each container node; based on the network performance parameters, resource performance parameters, and scheduling performance parameters of each container node, the performance weight of each container node is determined; based on the performance weight of each container node and the total number of file fragments, the number of file fragments to be distributed by each container node is determined. By fragmenting the large file, multiple smaller file fragments that are easy to distribute are obtained, and the performance weight of each container node is calculated based on the current performance parameters of each container node. The number of file fragments to be distributed by each container node is dynamically determined according to the performance weight, avoiding the problem that centralized storage and transmission methods cannot effectively manage and distribute large files in real time, thereby achieving the best distribution efficiency of large files in multi-container node scenarios. Therefore, the efficiency of large file distribution can be improved.

[0058] In one possible implementation, an exemplary method for obtaining network performance parameters includes:

[0059] Obtain the network bandwidth, network stability, and network latency parameters for each container node; determine the network performance parameters based on the network bandwidth, network stability, and network latency parameters and the corresponding preset adjustment parameters.

[0060] Specifically, after file splitting, file fragments need to be distributed. Before distribution, network performance metrics for different container nodes need to be calculated. Files with more fragments place higher demands on the network performance of container nodes. In this embodiment, network performance parameters are related to network bandwidth, network stability, and network latency. The following formula is an example formula for calculating network performance parameters:

[0061]

[0062] Where, N P Here, NetworkScore1 is the network bandwidth parameter, NetworkScore2 is the network stability parameter, NetworkScore3 is the network latency parameter, and δ,∈,ζ are preset adjustment parameters. The sum of the preset adjustment parameters is 1. When the network stability parameter indicates network stability, the above formula can calculate the sum of the network bandwidth parameter and the network stability parameter. When the network stability parameter indicates network instability, the above formula can calculate the difference between the network bandwidth parameter and the network stability parameter.

[0063] Furthermore, network bandwidth parameters are related to historical average bandwidth, historical uplink bandwidth, historical downlink bandwidth, and historical peak bandwidth. The following formula is an example formula for calculating network bandwidth parameters:

[0064] NetworkScore1=α*B avg +β*B up +γ*B down +δ*B peak

[0065] Where NetworkScore1 is the network bandwidth parameter, B avg B is the historical average bandwidth over the past seven days. up For the past day's historical uplink bandwidth, B down B represents the historical downlink bandwidth over the past day. peak The bandwidth is the historical peak bandwidth of the past seven days. α, β, γ, and δ are preset weights, and the sum of each weight is 1.

[0066] Specifically, the predicted historical average bandwidth for the past seven days can be calculated using the daily average bandwidth over the past seven days and the bandwidth movement factor predicted based on the ARIMA model. The following formula is an example formula for predicting the historical average bandwidth for the past seven days:

[0067]

[0068] Among them, B avg This is the historical average bandwidth over the past seven days. f is the average daily bandwidth of the i-th day over the past seven days. avg () is the bandwidth mobility factor predicted based on the ARIMA model.

[0069] Specifically, the predicted historical uplink bandwidth for the past day can be calculated using hourly uplink bandwidth data from the past day and an uplink bandwidth movement factor predicted based on the ARIMA model. The following formula is an example formula for predicting historical uplink bandwidth for the past day:

[0070]

[0071] Among them, B up This represents the historical uplink bandwidth for the past day. f represents the average uplink bandwidth in the j-th hour of the past day. up () is the future uplink bandwidth mobility factor predicted based on the ARIMA model.

[0072] Specifically, the predicted historical downlink bandwidth for the past day can be calculated using hourly downlink bandwidth data from the past day and a downlink bandwidth movement factor predicted based on the ARIMA model. The following formula is an example formula for predicting historical downlink bandwidth for the past day:

[0073]

[0074] Among them, B down This represents the historical downlink bandwidth for the past day. f represents the average downlink bandwidth in the j-th hour of the past day. down () is the future downlink bandwidth mobility factor predicted based on the ARIMA model.

[0075] Specifically, the Prophet model can be used to predict the historical peak bandwidth of container nodes over the past seven days. The following formula is an example formula for predicting historical peak bandwidth over the past seven days:

[0076]

[0077] Where g(t) represents the trend, and considering that the peak bandwidth cannot exceed the maximum network capacity of the interconnect in the network, a logistic trend is used for calculation. t is time, k is the growth rate, C is the upper limit of the data, s(t) is the periodic fluctuation, N is the number of terms in the Fourier series, and a n and b n These are Fourier level coefficients, where P is the period, ∈ t This is the error term, representing unidentified noise or anomalies.

[0078] Furthermore, network stability parameters are related to network packet loss rate and network latency jitter. The following formulas are example formulas for calculating network stability parameters:

[0079]

[0080] Where NetworkScore2 is the network stability parameter, θ and μ are the weights, and PL m The average packet loss rate over approximately 60 minutes can be 0.7 in this embodiment. h The average packet loss rate over the past 24 hours can be 0.3 or 0.7 PL in this embodiment. m +0.3PL h J is the network packet loss rate parameter. i The delay is approximately 60 minutes per minute, where J is the average delay. This is a parameter for network latency jitter.

[0081] Furthermore, the network latency parameter is related to the average network latency from the source container node to the target container node and the average network latency from the target container node to all other container nodes. In large file distribution networks, it is necessary to consider not only the network latency from the source node to the target container node, but also the network latency between container nodes, considering that distribution also needs to be done among themselves. Therefore, the latency parameter of a container node consists of two parts: one is the time-averaged network latency parameter from the source node to the container node, and the other is the spatially averaged network latency parameter from the container node to all other container nodes. The following formula is an example formula for calculating the network latency parameter:

[0082]

[0083] Where NetworkScore3 is the network latency parameter, ρ and τ are the weights, and L t This represents the network latency from the source node to the container node over approximately t minutes. L represents the average network latency from the source node to the container node over a 60-minute period. i The network latency from the target container node to other container nodes. The average network latency from the target container node to other container nodes.

[0084] In one possible implementation, an exemplary method for obtaining resource performance parameters includes:

[0085] Obtain the CPU performance parameters, memory performance parameters, disk performance parameters, and GPU performance parameters of each container node; determine the resource performance parameters based on the CPU performance parameters, memory performance parameters, disk performance parameters, and GPU performance parameters.

[0086] Specifically, in this embodiment, resource performance parameters are related to CPU performance parameters, memory performance parameters, disk performance parameters, and graphics processing unit (GPU) performance parameters. In this embodiment, resource performance parameters can be the read / write computation performance parameters of the container node. The following formula is an example formula for calculating resource performance parameters:

[0087] C p =α×C cpu +β×C memory +γ×C disk +δ×C gpu

[0088] Among them, C p For resource performance parameters, C cpu C represents the performance parameters of the central processing unit (CPU) of a container node. memory C represents the memory performance parameters of container nodes. disk For the disk performance parameters of container nodes, C gpu The performance parameters of the graphics processor of the container node are related to the number of graphics processors in this embodiment. The performance parameters of the graphics processor are related to the number of graphics processors. α, β, γ, and δ are weights, which can be 0.25 in this embodiment.

[0089] Furthermore, CPU performance parameters are related to CPU utilization, average CPU load, number of CPU cores, and CPU cache size. The following formulas are example formulas for calculating CPU performance parameters:

[0090] C cpu =α1×C cpu-usage (t)+α2×C cpu-load (t)+α3×C cpu-core +α4

[0091] ×C cpu-cache

[0092] Among them, C cpu For the performance parameters of the central processing unit, C cpu-usage (t) represents the CPU utilization rate, C cpu-load (t) represents the average load of the central processing unit, C cpu-core C is the number of central processing unit cores. cpu-cache α1 represents the CPU cache size, and α2, α3, and α4 represent weights.

[0093] Specifically, the LTSM model is used to predict CPU utilization based on historical data. The following formula is an example formula for predicting CPU utilization:

[0094] C cpu-usage (t)=LSTM(C cpu-usage(t-1), C cpu-usage (t-2)...C cpu-usage (tn))

[0095] Among them, C cpu-usage (t) represents the CPU utilization rate, C cpu-usage (t-1), C cpu-usage (t-2)…C cpu-usage (tn) represents historical data.

[0096] Specifically, the LTSM model is used to predict the average CPU load based on historical data. The following formula is an example formula for predicting the average CPU load:

[0097] C cpu-load (t)=LSTM(C cpu-load (t-1), C cpu-load (t-2)...C cpu-load (tn))

[0098] Among them, C cpu-load (t) represents the average load of the central processing unit, C cpu-load (t-1), C cpu-load (t-2)...C cpu-load (tn) represents historical data.

[0099] Specifically, the number of CPU cores is related to the number of cores in the target container node and the total number of cores across all container nodes. The following formula is an example formula for calculating the number of CPU cores:

[0100]

[0101] Among them, C cpu-core C is the number of central processing unit cores. core C represents the number of cores in the target container node. core-total This represents the total number of cores across all container nodes.

[0102] Specifically, the CPU cache size is related to the cache size of the target container node and the cache size of all container nodes. The following formula is an example formula for calculating the CPU cache size:

[0103]

[0104] Among them, C cpu-cache For the CPU cache size, C cache C represents the cache size of the target container node. cache-total This is the cache size for all container nodes.

[0105] Furthermore, memory performance parameters are related to memory utilization, memory bandwidth, and memory access latency. The following formulas are example formulas for calculating memory performance parameters:

[0106] C memory =β1×C mem-usage (t)+β2×C mem-band +β3×C mem-lant

[0107] Among them, C memory For memory performance parameters, C mem-usage (t) represents memory usage, C mem-band For memory bandwidth, C mem-lant β1, β2, and β3 are the memory access delays and their respective weights.

[0108] Specifically, memory utilization can be predicted based on historical average utilization combined with the LSTM algorithm. The following formula is an example formula for calculating memory utilization:

[0109] C mem-usage (t)=LSTM(C mem-usage (t-1), C mem-usage (t-2)...C mem-usage (tn))

[0110] Among them, C mem-usage (t) represents memory usage, C mem-usage (t-1), C mem-usage (t-2)...C mem-usage (tn) represents the historical average usage rate.

[0111] Specifically, the memory bandwidth is related to the total bandwidth of the target container node and the total memory bandwidth of all container nodes. The following formula is an example formula for calculating the memory bandwidth:

[0112]

[0113] Among them, C mem-band For memory bandwidth, C mem C represents the total bandwidth of the target container node. mem-total This represents the total memory bandwidth for all container nodes.

[0114] Specifically, memory access latency is related to the average access latency over the past 60 minutes. The following formula is an example formula for calculating memory access latency:

[0115]

[0116] Among them, C mem-lant For memory access latency, C mem-lant(t) represents the access delay at minute t within the last 60 minutes.

[0117] Furthermore, disk performance parameters are related to disk size, disk read speed, and disk write speed. The following formulas are example formulas for calculating disk performance parameters:

[0118] C disk =γ1×C disk-size +γ2×C disk-r +γ3×C disk-w

[0119] Among them, C disk For disk performance parameters, C disk-size For disk size, C disk-r For disk data read speed, C disk-w γ1, γ2, and γ3 represent disk data write speeds and are weights.

[0120] Specifically, the disk size is related to the disk size of the target container node and the largest disk size among all container nodes. The following formula is an example formula for calculating disk performance parameters:

[0121]

[0122] Among them, C disk-size For disk size, C disk C represents the disk size of the target container node. disk-max The largest disk among all container nodes.

[0123] Specifically, disk data read speed can be predicted based on the average hourly read speed over the past week combined with the LSTM algorithm. The following formula is an example formula for calculating disk data read speed:

[0124] C disk-r (t)=LSTM(C disk-r (t-1), C disk-r (t-2)...C disk-r (tn))

[0125] Among them, C disk-r (t) represents the disk data read speed, C disk-r (t-1), C disk-r (t-2)...C disk-r (tn) represents the average hourly read speed over the past week.

[0126] Specifically, disk data write speed can be predicted based on the average hourly write speed over the past week combined with the LSTM algorithm. The following formula is an example formula for calculating disk data write speed:

[0127] Cdisk-w (t)=LSTM(C disk-w (t-1), C disk-w (t-2)...C disk-w (tn))

[0128] Among them, C disk-w (t) represents the disk data write speed, C disk-w (t-1), C disk-w (t-2)...C disk-w (tn) represents the average hourly write speed over the past week.

[0129] In one possible implementation, an exemplary method for obtaining scheduling performance parameters includes:

[0130] Obtain the number of container nodes in the worker nodes to which the target container node belongs; determine the scheduling performance parameters based on the number of container nodes.

[0131] Specifically, when multiple container nodes are scheduled onto the same worker node, the communication speed between container nodes can reach its maximum, and the efficiency of synchronizing file shards will be optimal; however, each container node consumes CPU, thus reducing the computing performance of the container nodes. In this embodiment, the scheduling performance parameter is affinity, and the following formula is an example formula for calculating the scheduling performance parameter:

[0132] D p =log(N+1)

[0133] Among them, D p Here, N represents the scheduling performance parameter and the number of container nodes.

[0134] When the number of container nodes is relatively small, changes in the number of container nodes have a significant impact on scheduling performance parameters. This means that when the number of container nodes scheduled on a worker node is small, the impact on the calculation of the performance weight of the container nodes is relatively large. When the number of container nodes is relatively large, changes in the number of container nodes have a smaller impact on scheduling performance parameters. This means that when the number of container nodes scheduled on a worker node is large, the impact on the calculation of the performance weight of the container nodes is reduced.

[0135] In one possible implementation, an exemplary method for determining the number of file shards distributed by each container node based on the performance weight of each container node and the total number of file shards includes:

[0136] The performance weights of each container node are summed to obtain the total performance weight.

[0137] Specifically, based on the network performance parameters, resource performance parameters, and scheduling performance parameters of each container node, the performance weight of each container node can be determined. The following formula is an example formula for calculating the performance weight of each container node:

[0138]

[0139] Where W is the performance weight of each container node, and D p N is a scheduling performance parameter. p For network performance parameters, C p These are resource performance parameters.

[0140] Based on the performance weight of each container node, the sum of the performance weights of all container nodes is determined. The following formula is an example formula for calculating the sum of performance weights:

[0141]

[0142] Among them, W total W is the sum of performance weights. j Let be the performance weight of the j-th container node.

[0143] The number of file shards distributed to each container node is determined based on the sum of performance weights, the performance weights of each container node, and the total number of file shards.

[0144] Specifically, the following formula is an example formula for calculating the number of file fragments distributed to each container node:

[0145]

[0146] Among them, A i The number of file shards distributed by each container node, where A is the total number of file shards, and W is the number of file shards distributed. i W represents the performance weight of each container node. total This represents the sum of performance weights.

[0147] In one possible implementation, after determining the number of file shards distributed to each container node based on the performance weight of each container node and the total number of file shards, the method further includes:

[0148] When the target container node receives the first task and the network resources of the target container node are insufficient, a network performance adjustment factor is determined based on the bandwidth parameters and peak bandwidth parameters occupied by the first task on the target container node, and a resource performance adjustment factor is determined based on the number of CPUs occupied by the first task on the target container node and the total number of CPUs on the target container node.

[0149] Specifically, during the actual transmission of large files, container nodes may also need to perform other tasks, such as monitoring and data collection, command execution, script distribution, and log collection. However, the number of tasks that a container node can execute is limited, and tasks with higher priority need to be prioritized. Therefore, the upload and download speeds of container nodes need to be limited based on task priority and the number of tasks currently being executed.

[0150] In this embodiment, different task priorities are pre-set. When tasks of different priorities are executed simultaneously, the container node's CPU and network resources are prioritized for high-priority tasks, while CPU and network resource usage is limited for low-priority tasks. For example, the task priority order could be monitoring and data collection, command execution, large file distribution, script distribution, and log collection, with the task priorities decreasing in that order.

[0151] When the target container node receives the first task, if the priority of the first task is higher than the priority of the task distributing the large file to be distributed, and the network resources of the target container node are insufficient to guarantee the simultaneous execution of the first task and the task distributing the large file to be distributed, then the current performance weights of each container node need to be recalculated. A network performance adjustment factor can be determined based on the bandwidth and peak bandwidth parameters of the target container node occupied by the first task, thereby adjusting the network performance parameters of the target container node. Similarly, a resource performance adjustment factor can be determined based on the number of CPUs occupied by the first task and the total number of CPUs in the target container node, thereby adjusting the resource performance parameters of the target container node. The following formula is an example formula for calculating the network performance adjustment factor:

[0152]

[0153] Among them, P n B is the network performance adjustment factor, and B′ is the bandwidth parameter of the target container node occupied by the first task. peak This refers to the peak bandwidth parameter.

[0154] The following formula is an example formula for calculating the resource performance adjustment factor:

[0155]

[0156] Among them, P c C′ is the resource performance adjustment factor, where C′ is the number of CPUs occupied by the first task on the target container node, and C is the total number of CPUs on the target container node.

[0157] The current performance weight of the target container node is determined based on the network performance adjustment factor, resource performance adjustment factor, and performance weight.

[0158] Specifically, the following formula is an example formula for calculating the current performance weight:

[0159]

[0160] Among them, W new As the current performance weight, D p N is a scheduling performance parameter. p For network performance parameters, C p P is a resource performance parameter. n P is the network performance adjustment factor. c As a resource performance adjustment factor, To compensate for constants.

[0161] The number of file fragments distributed to each container node is determined based on the current performance weight and the total number of undistributed file fragments.

[0162] Specifically, when the weight of the target container node is updated, the number of shards for each container node also needs to be recalculated. The following formula is an example formula for calculating the current number of file shards distributed to each container node:

[0163]

[0164] Among them, A new-i Let A' be the current number of file fragments in the i-th container node, and W be the total number of currently undistributed file fragments. new-i Let be the current performance weight of the i-th container node. This is the sum of the performance weights of all container nodes.

[0165] In one possible implementation, after determining the number of file shards distributed to each container node based on the performance weight of each container node and the total number of file shards, the method further includes:

[0166] If the worker node to which the target container node belongs restarts, obtain the new container node and determine the performance weight of the new container node; based on the performance weight of the new container node, the performance weight of the container node with undistributed file shards, and the current total number of undistributed file shards, determine the number of file shards distributed by each container node with undistributed file shards and the number of file shards of the new container node.

[0167] Specifically, file fragments of the same large file carry the same identifier to distinguish different files. In this embodiment, each file fragment requires confirmation of receipt after being requested to avoid duplicate requests for file fragments. When concatenating file fragments, the total number of fragments transmitted can be used to determine whether the transmission is complete. During concatenation, the next file fragment is concatenated according to the index position marked on the file fragment, following the index order. Container nodes that fail repeated verification will be marked as untrusted and will not be requested again for a specified period. The checks include whether the container node can be accessed and whether the stored file fragment information is complete. If an abnormal container node status is found during the detection process, the container node will be automatically removed, and its stored file fragment indexes will be reassigned to other container nodes.

[0168] When a target container node is scheduled (i.e., the worker node to which the target container node belongs restarts), the file shards already distributed by the old container node will become invalid. At this point, a new container node needs to be determined, and the new container node should be used for distribution. The number of file shards distributed to each undistributed file shard and the number of file shards on the new container node can be determined based on the performance weight of the new container node, the performance weight of the container nodes that have not yet distributed file shards, and the total number of currently undistributed file shards. Furthermore, when anomalies are detected in the data of a container node, the source file will be re-sharded, and the index data of the problematic file shard will be replaced to ensure that the file data is error-free and can be updated quickly.

[0169] In one possible implementation, the file fragments can also be encrypted when they are distributed.

[0170] Specifically, file distribution needs to take into account the security of fragmented data. In this embodiment, CA (Certificate Authority) certificates and two-way negotiated encryption algorithms are used to ensure the legitimacy of each transmission node in the P2P (peer-to-peer) network. Figure 2 The flowchart for node authentication and verification provided in this disclosure is as follows: Figure 2 As shown. A newly joined node A needs to initiate a validity verification with the CA. After successful verification, the CA generates a certificate based on the node's ID, RSA public key, authorization start time, and authorization end time. The newly joined node A can then join the network and begin communicating with other nodes B. After exchanging certificates, the CA needs to verify the other party's certificate. If the verification passes, the RSA public key encryption algorithm KEY is returned to the other nodes B. Upon receiving the encrypted KEY, the newly joined node A decrypts it using its RSA private key to obtain a symmetric encryption KEY. Subsequently, both parties use the KEY exchanged to encrypt and decrypt messages for communication.

[0171] The added data encryption and access control mechanisms ensure data security during transmission, preventing unauthorized access and data leakage. This solution enables the rapid and secure distribution of large files globally, accelerating business deployment and updates, improving business agility and responsiveness, while reducing the risk of business interruption due to network issues.

[0172] Figure 3 This is a schematic diagram of a large file distribution device provided in this disclosure. Figure 3 As shown, the device 300 includes: an acquisition module 310, a first determination module 320, and a second determination module 330.

[0173] The acquisition module 310 is used to acquire the total number of file fragments of the large file to be distributed, the network performance parameters, resource performance parameters and scheduling performance parameters of each container node;

[0174] The first determining module 320 is used to determine the performance weight of each container node based on the network performance parameters, resource performance parameters and scheduling performance parameters of each container node.

[0175] The second determining module 330 is used to determine the number of file fragments distributed by each container node based on the performance weight of each container node and the total number of file fragments.

[0176] Optionally, the acquisition module is used for:

[0177] The network bandwidth parameters, network stability parameters, and network latency parameters of each container node are obtained. The network bandwidth parameters are related to the historical average bandwidth, historical uplink bandwidth, historical downlink bandwidth, and historical peak bandwidth. The network stability parameters are related to the network packet loss rate parameters and network latency jitter parameters. The network latency parameters are related to the average network latency from the source container node to the target container node and the average network latency from the target container node to the other container nodes.

[0178] The network performance parameters are determined based on the network bandwidth parameters, the network stability parameters, the network latency parameters, and the preset adjustment parameters corresponding to each parameter.

[0179] Optionally, the acquisition module is used for:

[0180] The CPU performance parameters, memory performance parameters, disk performance parameters, and graphics processor performance parameters of each container node are obtained. The CPU performance parameters are related to CPU utilization, CPU average load, number of CPU cores, and CPU cache size. The memory performance parameters are related to memory utilization, memory bandwidth, and memory access latency. The disk performance parameters are related to disk size, disk data read speed, and disk data write speed. The graphics processor performance parameters are related to the number of graphics processors.

[0181] The resource performance parameters are determined based on the CPU performance parameters, memory performance parameters, disk performance parameters, and graphics processor performance parameters.

[0182] Optionally, the acquisition module is used for:

[0183] Get the number of container nodes in the worker nodes of the target container node;

[0184] The scheduling performance parameters are determined based on the number of container nodes.

[0185] Optionally, the second determining module is used to:

[0186] The performance weights of each container node are summed to obtain the total performance weights;

[0187] The number of file fragments distributed by each container node is determined based on the sum of the performance weights, the performance weights of each container node, and the total number of file fragments.

[0188] Optionally, the device further includes:

[0189] After determining the number of file fragments distributed by each container node based on the performance weight of each container node and the total number of file fragments, when the target container node receives the first task and the network resources of the target container node are insufficient, a network performance adjustment factor is determined based on the bandwidth parameter and peak bandwidth parameter occupied by the first task in the target container node, and a resource performance adjustment factor is determined based on the number of CPUs occupied by the first task in the target container node and the total number of CPUs in the target container node. The priority of the first task is greater than the priority of the distribution task of the large file to be distributed.

[0190] Based on the network performance adjustment factor, the resource performance adjustment factor, and the performance weight, the current performance weight of the target container node is determined.

[0191] Based on the current performance weight and the total number of currently undistributed file shards, the number of current file shards distributed by each container node is determined.

[0192] Optionally, the device further includes:

[0193] After determining the number of file shards distributed by each container node based on the performance weight of each container node and the total number of file shards, if the worker node to which the target container node belongs restarts, a new container node is obtained and the performance weight of the new container node is determined.

[0194] Based on the performance weight of the new container node, the performance weight of the container node with undistributed file shards, and the total number of undistributed file shards, the number of file shards distributed by the container node for each undistributed file shard and the number of file shards of the new container node are determined.

[0195] This application also provides an electronic device for performing the above-described large file distribution method. Please refer to... Figure 4 It illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 4 As shown, the electronic device 4 includes: a processor 400, a memory 401, a bus 402, and a communication interface 403. The processor 400, the communication interface 403, and the memory 401 are connected via the bus 402. The memory 401 stores a computer program that can run on the processor 400. When the processor 400 runs the computer program, it executes the large file distribution method provided in any of the foregoing embodiments of this application.

[0196] The memory 401 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this device network element and at least one other network element is achieved through at least one communication interface 403 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0197] Bus 402 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 401 is used to store programs. After receiving an execution instruction, the processor 400 executes the program. The large file distribution method disclosed in any of the foregoing embodiments of this application can be applied to the processor 400, or implemented by the processor 400.

[0198] The processor 400 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 400 or by instructions in software form. The processor 400 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 401. The processor 400 reads the information in memory 401 and, in conjunction with its hardware, completes the steps of the above method.

[0199] The electronic device provided in this application embodiment and the large file distribution method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0200] This application also provides a computer-readable storage medium corresponding to the large file distribution method provided in the foregoing embodiments. The computer-readable storage medium shown can be an optical disc, on which a computer program is stored. When the computer program is run by a processor, it executes the large file distribution method provided in any of the foregoing embodiments.

[0201] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0202] The computer-readable storage medium provided in the above embodiments of this application and the large file distribution method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0203] This application also provides a computer program product 500, such as... Figure 5 As shown. This computer program product carries a computer program 501, the instructions of which can be used to execute the steps of the large file distribution method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0204] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0205] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0206] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0207] Additionally, as used herein, the “or” used in a list of items beginning with “at least one” indicates a separate list, such that a list of, for example, “at least one of A, B, or C” means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word “exemplary” does not imply that the described example is preferred or better than other examples.

[0208] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0209] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0210] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0211] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for distributing large files, characterized in that, include: Obtain the total number of file fragments of the large file to be distributed, the network performance parameters, resource performance parameters, and scheduling performance parameters of each container node; The performance weight of each container node is determined based on the network performance parameters, resource performance parameters, and scheduling performance parameters of each container node. The number of file fragments distributed by each container node is determined based on the performance weight of each container node and the total number of file fragments.

2. The method according to claim 1, characterized in that, Obtain network performance parameters, including: The network bandwidth parameters, network stability parameters, and network latency parameters of each container node are obtained. The network bandwidth parameters are related to the historical average bandwidth, historical uplink bandwidth, historical downlink bandwidth, and historical peak bandwidth. The network stability parameters are related to the network packet loss rate parameters and network latency jitter parameters. The network latency parameters are related to the average network latency from the source container node to the target container node and the average network latency from the target container node to the other container nodes. The network performance parameters are determined based on the network bandwidth parameters, the network stability parameters, the network latency parameters, and the preset adjustment parameters corresponding to each parameter.

3. The method according to claim 1, characterized in that, Obtain resource performance parameters, including: The CPU performance parameters, memory performance parameters, disk performance parameters, and graphics processor performance parameters of each container node are obtained. The CPU performance parameters are related to CPU utilization, CPU average load, number of CPU cores, and CPU cache size. The memory performance parameters are related to memory utilization, memory bandwidth, and memory access latency. The disk performance parameters are related to disk size, disk data read speed, and disk data write speed. The graphics processor performance parameters are related to the number of graphics processors. The resource performance parameters are determined based on the CPU performance parameters, memory performance parameters, disk performance parameters, and graphics processor performance parameters.

4. The method according to claim 1, characterized in that, Obtain scheduling performance parameters, including: Get the number of container nodes in the worker nodes of the target container node; The scheduling performance parameters are determined based on the number of container nodes.

5. The method according to claim 1, characterized in that, The determination of the number of file shards distributed by each container node based on the performance weight of each container node and the total number of file shards includes: The performance weights of each container node are summed to obtain the total performance weights; The number of file fragments distributed by each container node is determined based on the sum of the performance weights, the performance weights of each container node, and the total number of file fragments.

6. The method according to claim 1, characterized in that, After determining the number of file shards distributed by each container node based on the performance weight of each container node and the total number of file shards, the method further includes: When the target container node receives the first task and the network resources of the target container node are insufficient, a network performance adjustment factor is determined based on the bandwidth parameter and peak bandwidth parameter occupied by the first task in the target container node, and a resource performance adjustment factor is determined based on the number of CPUs occupied by the first task in the target container node and the total number of CPUs in the target container node, wherein the priority of the first task is greater than the priority of the distribution task of the large file to be distributed. Based on the network performance adjustment factor, the resource performance adjustment factor, and the performance weight, the current performance weight of the target container node is determined. Based on the current performance weight and the total number of currently undistributed file shards, the number of current file shards distributed by each container node is determined.

7. The method according to claim 1, characterized in that, After determining the number of file shards distributed by each container node based on the performance weight of each container node and the total number of file shards, the method further includes: If the worker node to which the target container node belongs restarts, obtain the newly generated container node and determine the performance weight of the new container node; Based on the performance weight of the new container node, the performance weight of the container node with undistributed file shards, and the total number of undistributed file shards, the number of file shards distributed by the container node for each undistributed file shard and the number of file shards of the new container node are determined.

8. A large file distribution device, characterized in that, include: The acquisition module is used to obtain the total number of file fragments of the large file to be distributed, the network performance parameters, resource performance parameters and scheduling performance parameters of each container node; The first determining module is used to determine the performance weight of each container node based on the network performance parameters, resource performance parameters and scheduling performance parameters of each container node. The second determining module is used to determine the number of file fragments distributed by each container node based on the performance weight of each container node and the total number of file fragments.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method as described in any one of claims 1-7.

11. A computer program product, characterized in that, Includes computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the method as described in any one of claims 1-7.

Images