A data uploading method and device based on dynamic prediction, equipment and medium
By building a network status prediction model and dynamically adjusting data sharding and upload modes, the problems of high signaling overhead and task queue congestion caused by fixed-size shards are solved, achieving more efficient bandwidth utilization and training task stability.
Patent Information
- Application Number
- CN202511001116.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-21
- Publication Date
- 2025-10-21
- Estimated Expiration
- 2045-07-21
AI Technical Summary
In the existing technology, when training data is segmented into fixed-size files for uploading, it cannot be adjusted in real time according to the network bandwidth, resulting in excessive signaling overhead, low bandwidth utilization, and possible congestion of the task queue.
By building a network status prediction model, we can dynamically predict the sharding parameters and network load in future time periods, and dynamically adjust the data sharding and upload mode to adapt to changes in network status and avoid task queue congestion.
It effectively avoids task queue congestion and excessive network load, and improves bandwidth utilization efficiency and the stability of training tasks.
Smart Images
Figure CN120583098B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data uploading method, device, equipment and medium based on dynamic prediction. Background Art
[0002] With the rapid development of artificial intelligence (AI), the training data for AI models is growing larger and larger, reaching TB (Terabyte) and even PB (Petabyte) levels. Therefore, during model training, due to network fluctuations and other issues, the entire data file is usually not uploaded directly to the training machine. Instead, the file is split into several small files and then uploaded to the training machine. The files are then merged on the machine to obtain the complete training data.
[0003] However, in current related methods, after the training data is segmented into fixed-size files, the signaling overhead during the upload process may be too large. At the same time, the fixed-size file segments cannot be adjusted in real time according to the network bandwidth conditions. During the idle period of bandwidth, the bandwidth utilization rate is very low when uploading fixed-size file segments. In addition, due to the large number of files that need to be uploaded, if the current bandwidth occupancy is high during the file upload process, it may cause congestion in the task queue. Summary of the Invention
[0004] The present application provides a data uploading method, apparatus, device and medium based on dynamic prediction, which can predict sharding parameters according to the constructed model so that the predicted sharding parameters are suitable for the network status of the future time period, thereby avoiding network problems when uploading sharded data.
[0005] This application provides a data uploading method based on dynamic prediction, including:
[0006] Generate network prediction datasets based on collected current network data and historical network data;
[0007] Inputting the network prediction data set into the target network state prediction model to predict the target sharding calculation parameters within a future preset time period through the target network state prediction model;
[0008] Calculate data sharding parameters based on target sharding parameters and current network data, and perform data sharding on the files to be uploaded in the current network data based on the data sharding parameters to obtain a number of sharded data to be uploaded;
[0009] The current network load is determined based on the current network data, and the task upload mode of the data upload task corresponding to the file to be uploaded is determined according to the current network load, and the plurality of fragmented data to be uploaded are uploaded to the data receiving end based on the task upload mode.
[0010] The present application also provides a data uploading device based on dynamic prediction, comprising:
[0011] A data set generation module is used to generate a network prediction data set based on the collected current network data and historical network data;
[0012] A parameter calculation module is used to input the network prediction data set into the target network state prediction model to predict the target sharding calculation parameters within a preset time period in the future through the target network state prediction model;
[0013] The data sharding module is used to calculate the data sharding parameters according to the target sharding calculation parameters and the current network data, and to perform data sharding on the files to be uploaded in the current network data according to the data sharding parameters to obtain a number of sharded data to be uploaded;
[0014] The data upload module is used to determine the current network load based on the current network data, and determine the task upload mode of the data upload task corresponding to the file to be uploaded according to the current network load, and upload several fragmented data to be uploaded to the data receiving end based on the task upload mode.
[0015] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned methods for uploading data based on dynamic prediction when executing the computer program.
[0016] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data uploading methods based on dynamic prediction are implemented.
[0017] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data uploading methods based on dynamic prediction when executed by a processor.
[0018] In the present application, a network prediction data set can be generated based on the collected current network data and historical network data; the network prediction data set is input into the target network state prediction model to predict the target sharding calculation parameters within a preset time period in the future through the target network state prediction model; data sharding parameters are calculated according to the target sharding calculation parameters and the current network data, and data sharding is performed on the files to be uploaded in the current network data according to the data sharding parameters to obtain a number of sharding data to be uploaded; the current network load is determined based on the current network data, and the task upload mode of the data upload task corresponding to the file to be uploaded is determined according to the current network load, and the number of sharding data to be uploaded is uploaded to the data receiving end based on the task upload mode.
[0019] It can be seen that through the method of the present application, the current network data can be collected, and a network prediction data set can be generated based on the collected current network data and historical network data. Then, the generated network prediction data set needs to be input into the target network state prediction model so that the target sharding calculation parameters within the future preset time period can be predicted through the target network state prediction model, and the predicted target sharding calculation parameters are the parameters used to calculate the data sharding parameters for data sharding. Furthermore, it is necessary to calculate the data sharding parameters based on the target sharding calculation parameters and the current network data, and then calculate the target sharding calculation parameters for data sharding based on the target sharding calculation parameters and the current network data, and perform data sharding on the files to be uploaded in the current network data according to the target sharding calculation parameters, and then select the upload mode according to the current network load, and upload the obtained plurality of sharding data to be uploaded to the data receiving end based on the selected upload mode. Therefore, on the one hand, the sharding parameters can be dynamically predicted through the constructed prediction model to obtain the most appropriate data sharding parameters in the future period, avoiding task queue congestion or excessive load; on the other hand, the task upload mode can be selected according to the current network load, and the appropriate method can be selected for task upload to avoid congestion of the task queue. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0021] Figure 1 A flow chart of a data uploading method based on dynamic prediction provided in an embodiment of the present application;
[0022] Figure 2 This is an architecture diagram of a data uploading method based on dynamic prediction provided in an embodiment of the present application;
[0023] Figure 3 A structural diagram of a data uploading device based on dynamic prediction provided in an embodiment of the present application. DETAILED DESCRIPTION
[0024] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0025] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.
[0026] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.
[0027] In current related methods, after the training data is segmented into fixed-size files, the signaling overhead during the upload process may be too large. At the same time, the fixed-size file segments cannot be adjusted in real time according to the network bandwidth conditions. During the idle bandwidth period, the bandwidth utilization rate is very low when uploading fixed-size file segments. In addition, due to the large number of files that need to be uploaded, if the current bandwidth occupancy is high during the file upload process, it may cause congestion in the task queue.
[0028] In order to overcome the above technical problems, the present application discloses a data uploading method, device, equipment and medium based on dynamic prediction, which can predict the sharding parameters according to the constructed model so that the predicted sharding parameters are suitable for the network status of the future time period, thereby avoiding network problems when uploading the sharded data.
[0029] The embodiment of the present application provides a data uploading method based on dynamic prediction. The method is described in detail in conjunction with the execution flow of the data uploading method based on dynamic prediction. The method includes:
[0030] Step S11: Generate a network prediction data set based on the collected current network data and historical network data.
[0031] In this embodiment, it is necessary to generate a network prediction data set by collecting current network data and historical network data. Specifically, it is necessary to collect current network data according to the preset monitoring component. It should be noted that the preset monitoring component is a component used to monitor the training cluster, such as the NodeExporter component and the PrometheusServer component. Through the NodeExporter, the network bandwidth usage and the resource usage of the machine nodes in the training cluster can be obtained, and the job data and file data can be collected at the same time, and the collected data can be stored.
[0032] Furthermore, the collected network data needs to be standardized. Specifically, it is necessary to determine the maximum and minimum values of the indicators corresponding to the various indicators in the current network data based on the historical network data; calculate the difference between the maximum and minimum values of the indicators to obtain a first target difference; determine the target indicator corresponding to the maximum and minimum values of the indicators, and determine the target network data corresponding to the target indicator in the current network data; calculate the difference between the target network data and the minimum value of the indicators to obtain a second target difference; and use the ratio between the second target difference and the first target difference as the standardized network data corresponding to the target indicator. It should be noted that the historical bandwidth, job characteristics and file characteristics in the historical data need to be selected as data characteristics, and the job characteristics and file characteristics refer to the number of jobs, job types, number of files and file types. Then, based on the maximum and minimum values corresponding to each data, the same type of data in the collected network data is standardized. For example, taking the historical bandwidth as an example, the historical maximum value corresponding to the historical bandwidth is x max , the corresponding historical minimum value is x min , when standardizing the collected network data, we can first determine the historical maximum bandwidth as x max And the historical minimum value x min Then calculate the historical maximum value as x max And the historical minimum value x min The difference between the bandwidth and the historical minimum value is calculated to obtain the first target difference. The difference between the bandwidth of the current network data collected and the historical minimum value is then calculated to obtain the second target difference. The ratio between the second target difference and the first target difference is used as the normalized network data corresponding to the bandwidth. It should be further explained that by normalizing the data, the data value can be compressed to the range of [0, 1], and the normalization is expressed as follows:
[0033] ;
[0034] Among them, x is the standardized data, x max is the historical maximum value, x min is the historical minimum, x i The data that needs to be standardized.
[0035] Finally, the obtained standardized network data needs to be used to generate a network prediction dataset.
[0036] In this way, a network prediction dataset can be generated based on standardized network data, thereby eliminating the dimensional differences between different features and reducing the errors caused by numerical differences, so as to improve the prediction accuracy in subsequent predictions.
[0037] Step S12: inputting the network prediction data set into the target network state prediction model to predict the target sharding calculation parameters within a future preset time period through the target network state prediction model.
[0038] In this embodiment, the generated network prediction dataset can be input into a target network state prediction model, so that the target network prediction model can predict target sharding calculation parameters for a future preset time period based on the network prediction dataset. Specifically, the network prediction dataset can be input into the target network state prediction model to predict the bandwidth for the future preset time period, thereby obtaining the predicted bandwidth for the future preset time period. Furthermore, the target network state prediction model can be used to predict the job weights corresponding to the files to be uploaded for the future preset time period, thereby obtaining the predicted weights for the files to be uploaded. It should be noted that since the target network state prediction model is a trained model that can achieve optimal prediction results, after inputting the network prediction dataset into the target network state prediction model, the possible bandwidth conditions, the number of files to be uploaded, and the corresponding job weights for the future time period can be obtained. It should be further noted that the target network state prediction model is a model obtained by training an LSTM-Attention (Long Short-Term Memory-Attention) prediction model. Therefore, the target network state prediction model can be used to predict network conditions for the future preset time period, such as network bandwidth and job weights, and use these predicted data as sharding calculation parameters. In this way, the sharding parameters can be dynamically predicted through the constructed prediction model to obtain the most appropriate data sharding parameters for the current or future period, avoiding task queue congestion or excessive load.
[0039] Step S13: Calculate data sharding parameters according to the target sharding calculation parameters and the current network data, and perform data sharding on the files to be uploaded in the current network data according to the data sharding parameters to obtain a plurality of sharded data to be uploaded.
[0040] In this embodiment, it is necessary to calculate the data sharding parameters based on the target sharding calculation parameters and the current network data. It should be noted that the target sharding parameters are parameters used to calculate the data sharding parameters, and the data sharding parameters are parameters used to shard the files. Therefore, the files to be uploaded in the current network data can be sharded according to the calculated data sharding parameters to obtain a number of sharded data to be uploaded. Specifically, it is necessary to determine the current network round-trip delay and the current network packet loss rate based on the current network data, and then calculate the sum of the current network packet loss rate and the second preset constant, and calculate the ratio of the current network round-trip delay and the sum to obtain the target ratio; the product of the predicted bandwidth, the target ratio and the predicted weight is used as the data sharding parameter corresponding to the file to be uploaded. In addition, the obtained data sharding parameter represents the size of the data that needs to be sharded. For example, if the calculated data sharding parameter is 20MB, the file to be uploaded needs to be divided into 20MB in size, and the expression of the data sharding parameter, that is, the sharding size, is as follows:
[0041] ;
[0042] After obtaining the data sharding parameters, the file to be uploaded can be sharded according to the data sharding parameters to obtain a number of sharded data to be uploaded. In this way, the network conditions can be dynamically predicted to obtain the most appropriate data sharding parameters for a period of time in the future, thereby ensuring that the uploaded file will not cause congestion in the task queue after the sharded files are uploaded.
[0043] Step S14: determine the current network load based on the current network data, and determine the task upload mode of the data upload task corresponding to the file to be uploaded according to the current network load, and upload the plurality of fragmented data to be uploaded to the data receiving end based on the task upload mode.
[0044] In this embodiment, it is necessary to determine the current network load based on the current network parameters, and determine the task upload mode suitable for the current network status based on the network load, and then upload a number of to-be-uploaded fragmented data to the data receiving end according to the selected task upload mode. Specifically, it is necessary to determine the current network load based on the current network data, and judge whether the current network load is greater than the preset network load threshold, and then select the corresponding task upload mode based on the obtained judgment result. In the case where the current network load is greater than the preset network load threshold, it indicates that the current data upload may cause network congestion, so it is necessary to set the current task upload mode to delayed upload. Furthermore, it is necessary to determine the priority of the data upload task corresponding to the to-be-uploaded file based on the predicted prediction weight, and then sort the data upload tasks according to the priority of the data upload task, so that the subsequent upload of the number of to-be-uploaded fragmented data corresponding to the to-be-uploaded file can be delayed according to the task upload order obtained after sorting. In another case, if the current network load is not greater than the preset network load threshold, the number of to-be-uploaded fragmented data obtained by fragmentation is directly uploaded.
[0045] It should be noted that after the data upload is completed, the receiving end needs to merge the data, and can determine whether to delay the merger based on the current machine load situation to avoid causing the machine load to be too high. Specifically, if the data upload is completed, it is determined whether the current machine load of the data receiving end is greater than the preset machine load threshold; if the current machine load is not greater than the preset machine load threshold, it indicates that the current machine load is in a normal state, so the received sharded data can be directly merged. On the other hand, if the current machine load is greater than the preset machine load threshold, it indicates that the current machine load is high, and data merging may cause the machine training task to be blocked or even down, so it is necessary to postpone the data merging of several sharded data to be uploaded until the current machine load is no greater than the preset machine load threshold. In this way, the files can be merged at different times, effectively avoiding the blocking or even downtime of the machine training task caused by the concurrent merging of multiple tasks, thereby improving stability.
[0046] In this embodiment, current network data can be collected, and a network prediction data set is generated based on the collected current network data and historical network data. The generated network prediction data set is then input into a target network state prediction model so that the target network state prediction model can predict the target sharding calculation parameters within a preset time period in the future. The predicted target sharding calculation parameters are the parameters used to calculate the data sharding parameters for data sharding. Furthermore, data sharding parameters need to be calculated based on the target sharding calculation parameters and the current network data. Then, target sharding calculation parameters for data sharding are calculated based on the target sharding calculation parameters and the current network data. The target sharding calculation parameters for data sharding are then calculated based on the target sharding calculation parameters and the current network data. The files to be uploaded in the current network data are then sharded based on the target sharding calculation parameters. Then, an upload mode is selected based on the current network load, and the obtained plurality of sharding data to be uploaded are uploaded to the data receiving end based on the selected upload mode. Thus, on the one hand, the sharding parameters can be dynamically predicted by the constructed prediction model to obtain the most appropriate data sharding parameters for a period of time in the future, thereby avoiding the situation where the task queue is blocked or the load is too high. On the other hand, the task upload mode can be selected based on the current network load, and the appropriate method for task upload is selected to avoid sharding the task queue.
[0047] As a preferred embodiment, before prediction is performed using the target network state prediction model, the initial preset network state prediction model needs to be trained and the parameters updated to obtain the target network state prediction model. Specifically, the model parameters and the gradient accumulation corresponding to the preset network state prediction model need to be initialized to obtain the current model parameters and the current gradient accumulation α, and the initial α is 0; further, the current gradient estimation value corresponding to the preset network state prediction model needs to be calculated using the preset batch size, the preset loss function, and the historical network data, wherein the calculation expression of the gradient estimation value is as follows:
[0048] ;
[0049] Where s is the gradient estimate, n is the batch size, is the gradient (partial derivative) of the loss function L with respect to θ, L(f(xi,θ)) is the loss of a single sample, xi, yi are a pair of samples in the historical network data. It should be noted that s represents the gradient direction of the current batch, which is used to guide parameter updates.
[0050] Furthermore, it is necessary to update the gradient accumulator based on the preset attenuation coefficient, the square value corresponding to the gradient estimate, and the current gradient accumulator, so that the updated gradient accumulator is used as the current gradient accumulator. The preset attenuation coefficient is used to control the attenuation rate of historical information and is usually close to 1, such as 0.9. The expression for the gradient accumulator update is as follows:
[0051] ;
[0052] Among them, α is the gradient accumulation, β is the preset attenuation coefficient, and s is the gradient estimation value.
[0053] Furthermore, after obtaining the current gradient accumulation, it is necessary to calculate the parameter model update amount according to the preset global learning rate, the first preset constant, the current gradient accumulation amount and the gradient estimation value to obtain the target model parameter update amount , where the calculation expression of the target model parameter update amount is as follows:
[0054] ;
[0055] in, is the target model parameter update amount, ε represents the global learning rate, δ is the first preset constant, α is the gradient accumulation amount, e is a natural constant, and s is the gradient estimate.
[0056] Then the current model parameters need to be updated based on the target model parameter update amount, so that the updated model parameters are used as the current model parameters, that is, , then it is necessary to jump to the step of calculating the current gradient estimate corresponding to the preset network state prediction model using the preset batch size, preset loss function, and historical network data to continuously update the parameters until the target cumulative square gradient is obtained, and the target cumulative square gradient is the minimum value corresponding to the current gradient accumulation. Finally, the current model parameters corresponding to the target cumulative square gradient are used as the target model parameters, and the network state prediction model corresponding to the target model parameters is used as the target network state prediction model. In this way, the parameters of the initial prediction model can be adjusted to obtain the target network state prediction model that meets the expectations, thereby achieving the best performance of the prediction model and ensuring the accuracy of the prediction.
[0057] As a preferred embodiment, Figure 2Figure 2 shows the framework of a data upload method based on dynamic prediction. The data collection module monitors the training cluster, obtaining network bandwidth usage and real-time machine load data, as well as historical job counts and uploaded job file data. The LSTM prediction engine, using the LSTM-Attention model, analyzes the cluster's historical network bandwidth usage, job counts, and uploaded job file data to predict network bandwidth usage, job counts, and uploaded job file data for a specific period of time. The sharding decider calculates the shard size and rate for each future job file upload based on the prediction module's predictions. The scheduler executes file uploads based on the sharding decider's results, monitoring network load and delaying upload tasks based on priority if the load exceeds a threshold. Furthermore, when merging files after completion, it also assesses machine load to determine if it will cause excessive machine load. If so, the merge task is postponed to avoid blocking training tasks or causing machine downtime.
[0058] Among them, first of all, the current network data needs to be collected by the data collection module, and the collected current network data includes the current bandwidth, as well as job characteristics and file characteristics. Then, the current network data and historical network data including historical bandwidth need to be pushed to the prediction engine. The future bandwidth is predicted through the target network status prediction model in the prediction engine, and the task priority weight and data sharding parameters are calculated through the sharding decider. Then, the sharding strategy is issued, and the scheduling executor performs data sharding according to the sharding strategy and feedbacks the actual bandwidth status and load status.
[0059] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.
[0060] See also Figure 3 As shown, an embodiment of the present application further provides a data uploading device based on dynamic prediction, comprising:
[0061] A data set generation module 11 is used to generate a network prediction data set based on the collected current network data and historical network data;
[0062] The parameter calculation module 12 is used to input the network prediction data set into the target network state prediction model to predict the target shard calculation parameters within a future preset time period through the target network state prediction model;
[0063] The data slicing module 13 is used to calculate data slicing parameters according to the target slicing calculation parameters and the current network data, and to slice the files to be uploaded in the current network data according to the data slicing parameters to obtain a plurality of sliced data to be uploaded;
[0064] The data upload module 14 is used to determine the current network load based on the current network data, and determine the task upload mode of the data upload task corresponding to the file to be uploaded according to the current network load, and upload several fragmented data to be uploaded to the data receiving end based on the task upload mode.
[0065] In this embodiment, current network data can be collected, and a network prediction data set is generated based on the collected current network data and historical network data. The generated network prediction data set is then input into a target network state prediction model so that the target network state prediction model can predict the target sharding calculation parameters within a preset time period in the future. The predicted target sharding calculation parameters are the parameters used to calculate the data sharding parameters for data sharding. Furthermore, data sharding parameters need to be calculated based on the target sharding calculation parameters and the current network data. Then, target sharding calculation parameters for data sharding are calculated based on the target sharding calculation parameters and the current network data. The target sharding calculation parameters for data sharding are then calculated based on the target sharding calculation parameters and the current network data. The files to be uploaded in the current network data are then sharded based on the target sharding calculation parameters. Then, an upload mode is selected based on the current network load, and the obtained plurality of sharding data to be uploaded are uploaded to the data receiving end based on the selected upload mode. Thus, on the one hand, the sharding parameters can be dynamically predicted by the constructed prediction model to obtain the most appropriate data sharding parameters for a period of time in the future, thereby avoiding the situation where the task queue is blocked or the load is too high. On the other hand, the task upload mode can be selected based on the current network load, and the appropriate method for task upload is selected to avoid sharding the task queue.
[0066] In some embodiments, the data set generation module 11 may specifically include:
[0067] A network data collection unit is used to collect current network data according to a preset monitoring component;
[0068] An indicator determination unit, configured to determine, based on historical network data, the maximum and minimum values of indicators corresponding to various indicators in current network data;
[0069] A first difference calculation unit is used to calculate the difference between the maximum value of the indicator and the minimum value of the indicator to obtain a first target difference;
[0070] A network data determination unit, configured to determine a target indicator corresponding to a maximum indicator value and a minimum indicator value, and determine target network data corresponding to the target indicator in the current network data;
[0071] a second difference calculation unit, configured to calculate a difference between the target network data and the minimum value of the indicator to obtain a second target difference;
[0072] a data normalization unit, configured to use a ratio between the second target difference and the first target difference as normalized network data corresponding to the target indicator;
[0073] The dataset generation unit is used to generate a network prediction dataset based on the standardized network data.
[0074] In some embodiments, the data uploading device based on dynamic prediction may further include:
[0075] A parameter initialization unit is used to initialize the model parameters and gradient accumulation corresponding to the preset network state prediction model to obtain the current model parameters and current gradient accumulation;
[0076] An estimated value calculation unit, configured to calculate a current gradient estimate corresponding to a preset network state prediction model using a preset batch size, a preset loss function, and historical network data;
[0077] a gradient updating unit, configured to update the gradient accumulation based on a preset attenuation coefficient, a square value corresponding to the gradient estimate, and a current gradient accumulation, so as to use the updated gradient accumulation as the current gradient accumulation;
[0078] An update amount calculation unit, configured to calculate a parameter model update amount according to a preset global learning rate, a first preset constant, a current gradient accumulation amount, and a gradient estimation value, so as to obtain a target model parameter update amount;
[0079] a model parameter updating unit, configured to update the current model parameters based on the target model parameter update amount, so as to use the updated model parameters as the current model parameters;
[0080] A step jump unit is used to jump to the step of calculating the current corresponding gradient estimate of the preset network state prediction model by using a preset batch size, a preset loss function, and historical network data until a target cumulative square gradient is obtained; the target cumulative square gradient is the minimum value corresponding to the current gradient accumulation;
[0081] The model determination unit is used to use the current model parameters corresponding to the target cumulative square gradient as the target model parameters, and use the network state prediction model corresponding to the target model parameters as the target network state prediction model.
[0082] In some embodiments, the parameter calculation module 12 may specifically include:
[0083] The bandwidth prediction unit is configured to input the network prediction data set into the target network state prediction model to predict the bandwidth within a future preset time period through the target network state prediction model to obtain the predicted bandwidth within the future preset time period;
[0084] The weight prediction unit is used to predict the job weights corresponding to the files to be uploaded within a preset time period in the future through a target network state prediction model to obtain the predicted weights corresponding to the files to be uploaded.
[0085] In some embodiments, the data sharding module 13 may specifically include:
[0086] A network parameter determination unit, configured to determine a current network round trip delay and a current network packet loss rate based on current network data;
[0087] a data calculation unit, configured to calculate a sum of a current network packet loss rate and a second preset constant, and calculate a ratio of a current network round-trip delay and the sum to obtain a target ratio;
[0088] The sharding parameter determining unit is configured to use the product of the predicted bandwidth, the target ratio, and the predicted weight as the data sharding parameter corresponding to the file to be uploaded.
[0089] In some embodiments, the data upload module 14 may specifically include:
[0090] A first data judgment unit is used to determine a current network load based on current network data, and to judge whether the current network load is greater than a preset network load threshold;
[0091] A first data uploading unit is configured to set the current task upload mode to deferred upload if the current network load is greater than a preset network load threshold, and determine the priority of the data upload task corresponding to the to-be-uploaded file based on the predicted weight; sort the data upload tasks according to the priority, and deferred uploading of a plurality of to-be-uploaded fragmented data corresponding to the to-be-uploaded file based on the task upload order obtained after sorting;
[0092] The second data uploading unit is configured to upload a number of fragmented data to be uploaded if the current network load is not greater than a preset network load threshold.
[0093] In some embodiments, the data uploading device based on dynamic prediction may further include:
[0094] A second data determination unit is configured to determine whether a current machine load of the data receiving end is greater than a preset machine load threshold if data uploading is completed;
[0095] A first data merging unit, configured to merge data of a plurality of shards to be uploaded if the current machine load is not greater than a preset machine load threshold;
[0096] The second data merging unit is configured to postpone merging of the data of the plurality of sharded data to be uploaded if the current machine load is greater than a preset machine load threshold, until the current machine load is no greater than the preset machine load threshold.
[0097] For the description of the features in the embodiment corresponding to the data uploading device based on dynamic prediction, please refer to the relevant description of the embodiment corresponding to the data uploading method based on dynamic prediction, which will not be repeated here.
[0098] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned embodiments of the data uploading method based on dynamic prediction.
[0099] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned embodiments of the data uploading method based on dynamic prediction when running.
[0100] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.
[0101] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned data uploading method embodiments based on dynamic prediction are implemented.
[0102] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of any of the above-mentioned data uploading method embodiments based on dynamic prediction.
[0103] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0104] The above is a detailed introduction to the data uploading method, device, equipment and medium based on dynamic prediction provided by this application. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of the claims of this application.
Claims
1. A data uploading method based on dynamic prediction, characterized in that: include: Generate network prediction datasets based on collected current network data and historical network data; Inputting the network prediction data set into the target network state prediction model to predict the target sharding calculation parameters within a future preset time period through the target network state prediction model; Calculating data sharding parameters according to the target sharding calculation parameters and the current network data, and sharding the files to be uploaded in the current network data according to the data sharding parameters to obtain a plurality of sharded data to be uploaded; Determining a current network load based on the current network data, and determining a task upload mode for a data upload task corresponding to a to-be-uploaded file according to the current network load, and uploading the plurality of to-be-uploaded fragmented data to a data receiving end based on the task upload mode; The step of inputting the network prediction data set into the target network state prediction model to predict the target sharding calculation parameters within a future preset time period through the target network state prediction model includes: Inputting the network prediction data set into a target network state prediction model to predict the bandwidth within a future preset time period through the target network state prediction model to obtain a predicted bandwidth within the future preset time period; Predicting the job weights corresponding to the files to be uploaded within the future preset time period using the target network state prediction model to obtain the predicted weights corresponding to the files to be uploaded; The step of calculating the data sharding parameters based on the target sharding parameters and the current network data includes: Determine the current network round trip delay and the current network packet loss rate according to the current network data; Calculating a sum of a current network packet loss rate and a second preset constant, and calculating a ratio of the current network round-trip delay and the sum to obtain a target ratio; The product of the predicted bandwidth, the target ratio and the predicted weight is used as a data sharding parameter corresponding to the file to be uploaded; the data sharding parameter represents the size of data that needs to be sharded.
2. The data uploading method based on dynamic prediction according to claim 1, characterized in that: The generating of a network prediction data set based on the collected current network data and historical network data includes: Collect current network data according to preset monitoring components; Determine, based on historical network data, the maximum value and the minimum value of an indicator corresponding to each indicator in the current network data; Calculating the difference between the maximum value of the indicator and the minimum value of the indicator to obtain a first target difference; Determining a target indicator corresponding to the maximum indicator value and the minimum indicator value, and determining target network data corresponding to the target indicator in the current network data; Calculating a difference between the target network data and the minimum value of the indicator to obtain a second target difference; Using the ratio between the second target difference and the first target difference as the normalized network data corresponding to the target indicator; A network prediction dataset is generated based on the normalized network data.
3. The data uploading method based on dynamic prediction according to claim 1, characterized in that: Before inputting the network prediction data set into the target network state prediction model to predict the target shard calculation parameters within a future preset time period through the target network state prediction model, the method further includes: Initialize the model parameters and gradient accumulation corresponding to the preset network state prediction model to obtain the current model parameters and current gradient accumulation; Calculate the current gradient estimation value corresponding to the preset network state prediction model by using the preset batch size, the preset loss function and the historical network data; updating the gradient accumulation amount based on a preset attenuation coefficient, a square value corresponding to the gradient estimation value, and the current gradient accumulation amount, so as to use the updated gradient accumulation amount as the current gradient accumulation amount; Calculating a parameter model update amount according to a preset global learning rate, a first preset constant, the current gradient accumulation amount, and the gradient estimate value to obtain a target model parameter update amount; updating the current model parameters based on the target model parameter update amount to use the updated model parameters as the current model parameters; Jump to the step of calculating the current corresponding gradient estimate of the preset network state prediction model by using the preset batch size, the preset loss function, and the historical network data until a target cumulative square gradient is obtained; the target cumulative square gradient is the minimum value corresponding to the current gradient accumulation; The current model parameters corresponding to the target cumulative square gradient are used as target model parameters, and the network state prediction model corresponding to the target model parameters is used as the target network state prediction model.
4. The data uploading method based on dynamic prediction according to claim 1, characterized in that: The determining of the current network load based on the current network data, determining a task upload mode of the data upload task corresponding to the to-be-uploaded file according to the current network load, and uploading the plurality of to-be-uploaded fragmented data to the data receiving end based on the task upload mode, includes: Determining a current network load based on the current network data, and determining whether the current network load is greater than a preset network load threshold; If the current network load is greater than the preset network load threshold, the current task upload mode is set to deferred upload, and the priority of the data upload task corresponding to the to-be-uploaded file is determined according to the predicted weight; the data upload tasks are sorted according to the priority, and the upload of the plurality of to-be-uploaded fragmented data corresponding to the to-be-uploaded file is deferred based on the task upload order obtained after sorting; If the current network load is not greater than the preset network load threshold, the plurality of fragmented data to be uploaded are uploaded.
5. The data uploading method based on dynamic prediction according to any one of claims 1 to 4, characterized in that: Also includes: If the data upload is completed, determining whether the current machine load of the data receiving end is greater than a preset machine load threshold; If the current machine load is not greater than the preset machine load threshold, merging the plurality of fragmented data to be uploaded; If the current machine load is greater than the preset machine load threshold, data merging of the plurality of fragmented data to be uploaded is postponed until the current machine load is no greater than the preset machine load threshold.
6. A data uploading device based on dynamic prediction, characterized in that: include: A data set generation module is used to generate a network prediction data set based on the collected current network data and historical network data; A parameter calculation module is used to input the network prediction data set into the target network state prediction model to predict the target shard calculation parameters within a future preset time period through the target network state prediction model; A data slicing module is configured to calculate data slicing parameters based on the target slicing calculation parameters and the current network data, and to perform data slicing on the files to be uploaded in the current network data according to the data slicing parameters to obtain a plurality of slicing data to be uploaded; A data uploading module is used to determine the current network load based on the current network data, and determine the task uploading mode of the data uploading task corresponding to the to-be-uploaded file according to the current network load, and upload the plurality of to-be-uploaded fragmented data to the data receiving end based on the task uploading mode; Wherein, the parameter calculation module includes: a bandwidth prediction unit, configured to input the network prediction data set into a target network state prediction model, so as to predict the bandwidth within a future preset time period using the target network state prediction model, so as to obtain the predicted bandwidth within the future preset time period; A weight prediction unit, configured to predict the job weights corresponding to the files to be uploaded within the future preset time period using the target network state prediction model, so as to obtain the predicted weights corresponding to the files to be uploaded; The data sharding module includes: a network parameter determination unit, configured to determine a current network round trip delay and a current network packet loss rate based on the current network data; a data calculation unit, configured to calculate a sum of a current network packet loss rate and a second preset constant, and calculate a ratio of the current network round-trip delay and the sum to obtain a target ratio; The fragmentation parameter determination unit is configured to use the product of the predicted bandwidth, the target ratio, and the predicted weight as a data fragmentation parameter corresponding to the file to be uploaded; the data fragmentation parameter indicates the size of data to be fragmented.
7. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the data uploading method based on dynamic prediction as described in any one of claims 1 to 5 when executing the computer program.
8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the data uploading method based on dynamic prediction according to any one of claims 1 to 5 are implemented.
Citation Information
Patent Citations
Document transmission method and device
CN104022852A
Big data safe and efficient transmission method based on quantum encryption and server
CN117834251A