A file slicing uploading method based on RMSRE self-adaptation

Abstract

The application belongs to the field of cloud computing large file uploading, and discloses a file slicing uploading method based on RMSRE self-adaptation, which collects historical uploading data of a large file of self-adaptive multi-dimensional time sequence, obtains observation values through the collected historical uploading data, carries out data preprocessing on the observation values, establishes a slicing prediction model according to the observation values after data preprocessing in the historical uploading data and in combination with a self-adaptive algorithm, adjusts a weight calculation formula of the slicing prediction model, adjusts the weight of the prediction model according to a new error adjustment term, and the slicing prediction model carries out weighted calculation on the original observation values to automatically find a weight combination, so that the self-adaptive multi-dimensional sequence large file uploading slicing method can more accurately and in real time calculate the optimal size of slicing during large file uploading, ensures the efficiency of large file uploading, and solves the problems of unstable network and resource waste encountered during large file uploading.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet file service, more particularly, the present application relates to a file slice uploading method based on RMSRE self-adaptation. BACKGROUND

[0002] File transfer is an important link in the field of Internet technology and application, in the file uploading process, a specific protocol is needed to limit the data format and unified interface operation, the commonly used file uploading protocols include HTTP, FTP, SFTP / SCP and RESTful API.

[0003] Different file uploading protocols have some differences, for example, HTTP / HTTPS protocol depends on the definition of range in the header, FTP depends on the "REST NNN" command to define the starting point and size of the file, SFTP supports the end of the file to meet the needs of breakpoint resume, and SCP does not support the function of breakpoint resume.

[0004] In the background of adaptive multi-dimensional time series large file transfer service, due to network fluctuation, network interruption and server exception, the uploading process may be interrupted or failed, and the file needs to be re-uploaded or the breakpoint resume is needed, especially in the network peak period or mobile office scene, the frequency of failure will increase, resulting in the extension of user waiting time, the increase of broadband resource occupation and the waste of server and other resources, however, the existing mainstream file uploading protocol cannot adaptively adjust the uploading size of the file, and cannot well solve the transmission failure problem and meet the requirements of adaptive multi-dimensional time series large file efficient transmission.

[0005] The existing technology still has the following problems:

[0006] The adaptive multi-dimensional time series large file uploading failure rate is high, and the transmission service reliability is low, resulting in repeated waste of computing, storage and network resources;

[0007] The existing mainstream file uploading protocol cannot adaptively adjust the uploading size of the file, and cannot well solve the transmission failure problem and meet the requirements of adaptive multi-dimensional time series large file efficient transmission.

[0008] In view of this, the present application provides a file slice uploading method based on RMSRE self-adaptation to solve the above problems. SUMMARY

[0009] In order to overcome the above-mentioned defects of the prior art, in order to achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0010] A file slice uploading method based on RMSRE self-adaptation, comprising:

[0011] collecting historical upload data of large files of adaptive multi-dimensional time series;

[0012] obtaining observation values from the collected historical upload data;

[0013] data preprocessing on the observation values;

[0014] establishing a slice prediction model according to the data-preprocessed observation values in the historical upload data and in combination with an adaptive algorithm;

[0015] adjusting the weight of the prediction model according to a new error adjustment term by using a calculation formula of the weight adjusted by the slice prediction model;

[0016] the slice prediction model performs weighted calculation on the original observation values to automatically find a weight combination;

[0017] Further, the method for collecting historical upload data of large files of adaptive multi-dimensional time series comprises:

[0018] S101, logging into a file upload system and entering a corresponding management interface;

[0019] S102, finding a historical record or log function and selecting a time period for which data needs to be collected.

[0020] Further, the method for obtaining observation values from the collected historical upload data comprises:

[0021] S201, exporting or downloading historical upload record data and saving it in a suitable format;

[0022] S202, opening the exported historical upload record data by using a data processing tool;

[0023] S203, extracting information of file size, slice size, average upload speed and upload stability as observation values.

[0024] Further, the method for data preprocessing on the observation values comprises:

[0025] S301, removing observation values that obviously deviate from a normal range by statistical methods or rules;

[0026] S302, selecting to delete observation values containing missing values or filling the missing values by using an interpolation method;

[0027] S303, extracting observation values as features according to business requirements and data analysis targets.

[0028] Further, the method for establishing a slice prediction model according to data-preprocessed observation values in historical upload data and in combination with an adaptive algorithm comprises:

[0029] The prediction model for calculating the optimal size of the slice during the uploading of a large file with real-time adaptive multi-dimensional time series includes:

[0030]

[0031] is the predicted value of the slice during the uploading of the t+1th slice, W is the weight of the observation value during the uploading of the t-i+1th slice, is the predicted value of the t-i+1th slice, N is the number of historical slice sequences, wherein the vector of the slice observation value is represented as:

[0032]

[0033] represents the slice size of the t-i+1th file, represents the average uploading speed of the t-i+1th slice, represents the uploading stability of the t-i+1th slice, and the calculation method is the relative root mean square error (RMSRE) during the uploading of the slice, X K is the instantaneous uploading speed at the kth moment, and there is

[0034]

[0035] Further, the method for adjusting the weight of the prediction model according to the new error adjustment term using the slice prediction model adjustment weight calculation formula includes:

[0036] The formula for adjusting the weight of the slice prediction model is:

[0037] W′ i = W i + 2k·e i+1 Y t-i+1 ;

[0038] In the formula, i=1, 2...N, N is the number of sequence data, W i is the i th weight before adjustment, W′ i is the i th weight after adjustment, k is a learning constant, e i+1 is the prediction error of the i+1th period.

[0039] Further, the method for automatically finding the weight combination by weighting the original observation value of the slice prediction model includes:

[0040] S401, set N as the initial value 2, set t as the initial value 2, calculate t+1=3, that is, the predicted value of the 3rd slice wherein t represents the sequence of the slice uploading, Y3 represents the actual size;

[0041] S402, calculating the prediction error

[0042] S403, adjusting the weights of t and the previous sequence:

[0043] W'1 = W1 + 2ke3Y2;

[0044] W'2 = W2 + 2ke3Y2;

[0045] S404, when t < N, let t = t + 1, continue to adjust the weights of W1, W2, W t of the weights, repeat the calculation method of steps S401-S403 until t = N, end the Nth weight adjustment;

[0046] S405, let N = N + 1, t = b, start a new round of adaptive weight adjustment calculation, repeat the calculation method of steps S401-S404 until N = A, obtain the completed adaptive prediction model;

[0047] S406, calculate the prediction value of the A+1th slice, that is, obtain the upload size of the slice;

[0048] S407, according to the prediction value and the actual value of the A+1th slice, correct the adaptive prediction model according to step S405, and make model correction for uploading the A+1+1th slice.

[0049] Further, an adaptive file slice uploading system implementing the RMSRE-based adaptive file slice uploading method comprises:

[0050] A data acquisition and preprocessing module is configured to acquire historical upload data of a large file of an adaptive multi-dimensional time sequence, obtain observation values from the acquired historical upload data, and perform data preprocessing on the observation values.

[0051] A slice prediction model module is configured to establish a slice prediction model according to the observation values obtained by preprocessing the historical upload data, and combine an adaptive algorithm.

[0052] A weight combination module is configured to use a slice prediction model to adjust a weight calculation formula, and adjust the weights of the prediction model according to a new error adjustment term.

[0053] An adaptive iteration slice uploading module is configured to perform weighted calculation on the original observation values by using the slice prediction model, and automatically find a weight combination.

[0054] An electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the RMSRE adaptive file slicing upload method when executing the computer program.

[0055] A computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program implements the RMSRE adaptive file slicing upload method when executed.

[0056] The technical effect and advantages of the RMSRE adaptive file slicing upload method of the present application are:

[0057] The adaptive multi-dimensional sequence large file upload slicing method can more accurately calculate the optimal size of slicing during large file upload in real time, ensure the efficiency of large file upload, and solve the problems of unstable network and resource waste encountered during large file upload.

[0058] A slicing dynamic prediction model based on RMSRE and adaptive weighted adjustment is proposed to provide optimal scale prediction for real-time large file upload, emphasize the different influence degrees of historical process slicing on new slicing, and ensure the dynamic iteration and optimization of the model. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 The RMSRE adaptive file slicing upload system of the present application is shown in Figure 1.

[0060] Figure 2 The RMSRE adaptive file slicing upload method of the present application is shown in Figure 2.

[0061] Figure 3 The slicing prediction model flowchart of the present application is shown in Figure 1.

[0062] Figure 4 The electronic device of the present application is shown in Figures 3 and 4. DETAILED DESCRIPTION

[0063] The technical solutions of the embodiments of the present application will be described in detail below with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0064] Embodiment 1

[0065] Please refer to Figure 1 and Figure 3As shown, the file slicing uploading method based on RMSRE self-adaption in the embodiment comprises a data acquisition and preprocessing module, a slice prediction model establishing module, a weight combination module and a self-adaptive iterative slice uploading module, each module is connected through wired and / or wireless mode to realize data transmission.

[0066] The data acquisition and preprocessing module is used for acquiring historical uploading data of a large file of self-adaptive multi-dimensional time sequence to obtain observation values, and then performing data preprocessing on the observation values.

[0067] The historical uploading data of a large file of self-adaptive multi-dimensional time sequence is acquired to obtain observation values, and the specific method for performing data preprocessing on the observation values comprises:

[0068] S101, log in to a file uploading system and enter a corresponding management interface;

[0069] S102, find a historical record or log function and select a time period for which data needs to be acquired;

[0070] S103, export or download historical uploading record data and save it in a suitable format (such as CSV, Excel, etc.);

[0071] S104, use a data processing tool (such as Excel, the pandas library of Python, etc.) to open the exported historical uploading record data;

[0072] S105, extract file size, slice size, average uploading speed and uploading stability and other information from the historical uploading record data as observation values;

[0073] S106, through statistical methods or rules, eliminate observation values that obviously deviate from a normal range, and determine a judgment standard of abnormal values according to the distribution of the observation values and business requirements, such as using a mean value plus or minus several times of a standard deviation as a threshold value;

[0074] For example, suppose there is a data set containing user age information, and it is found through analysis that the average value of the age is 30 years old and the standard deviation is 5 years old, a threshold value can be set, for example, values outside the range of (30-25, 30+25) are regarded as abnormal values, that is, data less than 20 years old or more than 40 years old can be removed from the data set;

[0075] S107, according to specific conditions, observation values containing missing values can be selected to be deleted, or missing values can be filled by using interpolation methods, and the processing method of the missing values can be selected according to the characteristics of the observation values and business requirements, common processing methods include deleting data containing missing values, filling missing values by using a mean value, a median or a mode, filling by using interpolation methods (such as linear interpolation, polynomial interpolation), etc.

[0076] For example, suppose there is a dataset containing user purchase amount information. After analysis, it is found that some data has missing values. Depending on the actual situation, if the proportion of missing values is small and has little effect on the overall result, you can choose to delete the data containing missing values. If there are many missing values and they have a significant impact on the result, you can use the mean or median to fill in the missing values to retain as much valid information as possible.

[0077] S108、According to the business requirements and data analysis goals, the observed values are extracted as features,

[0078] By processing and converting the original data, features that can reflect the essence of the problem are extracted. Common feature engineering methods include standardization, normalization, discretization, encoding, etc. to improve the expression ability and prediction accuracy of the model, and reduce the correlation and redundancy between features.

[0079] For example, suppose there is a dataset containing user personal information and consumption behavior data. When building a user consumption prediction model, you can extract multiple features as input variables according to business requirements and data analysis goals. For example, you can discretize age and convert continuous age information into virtual variables for different age groups. You can also standardize the consumption amount to eliminate the influence of different orders of magnitude. You can also encode the product category to map different products to a unified code representation. Through feature engineering, you can better utilize useful information in the data and improve the prediction ability of the model.

[0080] According to the above, the pre-processed file size, slice size, average upload speed and upload stability observation values can be obtained;

[0081] A slice prediction model module is established to calculate the optimal size of the slice during the upload of the adaptive multi-dimensional time series large file in real time. According to the observation values after data preprocessing in the historical upload data, combined with the adaptive algorithm, a slice prediction model is established to ensure the efficiency of the adaptive multi-dimensional time series large file upload.

[0082] The prediction model for calculating the optimal size of the slice during the upload of the adaptive multi-dimensional time series large file in real time includes:

[0083]

[0084] is the predicted value of the t-i+1 slice upload period, and N is the number of historical slice sequences, where the vector of slice observation values is represented as: is the predicted value of the t-i+1 slice upload period, and N is the number of historical slice sequences, where the vector of slice observation values is represented as:

[0085]

[0086] denotes the slice size of the t-i+1 file, denotes the average upload speed of the t-i+1 slice, denotes the upload stability of the t-i+1 slice, and the calculation method is the relative root mean square error (RMSRE) during the slice upload, X K is the instantaneous upload speed at the kth moment, and has

[0087]

[0088] a weight combination module, and the formula for adjusting the weight of the slice prediction model is:

[0089] W′ i = W i + 2k·e i+1 Y t-i+1 ;

[0090] In the formula, i = 1, 2...N, N is the number of sequence data, W i is the i th weight before adjustment, W′ i is the i th weight after adjustment, k is a learning constant, and e i+1 is the prediction error of the i+1 period.

[0091] In theory, the new weight combination is a new weight combination generated by adding a new error adjustment term to the old weight combination, and the weight adjustment range is the result calculated based on the prediction error, the original observation value, and the self-defined weight adjustment speed constant k, wherein k takes a value of 1 / N.

[0092] An adaptive iterative slice upload module, the prediction model performs weighted calculation on the original observation value, automatically finds the most suitable weight combination, and adjusts the model weight by continuously correcting the historical weight value and combining the prediction error, thereby improving the prediction accuracy of the adaptive iterative slice upload model.

[0093] The adaptive iterative slice upload method comprises:

[0094] Suppose that the first 10 slice observation values in the file slice upload process are known, and it is expected to predict the size of the 11 th slice by revising the adaptive prediction model:

[0095] S201, set N as the starting value 2, set t as the starting value 2, calculate t+1 = 3, that is, the prediction value of the 3 rd slice wherein t represents the order of slice upload, denotes the predicted size, and Y3 denotes the actual size.

[0096] S202, calculate the prediction error

[0097] S203, adjust the weights of t and previous sequences:

[0098] W'1 = W1 + 2ke3Y2;

[0099] W'2 = W2 + 2ke3Y2;

[0100] S204, when t < N, let t = t + 1, continue to adjust the weights of W1, W2, W t , repeat the calculation method of steps S201-S203 until t = N, end the Nth weight adjustment;

[0101] S205, let N = N + 1, t = 2, start a new round of adaptive weight adjustment calculation, repeat the calculation method of steps S201-S204 until N = 10, obtain the completed adaptive prediction model;

[0102] S206, calculate the prediction value of the 11th slice, that is, obtain the upload size of the slice;

[0103] S207, according to the prediction value and the actual value of the 11th slice, correct the adaptive prediction model according to step S205, and make model correction for uploading the 12th slice.

[0104] Embodiment 2

[0105] Please refer to Figure 2 , a file slice uploading method based on RMSRE adaptive includes:

[0106] Collect historical upload data of large files of adaptive multi-dimensional time series, obtain observation values from the collected historical upload data, and do data preprocessing on the observation values;

[0107] According to the observation values after data preprocessing in the historical upload data, combined with the adaptive algorithm, a slice prediction model is established;

[0108] Using the slice prediction model adjustment weight number calculation formula, adjusting the weight of the prediction model according to the new error adjustment term;

[0109] The slice prediction model performs weighted calculation on the original observation values, and automatically finds the weight combination.

[0110] Embodiment 3

[0111] Please refer to Figure 4As shown, the embodiment discloses to provide an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the method provided by the above-mentioned method when executing the computer program.

[0112] Since the electronic device introduced in the embodiment is the electronic device used to implement the soundproof room noise measurement method in the embodiment, based on the method of adaptive file slice uploading based on RMSRE in the embodiment, those skilled in the art can understand the specific implementation of the electronic device in the embodiment and its various forms, so the method of how to implement the method in the embodiment will not be introduced in detail. As long as the electronic device used to implement the method of adaptive file slice uploading based on RMSRE in the embodiment is implemented by those skilled in the art, it belongs to the scope of protection of the present application.

[0113] Embodiment 4

[0114] Please refer to Figure 4 As shown, the embodiment discloses to provide a computer readable storage medium, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the method provided by the above-mentioned method when executing the computer program.

[0115] The above embodiments can be realized all or partially by software, hardware, firmware or other any combination. When realized by software, the above embodiments can be realized all or partially in the form of computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on the computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired network or wireless network. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center and the like containing one or more available medium sets. The available medium can be magnetic medium (for example, floppy disk, hard disk, magnetic tape), optical medium (for example, DVD) or semiconductor medium. The semiconductor medium can be solid state disk.

[0116] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present application can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0118] In several embodiments provided by the present application, it should be understood that the disclosed system, device and method can be realized by other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units is only one, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0119] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0120] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit.

[0121] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0122] Finally: the above is only the preferred embodiment of the present application, and is not used to limit the present application, any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application, should be included in the protection scope of the present application.

Claims

1. A file slice upload method based on RMSRE adaptive upload, characterized in that, Includes the following steps: Collect historical upload data of large files using adaptive multi-dimensional time series analysis; Observations are obtained by collecting historical uploaded data; Perform data preprocessing on the observations; Based on the preprocessed observations from historical uploaded data, and combined with an adaptive algorithm, a slice prediction model is established. The methods for establishing the slice prediction model include: Predictive models for real-time calculation of the optimal slice size during large file uploads using adaptive multi-dimensional time series include: ； For the first Slice predictions during slice upload. For the first The weights of observations during the upload period of each slice. For the first Predicted values ​​during slice upload, , Let be the number of historical slice sequences, where the vector representation of slice observations is: ； express Slice size of each file express Average upload speed per slice express The upload stability of the first slice is calculated as the relative root mean square error during the upload period of that slice. The instantaneous upload speed at any given moment has : ； The calculation formula for adjusting the weights of the slice prediction model is used to adjust the weights of the prediction model according to the new error adjustment term; The slice prediction model performs weighted calculations on the original observations and automatically finds the weight combination.

2. The file slice upload method based on RMSRE adaptive upload according to claim 1, characterized in that, Methods for collecting historical upload data of large files using adaptive multi-dimensional time series data include: S101. Log in to the file upload system and enter the corresponding management interface; S102. Use the historical records or logs function to find the time period for which data needs to be collected.

3. The file slice upload method based on RMSRE adaptive upload according to claim 2, characterized in that, Methods for obtaining observations from historically uploaded data include: S201. Export or download historical upload data and save it in a suitable format; S202. Use a data processing tool to open the exported historical upload data; S203. Extract information on file size, slice size, average upload speed, and upload stability as observation values.

4. The file slice upload method based on RMSRE adaptive upload according to claim 3, characterized in that, Methods for preprocessing observations include: S301. Observations that deviate from the preset normal range are removed by using statistical methods or rules. S302. Select to delete observations containing missing values, or use interpolation methods to fill in missing values; S303. Based on business needs and data analysis objectives, extract the observed values ​​as features.

5. The file slice upload method based on RMSRE adaptive upload according to claim 4, characterized in that, The calculation formula for adjusting weights using a slice prediction model, and the method for adjusting the weights of the prediction model based on the new error adjustment term, include: The formula for adjusting the weights in the slice prediction model is: ； In the formula, , The number of sequence data. For the first time before the adjustment Individual weights, For the adjusted number Individual weights, For learning constant, For the first The prediction error for the period.

6. The file slice upload method based on RMSRE adaptive upload according to claim 5, characterized in that, The slice prediction model performs weighted calculations on the original observations, and methods for automatically finding weight combinations include: Assuming the observations of the first A slices during the file slice upload process are known, the goal is to predict the size of the (A+1)th slice by revising the adaptive prediction model: S401, Let The initial value is 'a'. The initial value is b, calculate That is, the first Predicted values ​​for each slice ,in This indicates the order in which the slices were uploaded. Indicates the magnitude of the prediction. Indicates the actual size; S402, Calculate the prediction error ; S403, to Adjust the weights of the preceding sequences: ； S404, when When, let Continue to , ... The weights are adjusted, and the calculation method in steps S401-S403 is repeated until... End of the first Weight adjustment; S405, let = +1, A new round of adaptive weighting calculation begins, repeating steps S401-S404 until... To obtain a complete adaptive prediction model; S406. Calculate the predicted value of the (A+1)th slice, that is, obtain the upload size of the slice; S407. Based on the predicted and actual values ​​of the (A+1)th slice, correct the adaptive prediction model according to step S405 to prepare for uploading the (A+1+1)th slice.

7. An adaptive file slice upload system, comprising implementing the RMSRE-based adaptive file slice upload method according to any one of claims 1-6, including: The data acquisition and preprocessing module is used to acquire historical upload data of large files with adaptive multi-dimensional time series, obtain observations from the acquired historical upload data, and perform data preprocessing on the observations; A slice prediction model module is established, which uses preprocessed observations from historically uploaded data and an adaptive algorithm to build a slice prediction model. The weight combination module uses the calculation formula for adjusting the weights of the slice prediction model and the method for adjusting the weights of the prediction model according to the new error adjustment term. The adaptive iterative slice upload module performs weighted calculations on the original observations by the slice prediction model and automatically finds the weight combination.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the file slice upload method based on RMSRE adaptive as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed, implements the file slice upload method based on RMSRE adaptive upload as described in any one of claims 1 to 6.

Images