Model training, file uploading method and device, medium, product and equipment

Abstract

This application discloses a model training and file upload method, apparatus, medium, product, and device. The method includes: acquiring sample data formed by uploading a sample file in chunks according to preset chunking parameters, wherein the sample data includes success rate and time information of chunking upload corresponding to the chunking upload parameters, and the chunking upload parameters include: chunking attributes and / or upload mode attributes, and file attributes of the sample file; based on the sample data, performing multiple rounds of iterative training on a neural network model until a preset training stopping condition is met to obtain a file upload model, thereby enabling the trained file upload model to predict the success rate and time efficiency of chunking upload of a file to be uploaded according to different parameters, thereby assisting the user in selecting the most suitable chunking upload parameters when uploading the file to be uploaded in chunks, and improving the success rate and time efficiency of chunking upload.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training and file uploading method, apparatus, medium, product and equipment. Background Technology

[0002] In existing technologies, when uploading large files with a large amount of data, the large file is usually divided into blocks on the client side, and then the smaller blocks are uploaded to the server side separately. After all the smaller blocks have been successfully uploaded, they are then merged back into a large file on the server side according to the block encoding sequence number.

[0003] However, there is no specific reference standard for the selection of file chunking method and upload mode in existing large file uploads. It usually depends on the developer's personal experience and judgment to make decisions, which makes it difficult to guarantee the success rate and time efficiency of large file uploads and affects the user experience. Summary of the Invention

[0004] To address the aforementioned technical issues, this application proposes a model training and file uploading method, apparatus, medium, product, and equipment that can improve the success rate and time efficiency of fragmented uploading.

[0005] To achieve the above objectives, embodiments of this application provide a model training method, including:

[0006] The sample data is obtained after the sample file is uploaded in segments according to preset segment upload parameters. The sample data includes the success rate and time information of segment upload corresponding to the segment upload parameters. The segment upload parameters include: segment attributes and / or upload mode attributes, and the file attributes of the sample file.

[0007] Based on the sample data, the neural network model is trained iteratively for multiple rounds until the preset training stopping condition is met, thus obtaining the file upload model.

[0008] Furthermore, the fragmentation attribute includes at least the fragment size, the upload mode attribute includes at least the number of concurrent threads, and the file attribute includes at least the file size.

[0009] Furthermore, the neural network model is a backpropagation (BP) neural network model, and the step of performing multiple rounds of iterative training on the neural network model based on the sample data includes:

[0010] During each round of iterative training,

[0011] The sample data is input into the neural network model for forward propagation to obtain the predicted value;

[0012] Based on the error function between the actual output value and the predicted value corresponding to the sample data, the gradient of the parameters of the neural network model is calculated through backpropagation;

[0013] Based on the gradient, the parameters of the neural network model are adjusted using gradient descent to complete this round of iterative training.

[0014] Furthermore, the predicted value is calculated using an activation function based on the sample data, as well as the weights between layers and the bias of each neuron in the neural network model.

[0015] Furthermore, the error function is determined based on the mean squared error function and the parameter regularization term, wherein the mean squared error function is used to calculate the mean squared error between the actual output value and the predicted value, and the parameter regularization term is a regularization term related to the weights between the layers in the neural network model.

[0016] Furthermore, the parameters of the neural network model include the weights between each layer and the bias of each neuron in the neural network model, and the gradient includes the weight gradient and the bias gradient, wherein the weight gradient is determined based on the partial derivative of the error function with respect to the weights between each layer, and the bias gradient is determined based on the partial derivative of the error function with respect to the bias of each neuron.

[0017] Furthermore, the training stopping condition includes at least one condition: the error calculated by the error function satisfies a preset error range.

[0018] Furthermore, the number of the segmented upload parameters is N sets, the number of the sample data is N sets, and the N sets of segmented upload parameters correspond one-to-one with the N sets of sample data, where N is a positive integer. Before performing multiple rounds of iterative training on the neural network model based on the sample data, the method further includes:

[0019] The N sets of sample data are preprocessed, wherein the preprocessing includes standardization, normalization, denoising and / or outlier handling.

[0020] Based on the preprocessed N sets of sample data, a training set and a test set are constructed.

[0021] Furthermore, the step of performing multiple rounds of iterative training on the neural network model based on the sample data until a preset training stopping condition is met to obtain the file upload model includes:

[0022] The neural network model is trained iteratively using the data training set to obtain the test model when the iteration stops.

[0023] The model to be tested is trained once using the data test set, and the generalization ability of the model to be tested is determined based on the training results of the first round of training, wherein the generalization ability is used to indicate whether the model to be tested is overfitting.

[0024] If overfitting occurs, the model to be tested will be used as the file upload model.

[0025] If overfitting occurs, adjust the parameters of the model under test and repeat the above steps until overfitting is not observed.

[0026] This application also provides a file upload method, including:

[0027] Retrieve the multi-part upload parameters corresponding to the file to be uploaded;

[0028] The multi-part upload parameters are input into the file upload model for prediction to obtain the success rate and time information of each part upload corresponding to the multi-part upload parameters, so as to select the target part upload parameters from the multi-part upload parameters. The file upload model is trained based on the model training method described above.

[0029] Based on the target chunk upload parameters, the file to be uploaded is uploaded in chunks.

[0030] This application also provides a model training apparatus, including:

[0031] The sample data acquisition module is used to acquire sample data formed after a sample file is uploaded in segments according to preset segment upload parameters. The sample data includes the success rate and time information of segment upload corresponding to the segment upload parameters. The segment upload parameters include: segment attributes and / or upload mode attributes, and the file attributes of the sample file.

[0032] The iterative training module is used to perform multiple rounds of iterative training on the neural network model based on the sample data until the preset training stopping condition is met, thereby obtaining the file upload model.

[0033] This application also provides a file upload device, including:

[0034] The parameter acquisition module is used to acquire the multi-part upload parameters corresponding to the file to be uploaded.

[0035] The prediction module is used to input the multi-part upload parameters into the file upload model for prediction, and obtain the success rate and time information of the upload of each part corresponding to the multi-part upload parameters, so as to select the target upload parameter from the multi-part upload parameters. The file upload model is trained based on the model training method described above.

[0036] The chunked upload module is used to upload the file to be uploaded in chunks based on the target chunked upload parameters.

[0037] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0038] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the method described in any of the above embodiments.

[0039] This application also provides a computer device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the steps of any of the methods described above.

[0040] In summary, the embodiments of this application have at least the following beneficial effects:

[0041] In this embodiment, sample data is obtained by uploading a sample file in segments according to preset segmentation parameters. The sample data includes the success rate and time information of the segmentation upload corresponding to the segmentation upload parameters. The segmentation upload parameters include segmentation attributes and / or upload mode attributes, as well as the file attributes of the sample file. Based on the sample data, a neural network model is iteratively trained multiple times until a preset training stopping condition is met, resulting in a file upload model. This allows the trained file upload model to predict the success rate and time efficiency of uploading a file in segments using different parameters, thereby assisting the user in selecting the most suitable segmentation upload parameters for uploading the file in segments, improving the success rate and time efficiency of segmentation uploads. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating the model training method provided in the embodiments of this application;

[0043] Figure 2 This is a schematic diagram of the BP neural network provided in an embodiment of this application;

[0044] Figure 3 This is a flowchart illustrating the file upload method provided in an embodiment of this application;

[0045] Figure 4 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application;

[0046] Figure 5 This is a schematic diagram of the file upload device provided in the embodiments of this application;

[0047] Figure 6 This is a schematic diagram of the computer device provided in the embodiments of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0049] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more. In the description of this application, the term "comprising" and its variations are open-ended, meaning "including but not limited to." The term "based on" means "at least partially based on." The term "according to" means "at least partially according to." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments."

[0050] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0051] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the application. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0052] See Figure 1 The diagram illustrates a flowchart of a model training method provided in an embodiment of this application. The method includes steps S101-S102, including:

[0053] S101, Obtain sample data formed after the sample file is uploaded in segments according to preset segment upload parameters, wherein the sample data includes the success rate and time information of segment upload corresponding to the segment upload parameters, and the segment upload parameters include: segment attributes and / or upload mode attributes, and the file attributes of the sample file;

[0054] It should be noted that:

[0055] Sample data refers to the data generated during and after each upload of a sample file, performed according to the set fragmented upload parameters, in one or more uploads. This data may include, for example, the transmission rate at different points in time during each upload, the time required for each upload (which can be carried by time information), the average upload rate, and whether the upload was successful (the success rate is determined by the success rate of each fragmented upload). Each upload process can be configured to operate under the same network bandwidth conditions and use the same backend upload interface for fragmented uploads.

[0056] The fragmentation attribute can be used to indicate the relevant parameters of the sample file that need to be processed before it is uploaded in fragments. For example, the partitioning method corresponding to dividing the sample file into multiple fragments (including the number of fragments and the size of each fragment). It should be understood that the sizes of the fragments can be at least partially the same or different.

[0057] The upload mode attribute can be used to indicate the upload method used when uploading in chunks, such as whether to use parallel upload and the number of threads required for parallel upload.

[0058] File attributes can include the file size, file type, and permission attributes of the sample file.

[0059] S102, Based on the sample data, the neural network model is trained iteratively for multiple rounds until the preset training stopping condition is met, thus obtaining the file upload model.

[0060] It should be noted that the neural network model in this embodiment is a computational model inspired by the structure and function of biological nervous systems. It consists of a large number of simple processing units (called neurons or nodes), which form a complex network through connections (similar to synaptic connections between biological neurons). The basic components of the neural network model include:

[0061] Neuron: The basic building block of the model, responsible for receiving input, performing weighted summation, and transforming the summation result through an activation function to generate an output signal.

[0062] Connection weights: The strength of the connections between neurons determines the degree to which the output of neurons in the previous layer affects the next layer.

[0063] Activation function: A non-linear transformation function that determines how a neuron responds to its input, such as Sigmoid, ReLU, Tanh, etc., which increases the expressive power of the network.

[0064] Network architecture: This includes the organization of the input layer, hidden layers, and output layer. The simplest architecture is the feedforward neural network, where information flows unidirectionally from the input layer through a series of hidden layers to the output layer, without repetition. More complex architectures include convolutional neural networks (CNNs), recurrent neural networks (RNNs), and long short-term memory networks (LSTMs).

[0065] Learning rules, such as those in the backpropagation algorithm, are used to adjust the weights in the network to minimize the difference between the predicted output and the actual target value (loss function). Through an iterative optimization process, the neural network can gradually learn complex patterns in the data.

[0066] For example, the training stopping condition may include the number of iterations reaching an iteration threshold and / or the convergence of the neural network model obtained after K consecutive iterations.

[0067] In one optional implementation, the fragmentation attribute includes at least the fragment size, the upload mode attribute includes at least the number of concurrent threads, and the file attribute includes at least the file size.

[0068] Optionally, the fragment attributes may also include the number of fragments, the merging strategy of each fragment after the upload is completed, the upload mode attributes may also include the upload security policy (used to instruct that fragments that meet the conditions are encrypted during fragment upload to enhance the security of data transmission), and the file attributes may also include the file type and permission attributes.

[0069] In some cases, taking a 10GB log file as an example, each upload process can be set to be under the same network bandwidth conditions and using the same backend upload interface to upload the log file in n chunks. The chunk size for each upload is recorded as s1, s2, ..., sn, the number of concurrent threads for each upload is recorded as p1, p2, ..., pn, and the total time required for each successful upload is recorded as t1, t2, ..., tn. Whether each upload is successful is represented by 0 and 1 (1 for success, 0 for failure), thus forming the corresponding sample data.

[0070] In one optional implementation, the neural network model is a backpropagation (BP) neural network model, and the step of performing multiple rounds of iterative training on the neural network model based on the sample data includes:

[0071] During each round of iterative training,

[0072] The sample data is input into the neural network model for forward propagation to obtain the predicted value;

[0073] Based on the error function between the actual output value and the predicted value corresponding to the sample data, the gradient of the parameters of the neural network model is calculated through backpropagation;

[0074] Based on the gradient, the parameters of the neural network model are adjusted using gradient descent to complete this round of iterative training.

[0075] It should be noted that, see Figure 2 In this embodiment, the BP (Backpropagation) neural network model can have a multi-layered structure with sequential communication connections. This multi-layered structure includes at least an input layer, hidden layers, and an output layer. The number of hidden layers is not limited to a single layer; it can also be two or more. Each layer can have at least one neuron. This BP neural network model is a multi-layered feedforward network trained using an error backpropagation algorithm. The BP neural network can learn and store a large number of input-output pattern mappings without needing to reveal the mathematical equations describing these mappings beforehand. Its basic idea is that the learning process consists of two processes: forward propagation of the signal and backward propagation of the error. The learning rule uses the steepest descent method, continuously adjusting the network parameters through backpropagation to minimize the sum of squared errors.

[0076] It's important to clarify the concepts of forward propagation and backpropagation. Forward propagation involves the input data starting from the input layer of the neural network and propagating forward through the network to the output layer. At each layer, neurons calculate their output values ​​based on their weights and activation functions, ultimately producing the network's predicted output. Backpropagation, on the other hand, calculates the error between the actual output and the expected output of the output layer and propagates this error back through the network, using a chain rule to distribute the error across every weight in the network. This means calculating the contribution of each weight to the total error, allowing for the use of gradient descent or other optimization algorithms to adjust the weights in the network and reduce the error in the next iteration.

[0077] For example, the error function can be determined by the mean squared error function; for instance, the mean squared error function can be directly used.

[0078] In one alternative implementation, the predicted value is calculated using an activation function based on the sample data, as well as the weights between layers and the bias of each neuron in the neural network model.

[0079] It should be noted that weights refer to the propagation parameters between two connected layers in a neural network model, and biases refer to the input bias terms configured for each neuron. Before iterative training, each weight is initialized to a random real number in the interval [-1, 1], and each neuron is initialized to a random real number in the interval [0, 1].

[0080] In one specific embodiment, forward propagation can be determined through the following process:

[0081] use This represents the output of the i-th neuron in the l-th layer. If the l-th layer has n neurons, then the output of the i-th neuron in the l-th layer is... l The output can be represented by the following vector:

[0082] and use W l To represent the propagation parameters (e.g., weights) between layer l and layer l+1, use The input bias term (bias) of the i-th neuron in the (l+1)-th layer is represented by... Let the weighted sum of the inputs to the i-th neuron in the (l+1)-th layer be represented. Then, the following expression for forward propagation can be obtained:

[0083]

[0084] a l+1 =g(z) l+1 )

[0085] Here, g() represents the activation function, which can be, for example, the sigmoid function.

[0086] In this embodiment, the predicted output value can be obtained by inputting sample data into a neural network model and performing the aforementioned forward propagation.

[0087] In one optional implementation, the error function is determined based on a mean squared error function and a parameter regularization term, wherein the mean squared error function is used to calculate the mean squared error between the actual output value and the predicted value, and the parameter regularization term is a regularization term related to the weights between layers in the neural network model.

[0088] In one specific embodiment, the error function can be expressed by the following formula:

[0089]

[0090] in, Represents the mean squared error function. s represents the parameter regularization term. l n represents the number of neurons in layer l. l This indicates the number of layers in a neural network model.

[0091] In one optional implementation, the parameters of the neural network model include the weights between layers and the bias of each neuron in the neural network model, and the gradient includes the weight gradient and the bias gradient, wherein the weight gradient is determined based on the partial derivative of the error function with respect to the weights between layers, and the bias gradient is determined based on the partial derivative of the error function with respect to the bias of each neuron.

[0092] In one specific embodiment, The partial derivative of the cost function with respect to the weighted sum of inputs to the i-th neuron in the l-th layer is expressed as:

[0093]

[0094] Therefore, the weight gradient can be expressed as:

[0095]

[0096] The bias gradient can be expressed as:

[0097]

[0098] At this point, this embodiment can be further combined with the above formula to use gradient descent to find the optimal parameters, as shown in the following derivation:

[0099] 1) For l from n l When -1 changes to 1, ΔW l =0, Δb i =0

[0100] 2) For the process of i changing from 1 to m, such that:

[0101]

[0102]

[0103] 3) Update the parameters of the neural network model using the following formula:

[0104]

[0105]

[0106] In one optional implementation, the training stopping condition includes at least one condition where the error calculated by the error function meets a preset error range.

[0107] It should be noted that in this embodiment, when the error calculated by the error function reaches a preset value, that is, falls within the preset error range, it means that the predicted value output by the trained neural network model is sufficiently similar to the true value, that is, the prediction accuracy of the trained neural network model is high enough, and iterative training can be stopped.

[0108] In one optional implementation, the number of the fragmented upload parameters is N sets, the number of the sample data is N sets, and the N sets of fragmented upload parameters correspond one-to-one with the N sets of sample data, where N is a positive integer. Before performing multiple rounds of iterative training on the neural network model based on the sample data, the method further includes:

[0109] The N sets of sample data are preprocessed, wherein the preprocessing includes standardization, normalization, denoising and / or outlier handling.

[0110] Based on the preprocessed N sets of sample data, a training set and a test set are constructed.

[0111] The training set is used for multiple rounds of iterative training of the neural network model, while the test set is used to test the neural network model after the iteration stops.

[0112] In this embodiment, by preprocessing the sample data of each group, the data training and generalization performance can be improved, so as to ensure that the sample data performs well in model training.

[0113] It should be noted that in this embodiment, standardization is performed by transforming the data distribution to a standard normal distribution, i.e., a mean of 0 and a variance of 1. Normalization scales the data to between 0 and 1. Denoising (e.g., filtering) is a process used to reduce or eliminate noise in the data, thereby improving data quality and the accuracy of subsequent analysis and applications. Outlier handling ensures that the data features learned by the model are accurate and reliable.

[0114] In one optional implementation, the step of performing multiple rounds of iterative training on the neural network model based on the sample data until a preset training stopping condition is met to obtain the file upload model includes:

[0115] The neural network model is trained iteratively using the data training set to obtain the test model when the iteration stops.

[0116] The model to be tested is trained once using the data test set, and the generalization ability of the model to be tested is determined based on the training results of the first round of training, wherein the generalization ability is used to indicate whether the model to be tested is overfitting.

[0117] If overfitting occurs, the model to be tested will be used as the file upload model.

[0118] If overfitting occurs, adjust the parameters of the model under test and repeat the above steps until overfitting is not observed.

[0119] It should be noted that the conditions for stopping iteration in this embodiment may include reaching an iteration threshold and / or the convergence of the neural network model obtained after K consecutive iterations. Stopping iteration indicates that the neural network model has met the training requirements and can be further tested to determine whether its generalization ability meets the requirements. Specifically, if the generalization ability indicates that the model under test is not overfitting, the parameters of the model under test can be saved to obtain a file upload model. If the generalization ability indicates that the model under test is overfitting, the steps of performing multiple rounds of iterative training on the neural network model using the data training set and subsequent steps are repeated until an overfitting model under test is obtained as the file upload model.

[0120] See Figure 3 The diagram illustrates a file upload method according to an embodiment of this application. The method includes steps S301-S303, including:

[0121] S301, Obtain the multi-part upload parameters corresponding to the file to be uploaded;

[0122] S302 The multi-part upload parameters are input into the file upload model for prediction to obtain the success rate and time information of the upload of each part corresponding to the multi-part upload parameters, so as to select the target upload parameter from the multi-part upload parameters. The file upload model is trained based on the model training method described above.

[0123] S303, Based on the target fragment upload parameters, upload the file to be uploaded in fragments.

[0124] Accordingly, this application also provides a model training apparatus capable of implementing all the processes of the model training method provided in the above embodiments.

[0125] See Figure 4 The diagram shows a schematic representation of the model training apparatus provided in an embodiment of this application, including:

[0126] The sample data acquisition module 401 is used to acquire sample data formed after a sample file is uploaded in segments according to preset segment upload parameters. The sample data includes the success rate and time information of segment upload corresponding to the segment upload parameters. The segment upload parameters include: segment attributes and / or upload mode attributes, and the file attributes of the sample file.

[0127] The iterative training module 402 is used to perform multiple rounds of iterative training on the neural network model based on the sample data until the preset training stopping condition is met, so as to obtain the file upload model.

[0128] In one optional implementation, the fragmentation attribute includes at least the fragment size, the upload mode attribute includes at least the number of concurrent threads, and the file attribute includes at least the file size.

[0129] In one optional implementation, the neural network model is a backpropagation (BP) neural network model, and the step of performing multiple rounds of iterative training on the neural network model based on the sample data includes:

[0130] During each round of iterative training,

[0131] The sample data is input into the neural network model for forward propagation to obtain the predicted value;

[0132] Based on the error function between the actual output value and the predicted value corresponding to the sample data, the gradient of the parameters of the neural network model is calculated through backpropagation;

[0133] Based on the gradient, the parameters of the neural network model are adjusted using gradient descent to complete this round of iterative training.

[0134] In one alternative implementation, the predicted value is calculated using an activation function based on the sample data, as well as the weights between layers and the bias of each neuron in the neural network model.

[0135] In one optional implementation, the error function is determined based on a mean squared error function and a parameter regularization term, wherein the mean squared error function is used to calculate the mean squared error between the actual output value and the predicted value, and the parameter regularization term is a regularization term related to the weights between layers in the neural network model.

[0136] In one optional implementation, the parameters of the neural network model include the weights between layers and the bias of each neuron in the neural network model, and the gradient includes the weight gradient and the bias gradient, wherein the weight gradient is determined based on the partial derivative of the error function with respect to the weights between layers, and the bias gradient is determined based on the partial derivative of the error function with respect to the bias of each neuron.

[0137] In one optional implementation, the training stopping condition includes at least one condition where the error calculated by the error function meets a preset error range.

[0138] In one optional implementation, the number of the fragmented upload parameters is N sets, the number of the sample data is N sets, and the N sets of fragmented upload parameters correspond one-to-one with the N sets of sample data, where N is a positive integer. Before performing multiple rounds of iterative training on the neural network model based on the sample data, the iterative training module is further configured to:

[0139] The N sets of sample data are preprocessed, wherein the preprocessing includes standardization, normalization, denoising and / or outlier handling.

[0140] Based on the preprocessed N sets of sample data, a training set and a test set are constructed.

[0141] In one optional implementation, the step of performing multiple rounds of iterative training on the neural network model based on the sample data until a preset training stopping condition is met to obtain the file upload model includes:

[0142] The neural network model is trained iteratively using the data training set to obtain the test model when the iteration stops.

[0143] The model to be tested is trained once using the data test set, and the generalization ability of the model to be tested is determined based on the training results of the first round of training, wherein the generalization ability is used to indicate whether the model to be tested is overfitting.

[0144] If overfitting occurs, the model to be tested will be used as the file upload model.

[0145] If overfitting occurs, adjust the parameters of the model under test and repeat the above steps until overfitting is not observed.

[0146] See Figure 5 The diagram illustrates the structure of a file upload device provided in an embodiment of this application, including:

[0147] The parameter acquisition module 501 is used to acquire the multi-part upload parameters corresponding to the file to be uploaded;

[0148] The prediction module 502 is used to input the multi-group segmented upload parameters into the file upload model for prediction, and obtain the success rate and time information of segmented upload corresponding to each of the multi-group segmented upload parameters, so as to select the target segmented upload parameter from the multi-group segmented upload parameters, wherein the file upload model is trained based on the model training method described above.

[0149] The chunked upload module 503 is used to upload the file to be uploaded in chunks based on the target chunked upload parameters.

[0150] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0151] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the method described in any of the above embodiments.

[0152] This application also provides a computer device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the steps of any of the methods described above.

[0153] See Figure 6 The computer device in this embodiment includes a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601, such as a model training and / or file upload program. When the processor 601 executes the computer program, it implements the steps in the various model training and / or file upload method embodiments described above, for example... Figure 1 Steps S101-S102 and / or shown Figure 3 The steps S301-S303 are shown.

[0154] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 602 and executed by the processor 601 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device.

[0155] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor 601 and a memory 602. Those skilled in the art will understand that the schematic diagram is merely an example of a computer device and does not constitute a limitation on the computer device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0156] The processor 601 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or processor 601 can be any conventional processor. The processor 601 is the control center of the computer device, connecting various parts of the entire computer device through various interfaces and lines.

[0157] The memory 602 can be used to store the computer programs and / or modules. The processor 601 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 602 and calling the data stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0158] Wherein, if the modules / units integrated into the computer device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 601, it can implement the steps of the various method embodiments described above. Wherein, the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0159] In summary, the embodiments of this application have at least the following beneficial effects:

[0160] In this embodiment, sample data is obtained by uploading a sample file in segments according to preset segmentation parameters. The sample data includes the success rate and time information of the segmentation upload corresponding to the segmentation upload parameters. The segmentation upload parameters include segmentation attributes and / or upload mode attributes, as well as the file attributes of the sample file. Based on the sample data, a neural network model is iteratively trained multiple times until a preset training stopping condition is met, resulting in a file upload model. This allows the trained file upload model to predict the success rate and time efficiency of uploading a file in segments using different parameters, thereby assisting the user in selecting the most suitable segmentation upload parameters for uploading the file in segments, improving the success rate and time efficiency of segmentation uploads.

[0161] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware platforms, or it can be implemented entirely by hardware. Based on this understanding, all or part of the technical solutions of this application that contribute to the background technology can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0162] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A model training method, characterized in that, include: The sample data is obtained after the sample file is uploaded in segments according to preset segment upload parameters. The sample data includes the success rate and time information of segment upload corresponding to the segment upload parameters. The segment upload parameters include: segment attributes and / or upload mode attributes, and the file attributes of the sample file. Based on the sample data, the neural network model is trained iteratively for multiple rounds until the preset training stopping condition is met, thereby obtaining a file upload model. The file upload model is used to predict the input multi-part upload parameters and obtain the success rate and time information of the upload of each part corresponding to the multi-part upload parameters, so as to select the target upload parameter from the multi-part upload parameters.

2. The model training method as described in claim 1, characterized in that, The fragmentation attribute includes at least the fragment size, the upload mode attribute includes at least the number of concurrent threads, and the file attribute includes at least the file size.

3. The model training method as described in claim 1, characterized in that, The neural network model is a backpropagation (BP) neural network model. The process of performing multiple rounds of iterative training on the neural network model based on the sample data includes: During each round of iterative training, The sample data is input into the neural network model for forward propagation to obtain the predicted value; Based on the error function between the actual output value and the predicted value corresponding to the sample data, the gradient of the parameters of the neural network model is calculated through backpropagation; Based on the gradient, the parameters of the neural network model are adjusted using gradient descent to complete this round of iterative training.

4. The model training method as described in claim 3, characterized in that, The predicted value is calculated using an activation function based on the sample data, the weights between layers in the neural network model, and the bias of each neuron.

5. The model training method as described in claim 3, characterized in that, The error function is determined based on the mean squared error function and the parameter regularization term. The mean squared error function is used to calculate the mean squared error between the actual output value and the predicted value. The parameter regularization term is a regularization term related to the weights between the layers in the neural network model.

6. The model training method as described in claim 3, characterized in that, The parameters of the neural network model include the weights between each layer and the bias of each neuron. The gradient includes the weight gradient and the bias gradient. The weight gradient is determined based on the partial derivative of the error function with respect to the weights between each layer, and the bias gradient is determined based on the partial derivative of the error function with respect to the bias of each neuron.

7. The model training method according to any one of claims 3-6, characterized in that, The training stopping condition includes at least one condition: the error calculated by the error function meets a preset error range.

8. The model training method according to any one of claims 1-6, characterized in that, The number of the segmented upload parameters is N sets, the number of the sample data is N sets, and the N sets of segmented upload parameters correspond one-to-one with the N sets of sample data, where N is a positive integer. Before performing multiple rounds of iterative training on the neural network model based on the sample data, the method further includes: The N sets of sample data are preprocessed, wherein the preprocessing includes standardization, normalization, denoising and / or outlier handling. Based on the preprocessed N sets of sample data, a training set and a test set are constructed.

9. The model training method as described in claim 8, characterized in that, The step of performing multiple rounds of iterative training on the neural network model based on the sample data until a preset training stopping condition is met to obtain the file upload model includes: The neural network model is trained iteratively using the data training set to obtain the test model when the iteration stops. The model to be tested is trained once using the data test set, and the generalization ability of the model to be tested is determined based on the training results of the first round of training, wherein the generalization ability is used to indicate whether the model to be tested is overfitting. If overfitting occurs, the model to be tested will be used as the file upload model. If overfitting occurs, adjust the parameters of the model under test and repeat the above steps until overfitting is not observed.

10. A file upload method, characterized in that, include: Retrieve the multi-part upload parameters corresponding to the file to be uploaded; The multi-part upload parameters are input into the file upload model for prediction to obtain the success rate and time information of the upload of each part corresponding to the multi-part upload parameters, so as to select the target upload parameter from the multi-part upload parameters. The file upload model is trained based on the model training method of any one of claims 1-9. Based on the target chunk upload parameters, the file to be uploaded is uploaded in chunks.

11. A model training device, characterized in that, include: The sample data acquisition module is used to acquire sample data formed after a sample file is uploaded in segments according to preset segment upload parameters. The sample data includes the success rate and time information of segment upload corresponding to the segment upload parameters. The segment upload parameters include: segment attributes and / or upload mode attributes, and the file attributes of the sample file. The iterative training module is used to perform multiple rounds of iterative training on the neural network model based on the sample data until a preset training stopping condition is met to obtain a file upload model. The file upload model is used to predict the input multi-part upload parameters to obtain the success rate and time information of the upload of each part corresponding to the multi-part upload parameters, so as to select the target upload parameter from the multi-part upload parameters.

12. A file upload device, characterized in that, include: The parameter acquisition module is used to acquire the multi-part upload parameters corresponding to the file to be uploaded. The prediction module is used to input the multi-part upload parameters into the file upload model for prediction, and obtain the success rate and time information of the upload of each part corresponding to the multi-part upload parameters, so as to select the target upload parameter from the multi-part upload parameters. The file upload model is trained based on the model training method of any one of claims 1-9. The chunked upload module is used to upload the file to be uploaded in chunks based on the target chunked upload parameters.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-10.

14. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method described in any one of claims 1-10.

15. A computer device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1-10.

Images