File uploading method and device, computer device and storage device

Abstract

The invention belongs to the technical field of data processing, and particularly relates to a file uploading method and device, a computer device and a storage device. The file uploading method comprises the following steps: S100, acquiring a file needing to be uploaded, splitting the file into blocks based on the file and a real-time network condition, and uploading the blocks to a server; s300, after the uploading of the blocks is completed, splitting the blocks based on the residual data and the real-time network condition, and uploading the blocks to a server; and S500, after each time of block uploading is completed, repeating the step S300 until the file is uploaded to the server. According to the method, each block is split based on the size of the data needing to be uploaded and the real-time network condition, so that the dynamic splitting of the file is realized, the variability of the network condition can be dealt with, the continuity and reliability of file uploading are ensured, and the condition of data loss is avoided.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a file uploading method, device, computer device and storage device. Background Art

[0002] Currently, the file upload function in most Web applications relies on traditional file data stream or multi-part upload technology. Traditional file data stream upload technology is based on uploading the entire file, and multi-part upload technology is to divide the file into multiple small parts and then upload them one by one. Whether it is traditional file data stream or multi-part upload technology, it is easy to be interrupted when the network is unstable, and it cannot be automatically restored, resulting in data loss. Summary of the invention

[0003] Based on this, the purpose of the present invention is to provide a file uploading method, device, computer device and storage device to fundamentally solve the problem that the existing file uploading technology is prone to interruption when the network is unstable and cannot be automatically restored, resulting in data loss.

[0004] The embodiment of the present invention is implemented by providing a file uploading method, comprising the following steps:

[0005] S100, obtaining a file to be uploaded, splitting the file into blocks based on the file and real-time network conditions, and uploading the blocks to a server;

[0006] S300, after the blocks are uploaded, split the blocks based on the remaining data and the real-time network status, and upload the blocks to the server;

[0007] S500, after each block upload is completed, repeat S300 until the file has been uploaded to the server.

[0008] In some implementations, the file uploading method further includes the following steps:

[0009] When uploading the blocks to the server, detecting a real-time upload rate;

[0010] Determining whether the real-time upload rate is greater than an upload rate threshold;

[0011] If so, continue uploading the blocks to the server;

[0012] If not, suspend uploading the blocks to the server and mark the paused position. After determining that the real-time upload rate is greater than the upload rate threshold, split the blocks based on the marked position and upload the blocks to the server.

[0013] In some embodiments, the pausing uploading the blocks to the server and marking the paused position comprises the following steps:

[0014] The uploading of the block to the server is paused, and a mark is made at the position of the nth character before the pause position.

[0015] In some implementations, after pausing uploading the blocks to the server, the following steps are included:

[0016] Based on the paused blocks, obtaining uploaded data and uploading remaining data;

[0017] The value of n is obtained according to the ratio of the uploaded data to the remaining uploaded data.

[0018] In some implementations, the file uploading method further includes the following steps:

[0019] When uploading the blocks to the server, extracting upload features;

[0020] Inputting the uploaded features into a machine learning model and outputting potential risk results;

[0021] Upload redundant data to the server based on the potential risk results.

[0022] In some implementations, the file uploading method further includes the following steps:

[0023] After uploading the blocks to the server, the server assembles the blocks and optimizes the blocks according to the file type during the assembly process;

[0024] The file type includes at least one of document, audio and video.

[0025] In some implementations, the file uploading method further includes the following steps:

[0026] After splitting the blocks, verifying the integrity of the blocks;

[0027] If the integrity check of the block fails, re-split the block;

[0028] After uploading the blocks to the server, the server verifies the integrity of the uploaded blocks;

[0029] If the integrity check of the uploaded block fails, re-upload the block.

[0030] Another embodiment of the present invention further provides a file uploading device, comprising:

[0031] A first splitting unit is used to obtain a file to be uploaded, split the file into blocks based on the file and the real-time network status, and upload the blocks to the server;

[0032] A second splitting unit is used to split the blocks into the blocks based on the remaining data and the real-time network status after the blocks are uploaded, and upload the blocks to the server;

[0033] After each block upload is completed, the second splitting unit is repeatedly executed until the file is uploaded to the server.

[0034] Another embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the file uploading method as described in any one of the embodiments of the present invention are implemented.

[0035] Another embodiment of the present invention further provides a storage device, wherein the storage device stores a computer program, and the computer program can be executed to implement the steps of the file uploading method as described in any one of the embodiments of the present invention.

[0036] The file uploading method provided by the embodiment of the present invention splits the files into blocks based on the size of the data to be uploaded and the real-time network conditions, thereby realizing dynamic file splitting, being able to cope with the variability of the network conditions, and ensuring the continuity and reliability of the file uploading, thereby avoiding the occurrence of data loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flowchart of a file uploading method provided by an embodiment of the present invention;

[0038] Figure 2A-2C is a schematic diagram of splitting into blocks provided by an embodiment of the present invention;

[0039] Figure 3 It is a structural schematic diagram of a file uploading device provided by an embodiment of the present invention;

[0040] The following specific implementations will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0041] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0042] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0044] Embodiment 1

[0045] See also Figure 1 , which shows a file upload method in a first embodiment of the present invention. For the sake of convenience, only the part related to the embodiment of the present invention is shown. The file upload method provided by the embodiment of the present invention includes the following steps:

[0046] S100, obtaining a file to be uploaded, splitting the file into blocks based on the file and real-time network conditions, and uploading the blocks to the server;

[0047] S300, after the blocks are uploaded, split the blocks based on the remaining data and the real-time network status, and upload the blocks to the server;

[0048] S500: After each block upload is completed, repeat S300 until the file has been uploaded to the server.

[0049] In this embodiment, the client first obtains the file to be uploaded to the server, for example, the user transfers the file in the local environment to the client, or the client obtains the file from the network environment. After obtaining the file, the blocks are split based on the file and the real-time network situation, and then the split blocks are uploaded to the server. After the split blocks are uploaded, the uploaded blocks are removed from the file as the remaining data, and the blocks are split based on the remaining data and the real-time network situation, and then the split blocks are uploaded to the server. After each block is uploaded, the operation of splitting the blocks based on the remaining data and the real-time network situation and uploading the split blocks to the server is repeated until the last block has been uploaded to the server, and the file has been uploaded to the server.

[0050] refer to Figure 2AAfter obtaining the file, split the block 1 based on the file and the real-time network situation, and then upload the split block 1 to the server. After the split block 1 is uploaded, remove the uploaded block 1 from the file as the remaining data, refer to Figure 2B , split out block 2 based on the remaining data and real-time network conditions, and then upload the split block 2 to the server. After the split block 2 is uploaded, remove the uploaded blocks 1 and 2 from the file as the remaining data, refer to Figure 2C , split out block 3 based on the remaining data and the real-time network situation, and then upload the split block 3 to the server. Repeat the operation of splitting out blocks and uploading blocks until the last block has been uploaded to the server, and the file has been uploaded to the server.

[0051] For example, obtain a file to be uploaded, the size of the file is 100G, split it into blocks based on the file and the real-time network situation, the size of each block is 10G, and upload the split blocks to the server. After the split blocks are uploaded to the server, the size of the remaining data is 90G, split it into blocks based on the remaining data and the real-time network situation, the size of each block is 11G, and upload the split blocks to the server. After the split blocks are uploaded to the server, the size of the remaining data is 79G, split it into blocks based on the remaining data and the real-time network situation, the size of each block is 9G, and upload the split blocks to the server. Repeat the operation of splitting into blocks and uploading blocks until the last block has been uploaded to the server, and the file has been uploaded to the server.

[0052] Splitting blocks based on files and real-time network conditions, and splitting blocks based on remaining data and real-time network conditions, both refer to splitting blocks based on file data and real-time network conditions, but based on files refers to the complete data of the file, and based on remaining data refers to the remaining data in the file after removing the uploaded blocks. The larger the file data and the better the real-time network conditions, the larger the split blocks are, thereby achieving faster and more reliable uploading of blocks to the server. When splitting, the file data and the real-time network conditions have different weights. Since the smoothness of the network is more important, the real-time network conditions have a greater weight than the file data, thereby further improving the speed and reliability of data upload.

[0053] Based on the status of the uploaded blocks, the remaining data will change, and the real-time network status will change in real time. Through the above file uploading method, each block is split based on the size of the data to be uploaded and the real-time network status, so as to realize dynamic file splitting, cope with the variability of network conditions, ensure the continuity and reliability of file upload, and avoid data loss.

[0054] In addition, the file upload method is compatible with multiple platforms and multiple browsers. In the existing file upload methods, in the use environment of multiple platforms and multiple browsers, users often face the problem of incompatibility of upload tools. The file upload method of the present invention has taken into account the uniformity and compatibility across browsers when it was designed. No matter which mainstream browser or operating system the user uses, the consistency and smoothness of the upload function can be guaranteed. The file upload method of the present invention deliberately optimizes the interface for interacting with the browser to ensure high adaptability and scalability between different platforms.

[0055] In some specific embodiments of the present invention, the file uploading method further includes the following steps:

[0056] When uploading chunks to the server, detect the real-time upload rate;

[0057] Determine whether the real-time upload rate is greater than the upload rate threshold;

[0058] If so, continue uploading the chunks to the server;

[0059] If not, pause uploading the blocks to the server and mark the pause position. After determining that the real-time upload rate is greater than the upload rate threshold, split the blocks based on the marked position and upload the blocks to the server.

[0060] In this embodiment, each time a block is uploaded to the server, the real-time upload rate is detected. It is determined whether the real-time upload rate is greater than the upload rate threshold, for example, the upload rate threshold is 10 kb / s. If so, it is determined that the block is being uploaded to the server normally, and the block is continuously uploaded to the server. If not, it is determined that an abnormality occurs during the process of uploading the block to the server, and the uploading of the block to the server is suspended, and the pause position is marked. At this time, the real-time upload rate is still continuously detected. After it is determined that the real-time upload rate is greater than the upload rate threshold, new blocks are split based on the marked position, and the blocks are uploaded to the server.

[0061] In the above operation of splitting new blocks based on the marking position and uploading the blocks to the server, the remaining data uploaded later can be used as new blocks based on the marking position of the block, and the blocks can be uploaded to the server; or blocks can be split from the remaining data uploaded later based on the marking position of the block, combined with the real-time network conditions, and then the blocks can be uploaded to the server, and the operations of splitting and uploading blocks can be repeated until the blocks for marking operations have been uploaded to the server.

[0062] By judging whether the real-time upload rate is normal, it is possible to distinguish between normal upload and abnormal upload. In the case of normal upload, data can be continuously uploaded. In the case of abnormal upload, data can be uploaded again after the abnormality is eliminated, which can avoid data upload errors. In addition, by resuming the upload from the breakpoint, it is possible to continue uploading from the location where the last upload was abnormal, rather than uploading the blocks from the beginning, which helps to improve the upload efficiency.

[0063] In some specific embodiments of the present invention, pausing uploading the blocks to the server and marking the paused position includes the following steps:

[0064] Pause uploading the chunk to the server and mark the nth character before the pause position.

[0065] In this embodiment, in the case of data upload exception, the uploading of the blocks to the server is suspended, and the position of the nth character before the pause position is marked. After it is determined that the real-time upload rate is greater than the upload rate threshold, new blocks are split based on the marked position and the blocks are uploaded to the server. Marking the position of the nth character before the pause position, and then splitting and uploading the blocks based on the marked position can ensure the integrity of the file and avoid data loss.

[0066] After uploading the blocks that have an upload exception and continue to upload after the exception is eliminated, they are uploaded to the server. Due to the proximity of the marking position, data may be uploaded repeatedly. The server processes the blocks to remove the duplicate data.

[0067] In some specific embodiments of the present invention, after pausing uploading the blocks to the server, the following steps are included:

[0068] Based on the paused blocks, get the uploaded data and upload the remaining data;

[0069] The value of n is obtained according to the ratio of the uploaded data to the remaining data to be uploaded.

[0070] In this embodiment, in the event of data upload anomalies, uploading blocks to the server is suspended. Based on the suspended blocks, the uploaded data and the remaining upload data are obtained. After obtaining the uploaded data and the remaining upload data, the value of n is obtained according to the ratio of the uploaded data to the remaining upload data. The position of the nth character before the pause position is marked, and after determining that the real-time upload rate is greater than the upload rate threshold, new blocks are split based on the marked position, and the blocks are uploaded to the server. Calculating the value of n can make the marking position more reasonable, avoid the situation where the value of n is too large, and the data becomes more complicated when the server processes the data later, and avoid the situation where the value of n is too small, and there is still a risk of data being incorrect.

[0071] In some specific embodiments of the present invention, the file uploading method further includes the following steps:

[0072] When uploading the chunks to the server, extract the upload features;

[0073] Input the uploaded features into the machine learning model and output potential risk results;

[0074] Upload redundant data to the server based on potential risk results.

[0075] In this embodiment, each time a block is uploaded to the server, the upload features of the upload process are extracted. The upload features are input into the trained machine learning model, and the potential risk results are output. According to the potential risk results, redundant data is uploaded to the server. The upload features include real-time upload rate, block data size, uploaded data size, to-be-uploaded data size, real-time network conditions, etc.

[0076] Machine learning methods can be mainly divided into the following categories: supervised learning, unsupervised learning, reinforcement learning, semi-supervised learning, self-supervised learning, multi-instance learning and active learning. Supervised learning is one of the most commonly used machine learning methods. It trains and learns a model by knowing the mapping relationship between input and output data, and then uses this model to predict new input data. Common supervised learning algorithms include linear regression, logistic regression, decision tree, random forest, neural network, etc. Unsupervised learning is to let the computer automatically learn and identify patterns in data without pre-set goals or labels. Common unsupervised learning algorithms include cluster analysis, dimensionality reduction, association rule mining, etc. Reinforcement learning is to let the computer automatically learn and optimize its own behavior by interacting with the environment. Common reinforcement learning algorithms include Q-learning, SARSA, deep Q network, policy gradient, etc. Semi-supervised learning combines the characteristics of supervised learning and unsupervised learning, and uses some labeled data and some unlabeled data for training and prediction. Common semi-supervised learning algorithms include generative models, collaborative training, etc. Self-supervised learning is a form of learning between supervision and unsupervised learning. It uses implicit information in the data to obtain labels, does not require people to annotate the data in advance, and uses neural network models to learn automatically. Multi-instance learning belongs to the supervised learning method. The training set contains labeled "sample packages", and each "package" contains some unlabeled examples. Active learning is to actively select samples based on the amount of information in the samples when there are not enough sample labels, and query labels from human "mentors" to supplement the samples. In the above file upload method, any of the above machine learning methods can be used.

[0077] The above-mentioned file upload method is integrated with a machine learning method, which has the ability to predict and identify potential risk results. When potential risk results are identified, it can actively upload redundant data so as to respond quickly and make automatic corrections when potential risks occur.

[0078] In some specific embodiments of the present invention, the file uploading method further includes the following steps:

[0079] After uploading the chunks to the server, the server assembles the chunks and optimizes the chunks according to the file type during the assembly process;

[0080] The file type includes at least one of document, audio and video.

[0081] In this embodiment, after uploading the blocks to the server, the server assembles the uploaded blocks. When assembling the uploaded files, the files are optimized according to the file type.

[0082] The file type includes at least one of documents, audio and video. For document files, the optimization processing may be, for example, automatic identification and indexing; for audio files, the optimization processing may be, for example, format conversion and quality enhancement; for video files, the optimization processing may be, for example, automatic transcoding and quality enhancement. The above file upload method greatly shortens the file processing time through real-time analysis and processing, thereby providing a more efficient data management method.

[0083] In some specific embodiments of the present invention, the file uploading method further includes the following steps:

[0084] After splitting the blocks, check the integrity of the blocks;

[0085] If the integrity check of the block fails, split the block again;

[0086] After uploading the chunks to the server, the server verifies the integrity of the uploaded chunks;

[0087] If the integrity check of the uploaded chunk fails, re-upload the chunk.

[0088] In this embodiment, after each block is split, the integrity of the block is checked. If the integrity check of the block passes, the block is uploaded to the server. If the integrity check of the block fails, the block is split again until the integrity check of the block passes, and then the block is uploaded to the server. After uploading the block to the server, the server verifies the integrity of the uploaded block. If the integrity check of the uploaded block passes, there is no need to re-upload the block. If the integrity check of the uploaded block fails, the block is re-uploaded until the integrity check of the uploaded block passes, and there is no need to re-upload the block. The above-mentioned file upload method can ensure that the integrity of the file is deeply checked, thereby achieving bit-level integrity protection of the file.

[0089] Embodiment 2

[0090] See also Figure 3 , is a schematic diagram of the structure of a file uploading device provided by the second embodiment of the present invention. For the convenience of description, only the part related to the embodiment of the present invention is shown. The file uploading device includes:

[0091] The first splitting unit 100 is used to obtain the file to be uploaded, split the file into blocks based on the file and the real-time network status, and upload the blocks to the server;

[0092] The second splitting unit 300 is used to split the blocks into blocks based on the remaining data and the real-time network status after the blocks are uploaded, and upload the blocks to the server;

[0093] After each block upload is completed, the second splitting unit 300 is repeatedly executed until the file is uploaded to the server.

[0094] In this embodiment, the client first obtains the file to be uploaded to the server, for example, the user transfers the file in the local environment to the client, or the client obtains the file from the network environment. After obtaining the file, the blocks are split based on the file and the real-time network situation, and then the split blocks are uploaded to the server. After the split blocks are uploaded, the uploaded blocks are removed from the file as the remaining data, and the blocks are split based on the remaining data and the real-time network situation, and then the split blocks are uploaded to the server. After each block is uploaded, the operation of splitting the blocks based on the remaining data and the real-time network situation and uploading the split blocks to the server is repeated until the last block has been uploaded to the server, and the file has been uploaded to the server.

[0095] refer to Figure 2A After obtaining the file, split the block 1 based on the file and the real-time network situation, and then upload the split block 1 to the server. After the split block 1 is uploaded, remove the uploaded block 1 from the file as the remaining data, refer to Figure 2B, split out block 2 based on the remaining data and real-time network conditions, and then upload the split block 2 to the server. After the split block 2 is uploaded, remove the uploaded blocks 1 and 2 from the file as the remaining data, refer to Figure 2C , split out block 3 based on the remaining data and the real-time network situation, and then upload the split block 3 to the server. Repeat the operation of splitting out blocks and uploading blocks until the last block has been uploaded to the server, and the file has been uploaded to the server.

[0096] For example, obtain a file to be uploaded, the size of the file is 100G, split it into blocks based on the file and the real-time network situation, the size of each block is 10G, and upload the split blocks to the server. After the split blocks are uploaded to the server, the size of the remaining data is 90G, split it into blocks based on the remaining data and the real-time network situation, the size of each block is 11G, and upload the split blocks to the server. After the split blocks are uploaded to the server, the size of the remaining data is 79G, split it into blocks based on the remaining data and the real-time network situation, the size of each block is 9G, and upload the split blocks to the server. Repeat the operation of splitting into blocks and uploading blocks until the last block has been uploaded to the server, and the file has been uploaded to the server.

[0097] Splitting blocks based on files and real-time network conditions, and splitting blocks based on remaining data and real-time network conditions, both refer to splitting blocks based on file data and real-time network conditions, but based on files refers to the complete data of the file, and based on remaining data refers to the remaining data in the file after removing the uploaded blocks. The larger the file data and the better the real-time network conditions, the larger the split blocks are, thereby achieving faster and more reliable uploading of blocks to the server. When splitting, the file data and the real-time network conditions have different weights. Since the smoothness of the network is more important, the real-time network conditions have a greater weight than the file data, thereby further improving the speed and reliability of data upload.

[0098] Based on the situation of the uploaded blocks, the remaining data will change, and the real-time network situation will change in real time. Through the above-mentioned file uploading device, each block is split based on the size of the data to be uploaded and the real-time network situation, so as to realize dynamic file splitting, cope with the variability of network conditions, and ensure the continuity and reliability of file uploading, and avoid data loss.

[0099] In addition, the file upload device is compatible with multiple platforms and multiple browsers. In existing file upload devices, users often face the problem of incompatibility of upload tools in the use environment of multiple platforms and multiple browsers. The file upload device of the present invention has taken into account the uniformity and compatibility across browsers when it was designed. No matter which mainstream browser or operating system the user uses, the consistency and smoothness of the upload function can be guaranteed. The file upload device of the present invention has specially optimized the interface for interacting with the browser to ensure high adaptability and scalability between different platforms.

[0100] In some specific embodiments of the present invention, the file uploading device further includes:

[0101] A rate detection unit, used to detect the real-time upload rate when uploading the blocks to the server;

[0102] A rate determination unit, used to determine whether the real-time upload rate is greater than an upload rate threshold;

[0103] A first uploading unit, configured to continuously upload the blocks to the server if it is determined that the real-time upload rate is greater than the upload rate threshold;

[0104] The second uploading unit is used to pause uploading the blocks to the server and mark the paused position if it is determined that the real-time upload rate is not greater than the upload rate threshold. After it is determined that the real-time upload rate is greater than the upload rate threshold, the blocks are split based on the marked position and uploaded to the server.

[0105] In this embodiment, each time a block is uploaded to the server, the real-time upload rate is detected. It is determined whether the real-time upload rate is greater than the upload rate threshold, for example, the upload rate threshold is 10 kb / s. If so, it is determined that the block is being uploaded to the server normally, and the block is continuously uploaded to the server. If not, it is determined that an abnormality occurs during the process of uploading the block to the server, and the uploading of the block to the server is suspended, and the pause position is marked. At this time, the real-time upload rate is still continuously detected. After it is determined that the real-time upload rate is greater than the upload rate threshold, new blocks are split based on the marked position, and the blocks are uploaded to the server.

[0106] In the above-mentioned operation of splitting new blocks based on the marking position and uploading the blocks to the server, the remaining data uploaded later can be used as new blocks based on the marking position of the block, and the blocks can be uploaded to the server; or the blocks can be split from the remaining data uploaded later based on the marking position of the block, combined with the real-time network conditions, and then the blocks can be uploaded to the server, and the operations of splitting and uploading blocks can be repeated until the block for marking operation has been uploaded to the server.

[0107] By judging whether the real-time upload rate is normal, it is possible to distinguish between normal upload and abnormal upload. In the case of normal upload, data can be continuously uploaded. In the case of abnormal upload, data can be uploaded again after the abnormality is eliminated, which can avoid data upload errors. In addition, by resuming the upload from the breakpoint, it is possible to continue uploading from the location where the last upload was abnormal, rather than uploading the blocks from the beginning, which helps to improve the upload efficiency.

[0108] In some specific embodiments of the present invention, the judging unit includes:

[0109] The pause module is used to pause uploading the blocks to the server and mark the position of the nth character before the pause position.

[0110] In this embodiment, in the case of data upload exception, the uploading of the blocks to the server is suspended, and the position of the nth character before the pause position is marked. After it is determined that the real-time upload rate is greater than the upload rate threshold, new blocks are split based on the marked position and the blocks are uploaded to the server. Marking the position of the nth character before the pause position, and then splitting and uploading the blocks based on the marked position can ensure the integrity of the file and avoid data loss.

[0111] After uploading the blocks that have an upload exception and continue to upload after the exception is eliminated, they are uploaded to the server. Due to the proximity of the marking position, data may be uploaded repeatedly. The server processes the blocks to remove the duplicate data.

[0112] In some specific embodiments of the present invention, the pause module includes:

[0113] The acquisition submodule is used to obtain the uploaded data and upload the remaining data based on the paused blocks;

[0114] The value obtaining submodule is used to obtain the value of n according to the ratio of the uploaded data to the remaining uploaded data.

[0115] In this embodiment, in the event of data upload anomalies, uploading blocks to the server is suspended. Based on the suspended blocks, the uploaded data and the remaining upload data are obtained. After obtaining the uploaded data and the remaining upload data, the value of n is obtained according to the ratio of the uploaded data to the remaining upload data. The position of the nth character before the pause position is marked, and after determining that the real-time upload rate is greater than the upload rate threshold, new blocks are split based on the marked position, and the blocks are uploaded to the server. Calculating the value of n can make the marking position more reasonable, avoid the situation where the value of n is too large, and the data becomes more complicated when the server processes the data later, and avoid the situation where the value of n is too small, and there is still a risk of data being incorrect.

[0116] In some specific embodiments of the present invention, the file uploading device further includes:

[0117] A feature extraction unit, used for extracting upload features when uploading the blocks to the server;

[0118] A result output unit, which is used to input the uploaded features into the machine learning model and output potential risk results;

[0119] The data uploading unit is used to upload redundant data to the server according to the potential risk results.

[0120] In this embodiment, each time a block is uploaded to the server, the upload features of the upload process are extracted. The upload features are input into the trained machine learning model, and the potential risk results are output. According to the potential risk results, redundant data is uploaded to the server. The upload features include real-time upload rate, block data size, uploaded data size, to-be-uploaded data size, real-time network conditions, etc.

[0121] Machine learning methods can be mainly divided into the following categories: supervised learning, unsupervised learning, reinforcement learning, semi-supervised learning, self-supervised learning, multi-instance learning and active learning. Supervised learning is one of the most commonly used machine learning methods. It trains and learns a model by knowing the mapping relationship between input and output data, and then uses this model to predict new input data. Common supervised learning algorithms include linear regression, logistic regression, decision tree, random forest, neural network, etc. Unsupervised learning is to let the computer automatically learn and identify patterns in data without pre-set goals or labels. Common unsupervised learning algorithms include cluster analysis, dimensionality reduction, association rule mining, etc. Reinforcement learning is to let the computer automatically learn and optimize its own behavior by interacting with the environment. Common reinforcement learning algorithms include Q-learning, SARSA, deep Q network, policy gradient, etc. Semi-supervised learning combines the characteristics of supervised learning and unsupervised learning, and uses some labeled data and some unlabeled data for training and prediction. Common semi-supervised learning algorithms include generative models, collaborative training, etc. Self-supervised learning is a form of learning between supervision and unsupervised learning. It uses implicit information in the data to obtain labels, does not require people to annotate the data in advance, and uses neural network models to learn automatically. Multi-instance learning belongs to the supervised learning method. The training set contains labeled "sample packages", and each "package" contains some unlabeled examples. Active learning is to actively select samples based on the amount of information in the samples when the sample labels are insufficient, and query the labels from human "mentors" to supplement the samples. In the above-mentioned file uploading device, any of the above-mentioned machine learning methods can be used.

[0122] The above-mentioned file upload device is integrated with a machine learning method, has the ability to predict and identify potential risk results, and can actively upload redundant data when potential risk results are identified, so as to respond quickly and perform automatic correction when potential risks occur.

[0123] In some specific embodiments of the present invention, the file uploading device further includes:

[0124] The optimization processing unit is used for assembling the blocks after uploading the blocks to the server, and optimizing the blocks according to the file type during the assembly process;

[0125] The file type includes at least one of document, audio and video.

[0126] In this embodiment, after uploading the blocks to the server, the server assembles the uploaded blocks. When assembling the uploaded files, the files are optimized according to the file type.

[0127] The file type includes at least one of documents, audio and video. For document files, the optimization processing may be, for example, automatic identification and indexing; for audio files, the optimization processing may be, for example, format conversion and quality enhancement; for video files, the optimization processing may be, for example, automatic transcoding and quality enhancement. The above-mentioned file uploading device greatly shortens the file processing time through real-time analysis and processing, thereby providing a more efficient data management method.

[0128] In some specific embodiments of the present invention, the file uploading device further includes:

[0129] A first integrity checking unit, used for checking the integrity of the blocks after the blocks are split;

[0130] A re-splitting unit, used for re-splitting the blocks if the integrity check of the blocks fails;

[0131] A second integrity checking unit, configured to enable the server to check the integrity of the uploaded blocks after uploading the blocks to the server;

[0132] The re-upload unit is used to re-upload the chunk if the integrity check of the uploaded chunk fails.

[0133] In this embodiment, after each block is split, the integrity of the block is checked. If the integrity check of the block passes, the block is uploaded to the server. If the integrity check of the block fails, the block is split again until the integrity check of the block passes, and then the block is uploaded to the server. After uploading the block to the server, the server checks the integrity of the uploaded block. If the integrity check of the uploaded block passes, there is no need to re-upload the block. If the integrity check of the uploaded block fails, the block is re-uploaded until the integrity check of the uploaded block passes, and there is no need to re-upload the block. The above-mentioned file upload device can ensure that the integrity of the file is deeply checked, thereby achieving bit-level integrity protection for the file.

[0134] An embodiment of the present invention provides a computer device, the computer device comprising a processor, the processor being configured to implement the steps of the file uploading method as described above when executing a computer program stored in a memory.

[0135] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program (instructions) is stored. When the computer program (instructions) is executed by a processor, the steps of the file uploading method as described above are implemented.

[0136] Exemplarily, the computer program may be divided into one or more modules, one or more modules are stored in a memory and executed by a processor to complete the present invention. One or more modules may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in a computer device. For example, the computer program may be divided into the steps of the file uploading method provided by each of the above method embodiments.

[0137] Those skilled in the art will appreciate that the above description of the computer device is merely an example and does not constitute a limitation on the computer device. The computer device may include more or fewer components than described above, or a combination of certain components, or different components, such as input and output devices, network access devices, buses, etc.

[0138] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

[0139] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the computer device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as an interface display function, an interface interaction function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as a map interface, a selection interface, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0140] If the module / unit integrated in the computer device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, electrical signal and software distribution medium, etc.

[0141] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0142] The above-described embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A file uploading method, characterized in that: The following steps are involved: S100, obtaining a file to be uploaded, splitting the file into blocks based on the file and real-time network conditions, and uploading the blocks to a server; S300, after the blocks are uploaded, split the blocks based on the remaining data and the real-time network status, and upload the blocks to the server; S500, after each block upload is completed, repeat S300 until the file has been uploaded to the server.

2. The file uploading method according to claim 1, characterized in that: The file uploading method further comprises the following steps: When uploading the blocks to the server, detecting a real-time upload rate; Determining whether the real-time upload rate is greater than an upload rate threshold; If so, continue uploading the blocks to the server; If not, suspend uploading the blocks to the server and mark the paused position. After determining that the real-time upload rate is greater than the upload rate threshold, split the blocks based on the marked position and upload the blocks to the server.

3. The file uploading method according to claim 2, characterized in that: The pausing uploading the blocks to the server and marking the paused positions comprises the following steps: The uploading of the block to the server is paused, and a mark is made at the position of the nth character before the pause position.

4. The file uploading method according to claim 3, characterized in that: After pausing uploading the blocks to the server, the following steps are included: Based on the paused blocks, obtaining uploaded data and uploading remaining data; The value of n is obtained according to the ratio of the uploaded data to the remaining uploaded data.

5. The file uploading method according to claim 1, characterized in that: The file uploading method further comprises the following steps: When uploading the blocks to the server, extracting upload features; Inputting the uploaded features into a machine learning model and outputting potential risk results; Upload redundant data to the server based on the potential risk results.

6. The file uploading method according to claim 1, characterized in that: The file uploading method further comprises the following steps: After uploading the blocks to the server, the server assembles the blocks and optimizes the blocks according to the file type during the assembly process; The file type includes at least one of document, audio and video.

7. The file uploading method according to claim 1, characterized in that: The file uploading method further comprises the following steps: After splitting the blocks, verifying the integrity of the blocks; If the integrity check of the block fails, re-split the block; After uploading the blocks to the server, the server verifies the integrity of the uploaded blocks; If the integrity check of the uploaded block fails, re-upload the block.

8. A file uploading device, characterized in that: include: A first splitting unit is used to obtain a file to be uploaded, split the file into blocks based on the file and the real-time network status, and upload the blocks to the server; A second splitting unit is used to split the blocks into the blocks based on the remaining data and the real-time network status after the blocks are uploaded, and upload the blocks to the server; After each block upload is completed, the second splitting unit is repeatedly executed until the file is uploaded to the server.

9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the file uploading method according to any one of claims 1 to 7 when executing the computer program.

10. A storage device, characterized in that: The storage device stores a computer program, and the computer program can be executed to implement the steps of the file uploading method according to any one of claims 1 to 7.

Images