Method and system for breakpoint resuming of multi-thread file and electronic equipment

Abstract

The invention provides a breakpoint resuming method and system for a multi-thread file and electronic equipment. The method comprises the steps that an uploaded file is selected by a front end, the uploaded file is segmented and sliced according to the fixed size and the fixed number, a file unique hash value is generated for a plurality of sliced small files through md5, and an uploading statement is initiated; the back end receives the uploading statement and judges whether the slice is uploaded or not according to the hash value of the uploaded slice, if yes, the slice array is screened, and uploaded data is excluded; if not, the slice array is traversed in sequence, a back-end interface is connected for uploading, and whether all slices are uploaded or not is judged; if yes, a request is initiated to a back end, and the small files of the multiple slices are applied to be verified and combined into one file; and if not, prompting failure, and re-uploading. By means of the method and device, the trouble that in the multi-thread concurrence process, when a client uploads a large file overtime or fails, operation needs to be conducted again can be avoided, and meanwhile the efficiency of obtaining the client file is improved.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a method, system and electronic device for breakpoint-resume transmission of multi-threaded files. Background Art

[0002] With the prevalence of microservices, the demand for file upload tasks is also increasing, especially in banking systems, where file upload tasks are widely used in various scenarios, such as demand document upload, customer ID card or business license upload and archiving, business file interaction between systems, etc. Because banking services are highly concurrent requests, if the interface processing time is long when uploading files, the customer's front-end page will be stuck and spinning, and it is easy to get stuck when many people upload large files at the same time, which is bound to bring certain impacts.

[0003] Due to changes in market demand, many systems are now taking the agile route, and may need to make a delivery every two weeks. Delivery involves system release, and system release involves application restart. When a node of the application is restarting, it may be executing a file upload task, and the upload fails. The customer will re-upload after the system restarts. Or if the network becomes worse during the upload process, the upload fails; this may be the case: part of the fragment is uploaded, and part is not uploaded; because some of the uploaded fragments have been saved, it means that we actually only need to upload the unuploaded fragments. If the entire file is re-uploaded at this time, it will not only cause a bad customer experience, but also cause greater pressure on the server due to repeated uploads. This may also cause repeated processing of business data. Frequent application restarts will greatly increase the probability of this problem, and also waste the system's storage resources. Therefore, before slicing, we need to request to verify whether the file already exists: partially exists, fully exists, or does not exist at all; if it exists partially, find the non-existent fragments to upload.

[0004] With the continuous advancement of loan business, the number of customers has shown an explosive and amazing growth trend. It is becoming increasingly difficult to conduct comprehensive, meticulous and thorough pre-loan investigations and accurate data verification for such a large and increasing number of customers. It has almost become an arduous task that is almost impossible to complete. In such an urgent and severe situation, it is urgent to use advanced and powerful scientific and technological means to achieve efficient acquisition and rigorous review of user data. Summary of the invention

[0005] In view of the technical problems existing in the prior art, the present invention provides a method, system and electronic device for multi-threaded file breakpoint resumption, which are used to solve the technical problem of system multi-threaded large file slice breakpoint resumption.

[0006] According to a first aspect of the present invention, the present invention provides a method for multi-threaded file breakpoint resume transmission, comprising the following steps:

[0007] The front end selects the uploaded file, divides and slices the uploaded file according to a fixed size or a fixed number, generates a unique hash value for the sliced ​​small files using md5, and uses multi-threading to call the back-end interface to upload and initiate an upload statement;

[0008] The backend receives the upload statement and determines whether the slice has been uploaded based on the hash value of the uploaded slice. If so, the slice array is filtered to exclude the uploaded data. If not, the slice array is traversed in turn, connected to the backend interface for upload, and determines whether all slices have been uploaded. If so, a request is sent to the backend to apply for verification of multiple slices of small files and merge them into a large file. If not, it prompts failure and re-uploads.

[0009] Based on the above technical solution, the present invention can also make the following improvements.

[0010] Optionally, the front end selects the uploaded file and divides and slices the uploaded file according to a fixed size or a fixed number, including:

[0011] The front-end initiates an interface declaration to start uploading, divides the video stream into segments according to the preset length, calls the interface to upload the segments, and after all the uploads are completed, tells the back-end to merge the videos and return the merged file parameters.

[0012] Optionally, the upload is initiated by calling the backend interface using multithreading, and the upload statement includes:

[0013] Divide the file into multiple parts and start multiple threads. Each thread is responsible for copying a file. Each thread reads a part of the file. The file contents read by each thread are assembled into a complete file, and an upload statement is initiated. In this process, when the program is interrupted and restarted, there is no need to copy again. Start copying from the completed place, use random access stream to record the offset, and set the corresponding file pointer offset for each read and write operation to complete reading and writing only part of the data.

[0014] Optionally, the backend receives the upload statement and determines whether the slice has been uploaded according to the hash value of the uploaded slice, including:

[0015] The back-end system receives the declaration of the start of file upload, first creates a temporary folder, and then verifies whether there is a slice file in the temporary file for the file transferred from the front end; the temporary folder is named according to the current timestamp + n random numbers.

[0016] Optionally, the verification temporary file includes:

[0017] Determine whether the slice file exists in the temporary folder based on the hash value of the slice file. If it exists, skip the current slice directly; if it does not exist, upload the file stream of the slice to the nas temporary folder.

[0018] Optionally, judging whether a slice has been uploaded according to a hash value of the uploaded slice includes:

[0019] Merge the video stream files in the temporary folder into a complete video and delete the temporary folder; obtain all the slice files in the temporary folder and determine whether the number of slice files matches the number on the front end; if they match, determine whether all slices have been uploaded. If so, read and merge all slice files into a new file according to the order of the slice files to form a complete file. After transcoding the file through the transcoding tool, upload it to the imaging platform for storage and delete the temporary folder and slice files.

[0020] According to a second aspect of the present invention, a system for multi-threaded file breakpoint resume is provided, comprising:

[0021] The front-end processing module is used to select the uploaded file on the front end, segment the uploaded file into slices according to a fixed size or a fixed number, generate a unique hash value for the sliced ​​small files using md5, and use an asynchronous method to call the back-end interface for upload in multiple threads to initiate an upload statement;

[0022] The back-end receiving and judging module is used for the back-end to receive the upload statement and judge whether the slice has been uploaded according to the hash value of the uploaded slice. If so, the slice array is screened to exclude the uploaded data; if not, the slice array is traversed in turn, the back-end interface is connected for uploading, and it is judged whether all slices have been uploaded; if so, a request is sent to the back-end to apply for verification of the small files of multiple slices and merge them into a large file; if not, a failure prompt is given and the file is uploaded again.

[0023] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for resuming breakpoint transmission of multi-threaded files as described above is implemented.

[0024] Technical effects and advantages of the present invention:

[0025] The present invention provides a method, system and electronic device for multi-threaded file breakpoint resume transmission, which slices a large file into slices through the front end, performs md5 on the sliced ​​small files to generate a unique hash value for the file, uses an asynchronous method to call the back-end interface for multi-threaded upload, and the back-end system verifies the small files of the multiple slices and merges them into a large file. The present invention can avoid the trouble of re-operation when a client uploads a large file in the multi-threaded concurrent process due to timeout or failure, and also improves the efficiency of obtaining client files.

[0026] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The present invention is a flowchart of a method for resuming breakpoint transfer of multi-threaded files provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] It should be noted that the present invention is mainly applied to the field of Internet distributed microservice architecture.

[0030] It is understandable that based on the defects in the background technology, the embodiment of the present invention proposes a method for multi-threaded file breakpoint resumption, specifically as follows: Figure 1 As shown, the following steps are included:

[0031] First, the front-end selects the uploaded file, divides and slices it according to a fixed size or a fixed number, and uses MD5 to generate a unique hash value for the sliced ​​small files. It then uses an asynchronous method to call the back-end interface for upload in multiple threads and initiate an upload statement.

[0032] In this embodiment, the front end selects the uploaded file, and divides the uploaded file into slices according to a fixed size or a fixed number, specifically including:

[0033] The front end initiates the BGQSCreateLs interface to declare the start of upload, slices the video stream into 1.5M segments, calls the BGQSQpUpload interface to use the formData mode for segmented upload, and waits for all uploads to be completed (the front end uses promise.all to listen), calls BGQSHbUpload to tell the background to merge the videos and return the merged file parameters (path and id).

[0034] The front end generates a unique hash value for the file through md5, and slices the file according to a fixed size or a fixed number; the asynchronous method is used to call the backend interface for upload through multi-threading. When using multi-threading, each thread reads a part of the file, and then assembles the file content read by each thread into a complete file. In this process, the file can be divided into multiple parts, that is, the file is sliced, and multiple threads are opened, and each thread is responsible for copying a file. When the program is interrupted and opened again, it does not need to be copied again, but starts copying from the completed place. In this process, RandomAccessFile random access stream can be used to record the offset. Each read and write operation can set the corresponding file pointer offset to achieve reading and writing only part of the data. The offset can be long type data. A long type data occupies 8 bytes. A document with a fixed length of 8*number of threads can be set, and every 8 bytes represents a set of offset data. For example, the third group of data is recorded at the offset of (3-1)*8, and the sixth group of data is recorded at the offset of (6-1)*8.

[0035] Secondly, the backend receives the upload statement and determines whether the slice has been uploaded based on the hash value of the uploaded slice. If so, the slice array is filtered to exclude the uploaded data; if not, the slice array is traversed in turn, connected to the backend interface for upload, and it is determined whether all slices have been uploaded; if so, a request is sent to the backend to apply for verification of multiple slices of small files and merge them into a large file; if not, it prompts failure and re-uploads.

[0036] In this implementation step, the backend receives the declaration of the start of file upload, first creates a temporary folder (the folder name is composed of the current timestamp + 6-digit random number), and then verifies whether the slice file exists in the temporary file. Verifying the temporary file includes: judging whether the slice file exists in the temporary folder according to the hash value of the slice file, if it exists, directly skipping the current slice; if the slice does not exist, uploading the file stream of the slice to the nas temporary folder.

[0037] In this embodiment, the backend receives the upload statement, creates a temporary file, receives the uploaded slice from the frontend, and filters the file uniqueness to ensure the uniqueness and integrity of the upload.

[0038] The step of determining whether a slice has been uploaded according to the hash value of the uploaded slice includes:

[0039] The backend scans the temporary file for the slice according to the hash value of the uploaded slice to determine whether the slice has been uploaded. If so, the slice group is filtered to exclude the uploaded data. If not, the slice array is traversed in turn and connected to the backend interface for uploading.

[0040] Merge the video stream files in the temporary folder into a complete video, transcode it into the h264 file format of .mp4, and delete the temporary folder; obtain all the slice files in the temporary folder, and determine whether the number of slice files matches the number of the front end. If they match, determine whether all slices have been uploaded, including: according to the order of the slice files, read and merge all slice files into a new file to form a complete file. After transcoding the file through the transcoding tool, upload it to the imaging platform for storage. Finally, delete the temporary folder and slice files. At this point, a multi-threaded file breakpoint resume process is completed.

[0041] In summary, the embodiment of the present invention provides a method for multi-threaded file breakpoint resumption, which has significant advantages and powerful functions. It can effectively solve a series of related situations such as acquisition timeout or failure that may occur during the client file upload process to the greatest extent, thereby effectively providing solid guarantees and strong support for the smooth and normal operation of the loan business. Through this specific implementation method of multi-threaded breakpoint resumption, the success rate and stability of file data acquisition can be greatly improved, the risk of business obstruction or interruption caused by data acquisition problems can be reduced, and the loan business can be continuously promoted and carried out in a reliable and smooth environment, which plays an important role in the efficient operation of related businesses in the financial field.

[0042] It should be noted that before this technology was realized, the main operating mode of the Zhongshangdai product launched by Wuhan Zhongbang Bank was to promote the development of related businesses through ordinary file uploads. When this technical solution was successfully applied to the product, the positive impact was extremely significant. It greatly reduced the probability of file upload failure and improved the efficiency of server use. At the same time, it also significantly improved the product's credit limit speed, making the entire process faster and smoother. In this case, the product also becomes easier to promote and acquire customers.

[0043] According to a second aspect of the present invention, a system for multi-threaded file breakpoint resume is provided, comprising:

[0044] The front-end processing module is used to select the uploaded file on the front end, segment the uploaded file into slices according to a fixed size or a fixed number, generate a unique hash value for the sliced ​​small files using md5, and use an asynchronous method to call the back-end interface for upload in multiple threads to initiate an upload statement;

[0045] The back-end receiving and judging module is used for the back-end to receive the upload statement and judge whether the slice has been uploaded according to the hash value of the uploaded slice. If so, the slice array is screened to exclude the uploaded data; if not, the slice array is traversed in turn, the back-end interface is connected for uploading, and it is judged whether all slices have been uploaded; if so, a request is sent to the back-end to apply for verification of the small files of multiple slices and merge them into a large file; if not, a failure prompt is given and the file is uploaded again.

[0046] It can be understood that the system for resuming multi-threaded file breakpoints provided by the present invention corresponds to the method for resuming multi-threaded file breakpoints provided by the aforementioned embodiments. The relevant technical features of the system for resuming multi-threaded file breakpoints can be referred to the relevant technical features of the method for resuming multi-threaded file breakpoints, which will not be repeated here.

[0047] In addition, an embodiment of the present invention further provides an electronic device, which includes: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. The processor can call the logic instructions in the memory to execute the steps of the method for multi-threaded file breakpoint resumption.

[0048] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0049] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0050] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0051] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for resuming breakpoint transfer of multi-threaded files, characterized in that: The following steps are involved: The front end selects the uploaded file, divides and slices the uploaded file according to a fixed size or a fixed number, and uses md5 to generate a unique hash value for the multiple small files after slicing. It then uses multi-threading to call the back-end interface to upload and initiate an upload statement. The backend receives the upload statement and determines whether the slice has been uploaded based on the hash value of the uploaded slice; if so, the slice array is filtered to exclude the uploaded data; if not, the slice array is traversed in turn, connected to the backend interface for upload, and determines whether all slices have been uploaded; if so, a request is sent to the backend to apply for verification of multiple slices of small files and merge them into a large file; if not, it prompts failure and re-uploads.

2. A method for multi-threaded file breakpoint resumption according to claim 1, characterized in that: The front end selects the uploaded file and divides the uploaded file into slices according to a fixed size or a fixed number, including: The front-end initiates an interface declaration to start uploading, divides the video stream into segments according to the preset length, calls the interface to upload the segments, and after all the uploads are completed, tells the back-end to merge the videos and return the merged file parameters.

3. A method for multi-threaded file breakpoint resumption according to claim 1, characterized in that: The upload statement initiated by using multi-threaded calls to the backend interface includes: Divide the file into multiple parts and start multiple threads. Each thread is responsible for copying a file. Each thread reads a part of the file. The file contents read by each thread are assembled into a complete file, and an upload statement is initiated. In this process, when the program is interrupted and restarted, there is no need to copy again. Start copying from the completed place, use random access stream to record the offset, and set the corresponding file pointer offset for each read and write operation to complete reading and writing only part of the data.

4. A method for multi-threaded file breakpoint resumption according to claim 1, characterized in that: The backend receives the upload statement and determines whether the slice has been uploaded according to the hash value of the uploaded slice, including: The back-end system receives the declaration of the start of file upload, first creates a temporary folder, and then verifies whether there is a slice file in the temporary file for the file transferred from the front end; the temporary folder is named according to the current timestamp + n random numbers.

5. A method for multi-threaded file breakpoint resume according to claim 4, characterized in that: The verification temporary file includes: Determine whether the slice file exists in the temporary folder based on the hash value of the slice file. If it exists, skip the current slice directly; if it does not exist, upload the file stream of the slice to the nas temporary folder.

6. A method for multi-threaded file breakpoint resume according to claim 1, characterized in that: The step of determining whether a slice has been uploaded according to the hash value of the uploaded slice includes: Merge the video stream files in the temporary folder into a complete video and delete the temporary folder; obtain all the slice files in the temporary folder and determine whether the number of slice files matches the number on the front end; if they match, determine whether all slices have been uploaded. If so, read and merge all slice files into a new file according to the order of the slice files to form a complete file. After transcoding the file through the transcoding tool, upload it to the imaging platform for storage and delete the temporary folder and slice files.

7. A system for multi-threaded file breakpoint resuming, characterized in that: include: The front-end processing module is used to select the uploaded file on the front end, segment the uploaded file into slices according to a fixed size or a fixed number, generate a unique hash value for the sliced ​​small files using md5, and use multi-threading to call the back-end interface to upload and initiate an upload statement; The back-end receiving and judging module is used for the back-end to receive the upload statement and judge whether the slice has been uploaded according to the hash value of the uploaded slice. If so, the slice array is screened to exclude the uploaded data; if not, the slice array is traversed in turn, the back-end interface is connected for uploading, and it is judged whether all slices have been uploaded; If yes, a request is made to the backend to verify the small files of multiple slices and merge them into a large file; if no, a failure message is displayed and the file is uploaded again.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, a method for resuming breakpoint transfer of multi-threaded files as described in any one of claims 1 to 6 is implemented.

Images