File downloading method and device, equipment and storage medium

Abstract

The invention discloses a file downloading method and device, equipment and a storage medium, and relates to the technical field of file processing, and the method comprises the steps: obtaining a to-be-downloaded fragmentation task from a file fragmentation information table corresponding to a to-be-downloaded file at regular time through a plurality of file downloading instances; locking the fragment task to be downloaded through an optimistic lock; when the locking is successful, determining a target downloading starting position and a target downloading ending position of the to-be-downloaded fragmentation task according to a current fragmentation position field in the file fragmentation information table; and downloading the to-be-downloaded fragmentation task based on the target downloading starting position and the target downloading ending position, and generating a complete downloading file based on the downloaded to-be-downloaded fragmentation task. By applying the technical scheme, the technical problem that file downloading fails due to the fact that file downloading is interrupted when a file downloading instance fails in a file downloading scheme in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of file processing technology, and in particular to a file downloading method, device, equipment and storage medium. Background Art

[0002] With the rapid development of Internet technology, it has provided great convenience to people's lives. People can download various documents, videos, audio and other data they need through the Internet. At present, when downloading files, it is usually downloaded from the file header to the end of the file through a program, or a file is downloaded in segments through an application instance. However, since file downloading in this way depends on a single instance, if the instance fails, the file download will be interrupted, which will lead to the failure of file download. Summary of the invention

[0003] The main purpose of the present application is to provide a file downloading method, apparatus, device and storage medium, aiming to solve the technical problem that the file downloading solution in the prior art will interrupt the file downloading when a failure occurs in the file download instance, resulting in the failure of the file downloading.

[0004] To achieve the above purpose, the present application proposes a file downloading method, the method comprising:

[0005] Obtain the to-be-downloaded fragment task from the file fragment information table corresponding to the to-be-downloaded file periodically through a number of file download instances;

[0006] Lock the shard task to be downloaded through optimistic locking;

[0007] When the locking is successful, the target download starting position and the target download ending position of the to-be-downloaded segment task are determined according to the current segment position field in the file segment information table;

[0008] Downloading the to-be-downloaded segment task based on the target download starting position and the target download ending position;

[0009] Generate a complete download file based on the downloaded fragment tasks.

[0010] In one embodiment, before the step of periodically acquiring the to-be-downloaded fragment tasks from the file fragment information table corresponding to the to-be-downloaded file through a plurality of file download instances, the step further includes:

[0011] Determine the file segmentation method corresponding to the file to be downloaded according to the file segmentation configuration table;

[0012] If the file segmentation method is a size-based segmentation method, determining the file size of the file to be downloaded according to the file information table and the file download toolkit;

[0013] Based on the file segmentation configuration table and the file size, the file to be downloaded is segmented by a preset file segmentation algorithm to obtain a plurality of file segmentation tasks;

[0014] The file segmentation task is stored in a file segmentation information table, and the file execution status corresponding to the file segmentation task in the file segmentation information table is initialized.

[0015] In one embodiment, the step of performing file segmentation on the to-be-downloaded file by a preset file segmentation algorithm based on the file segmentation configuration table and the file size to obtain a plurality of file segmentation tasks includes:

[0016] Determine the target fragment size corresponding to the file to be downloaded according to the file fragment configuration table;

[0017] Determining a total number of shards based on the target shard size and the file size;

[0018] The file to be downloaded is segmented based on the total number of segments using a preset file segmentation algorithm to obtain a plurality of file segmentation tasks.

[0019] In one embodiment, the step of performing file segmentation on the to-be-downloaded file based on the total number of segments by using a preset file segmentation algorithm to obtain a plurality of file segmentation tasks includes:

[0020] Determine the fragment sequence number corresponding to each file fragment in the file to be downloaded based on the total number of fragments by using a preset file fragmentation algorithm;

[0021] Determine the download start position and download end position of each file segment based on the segment sequence number and the target segment size;

[0022] Determine the current fragment position of each file fragment according to the download starting position and the download starting position offset;

[0023] Generate the fragment field information of each file fragment based on the fragment sequence number, the current fragment position, the download start position and the download end position;

[0024] The file to be downloaded is segmented according to the segmentation field information to obtain a plurality of file segmentation tasks.

[0025] In one embodiment, after the step of determining the file segmentation method corresponding to the file to be downloaded according to the file segmentation configuration table, the method further includes:

[0026] If the file segmentation method is segmentation by number of lines, then determining the total number of lines of the file to be downloaded;

[0027] Determine the number of target fragment rows corresponding to the file to be downloaded according to the file fragment configuration table;

[0028] The total number of shards is determined based on the target number of shard lines and the total number of lines in the file.

[0029] In one embodiment, after the step of downloading the to-be-downloaded segment task based on the target download starting position and the target download ending position, the method further includes:

[0030] When the download of the to-be-downloaded segment task is completed, updating the task execution status corresponding to the to-be-downloaded segment task to the execution completion status;

[0031] When the status update is completed, the current fragment position field is updated based on the target fragment size corresponding to the file to be downloaded;

[0032] When the field update is completed, return to the step of determining the target download start position and the target download end position of the to-be-downloaded segment task according to the current segment position field in the file segment information table.

[0033] In one embodiment, the method further comprises:

[0034] Real-time detection of whether there is a target shard task with a task execution status of execution failure in the file to be downloaded;

[0035] If it exists, the task execution state is switched to the executing state, and the execution interruption position and execution end position of the target slice task are determined;

[0036] The target segment task continues to be downloaded based on the execution interruption position and the execution end position.

[0037] In addition, to achieve the above purpose, the present application also proposes a file downloading device, the device comprising:

[0038] The task acquisition module is used to obtain the task of the to-be-downloaded fragment from the file fragment information table corresponding to the to-be-downloaded file at regular intervals through a number of file download instances;

[0039] A task locking module, used to lock the shard task to be downloaded through optimistic locking;

[0040] A download position determination module, used to determine the target download start position and target download end position of the to-be-downloaded segment task according to the current segment position field in the file segment information table when locking is successful;

[0041] A task download module, used for downloading the to-be-downloaded segment task based on the target download starting position and the target download ending position;

[0042] The download file generation module is used to generate a complete download file based on the downloaded fragment tasks to be downloaded.

[0043] In addition, to achieve the above objectives, the present application also proposes a file downloading device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the file downloading method described above.

[0044] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the file downloading method described above are implemented.

[0045] The present application provides a file downloading method, which discloses that a to-be-downloaded fragment task is obtained from a file fragment information table corresponding to a to-be-downloaded file at regular intervals through a plurality of file downloading instances; the to-be-downloaded fragment task is locked through an optimistic lock; when the locking is successful, a target download starting position and a target download ending position of the to-be-downloaded fragment task are determined according to a current fragment position field in the file fragment information table; the to-be-downloaded fragment task is downloaded based on the target download starting position and the target download ending position; a complete download file is generated based on the downloaded to-be-downloaded fragment task; compared with the file downloading scheme in the prior art that relies on a single instance, which will cause the file download to be interrupted when the instance fails, the present invention obtains the to-be-downloaded fragment task through a plurality of file downloading instances, and downloads the to-be-downloaded fragment task based on the target download starting position and the target download ending position of the to-be-downloaded fragment task, and finally generates a complete download file based on the downloaded to-be-downloaded fragment task, thereby solving the technical problem that the file downloading scheme in the prior art interrupts the file download when the file downloading instance fails, causing the file download to fail. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1 A flowchart of the first embodiment of the method for downloading application files is provided;

[0049] Figure 2 This is a diagram showing the file segmentation information table in the file download method of this application;

[0050] Figure 3 A schematic diagram of the process flow generated for the task of downloading the fragments to be downloaded in the method for downloading the application file;

[0051] Figure 4 A flowchart of the second embodiment of the method for downloading application files is provided;

[0052] Figure 5 This is a diagram showing the file segmentation configuration table in the file download method of this application;

[0053] Figure 6 A flowchart of the third embodiment of the method for downloading application files is provided;

[0054] Figure 7 This is a flowchart of file download processing in the file download method of this application;

[0055] Figure 8 This is a schematic diagram of the module structure of the file downloading device according to the embodiment of the present application;

[0056] Fig. 9 This is a schematic diagram of the device structure of the hardware operating environment involved in the file downloading method in the embodiment of the present application.

[0057] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0058] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0059] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0060] The main solution of the embodiment of the present application is: through a number of file download instances, periodically obtain the to-be-downloaded fragment task from the file fragment information table corresponding to the to-be-downloaded file; lock the to-be-downloaded fragment task through optimistic locking; when the locking is successful, determine the target download starting position and the target download ending position of the to-be-downloaded fragment task according to the current fragment position field in the file fragment information table; download the to-be-downloaded fragment task based on the target download starting position and the target download ending position.

[0061] Since the file downloading scheme in the prior art usually downloads a file in segments through an application instance, however, since the file downloading in this way depends on a single instance, if the instance fails, the file downloading will be interrupted, which will lead to the failure of the file downloading.

[0062] The present application provides a solution, which makes it possible to obtain to-be-downloaded fragment tasks through several file download instances, and download the to-be-downloaded fragment tasks based on the target download starting position and the target download ending position of the to-be-downloaded fragment tasks, and finally generate a complete download file based on the downloaded to-be-downloaded fragment tasks, thereby solving the technical problem in the prior art that the file download solution will interrupt the file download when a file download instance fails, resulting in a file download failure.

[0063] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a file download device, etc. The following takes a file download device as an example (hereinafter referred to as a device) to illustrate this embodiment and the following embodiments.

[0064] Based on this, the present application embodiment provides a file download method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the method for downloading documents of this application.

[0065] In this embodiment, the file downloading method includes steps S10 to S50:

[0066] Step S10: obtaining the to-be-downloaded segment tasks from the file segment information table corresponding to the to-be-downloaded file through a plurality of file download instances at regular intervals.

[0067] It should be understood that the above-mentioned file download instance can be a specific operation or code example for obtaining a file (such as a document, picture, video, audio, application installation package, etc.) from the network or other storage media to a local computer or device through some method (such as programming, using software tools or browsers, etc.). This embodiment does not limit the specific content of the file download instance.

[0068] It can be understood that the above-mentioned file segmentation information table can be a table for configuring the segmentation information of the file to be downloaded, wherein the file to be downloaded in this embodiment can be a file in TXT format, wherein each row corresponds to a piece of data in the table and also corresponds to a JAVA object, and each row can be separated by the "|" character for each field. In this scheme, the file data can be downloaded regularly (the content of the downloaded file is different each time) and each row can be parsed into a Java object and then stored in the database.

[0069] In this embodiment, refer to Figure 2 , Figure 2 This is a diagram showing the file sharding information table in the file download method of this application. The fields and descriptions of the file sharding information table T_FILE_SHARDING_INFO are as follows Figure 2 As shown, the fields in the file sharding information table may include: file ID (FILE_INFO_ID), sharding mode SHARDING_MODE, sharding sequence number SHARDING_SEQUENCE, planned target download start position START_POSITION, planned target download end position END_POSITION, target shard size DEST_SIZE, offset from the target download start position START_OFFSET, generally 0, offset from the target end position END_OFFSET, actual shard size ACTUAL_SIZE, current position CURRENT_POSITION, file execution status EXEC_STATUS, where 0 indicates not executed, 1 indicates completed execution, 2 indicates executing, and 9 indicates execution failure. In actual applications, since a file to be downloaded can be divided into multiple fragments, each fragment corresponds to a fragment sequence number SHARDING_SEQUENCE; the planned target start download position START_POSITION can be a field value generated according to the file size and file configuration. In fact, the program starts downloading from START_POSITION+START_OFFSET; the planned target end download position END_POSITION can be a field value generated according to the file size and file configuration. In fact, the program ends downloading from END_POSITION+END_OFFSET; the target fragment size DEST_SIZE can be the fragment size configured in the configuration file; the actual fragment size ACTUAL_SIZE=(END_POSITION+END_OFFSET)-(START_POSITION+START_OFFSET); the current position CURRENT_POSITION can be the position that has been downloaded so far.

[0070] It should be noted that the above-mentioned to-be-downloaded segment task may be a file segment task that has not been executed or failed to be executed in the to-be-downloaded file, wherein the file segment task may be a task for downloading a file segment in the to-be-downloaded file. Figure 3 , Figure 3 This is a flow chart of the generation of the task of downloading the fragments to be downloaded in the method for downloading the application file. Figure 3As shown, to generate a file sharding task, you can first read the file sharding information table to obtain the file information, then connect to the file server to obtain the file size, and read the file sharding configuration table to configure the file sharding information, and finally generate and store the file sharding information.

[0071] Step S20: Lock the to-be-downloaded segment task through optimistic locking.

[0072] It should be noted that the above optimistic lock can be a concurrency control mechanism, which assumes that concurrent operations will not conflict, so it is not locked before the operation. When the operation is completed, check whether there is a conflict, and if there is a conflict, perform corresponding processing (such as retry, throw an exception, etc.). In this embodiment, each sharding task can be processed by optimistic locking, so that multiple instances can be realized to process each file shard concurrently, so that even if a certain instance fails, there are other instances to take over its task. In addition, even if the download process is interrupted, there is a retry mechanism, and the download process is interrupted from each shard. At the same time, the number of application instances can be horizontally expanded to improve the file download processing efficiency, and it will not be too dependent on network speed and stability, so that stable and efficient download of large files can be achieved.

[0073] It should be understood that in this embodiment, the device can scan the file segment information table every 5 seconds through a multi-instance application to extract the segment tasks to be downloaded that have not been executed or failed to execute in the file segment information table, and use an optimistic lock to lock the task so that other application instances cannot obtain and modify the task.

[0074] Step S30: When the locking is successful, the target download start position and the target download end position of the to-be-downloaded segment task are determined according to the current segment position field in the file segment information table.

[0075] It can be understood that the current segment position field may be a field in the file segment information table used to indicate the position to which the file to be downloaded has been currently downloaded, that is, the current position field CURRENT_POSITION in the file segment information table.

[0076] It should be noted that the above-mentioned target download starting position can be the position where the downloading of the to-be-downloaded segment task starts. In this embodiment, the program can start downloading the to-be-downloaded segment task from START_POSITION+START_OFFSET; the above-mentioned target download ending position can be the position where the downloading of the to-be-downloaded segment task ends. In this embodiment, the program can end downloading the to-be-downloaded segment task from END_POSITION+END_OFFSET.

[0077] Step S40: Downloading the to-be-downloaded segment task based on the target download start position and the target download end position.

[0078] In this embodiment, the device can determine the starting position of the file download according to the CURRENT_POSITION field of the file segmentation configuration table, and determine the ending position of the file download according to END_POSITION+END_OFFSET, and finally download the file according to the file segmentation download method, and then the application processes the downloaded data (for example, for TXT files, each downloaded line of data is converted into a Java object and then stored in the database).

[0079] Step S50: Generate a complete download file based on the completed download segment tasks.

[0080] In practical applications, after all the tasks of downloading the fragments to be downloaded in the file to be downloaded are completed, the downloaded data can be merged to finally obtain a downloaded file.

[0081] The present embodiment provides a file downloading method, which discloses that a to-be-downloaded fragment task is obtained from a file fragment information table corresponding to a to-be-downloaded file at regular intervals through a plurality of file downloading instances; the to-be-downloaded fragment task is locked through an optimistic lock; when the locking is successful, a target download starting position and a target download ending position of the to-be-downloaded fragment task are determined according to a current fragment position field in the file fragment information table; the to-be-downloaded fragment task is downloaded based on the target download starting position and the target download ending position; a complete download file is generated based on the downloaded to-be-downloaded fragment task; compared with the file downloading scheme in the prior art that relies on a single instance, which will cause the file download to be interrupted when the instance fails, since the present embodiment obtains the to-be-downloaded fragment task through a plurality of file downloading instances, and downloads the to-be-downloaded fragment task based on the target download starting position and the target download ending position of the to-be-downloaded fragment task, a complete download file is finally generated based on the downloaded to-be-downloaded fragment task, thereby solving the technical problem that the file downloading scheme in the prior art interrupts the file download when the file download instance fails, causing the file download to fail.

[0082] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 , Figure 4 A flowchart diagram of the second embodiment of the method for downloading documents of this application is provided.

[0083] In this embodiment, before step S10, the file downloading method further includes steps S01 to S04:

[0084] Step S01: Determine the file segmentation method corresponding to the file to be downloaded according to the file segmentation configuration table.

[0085] It should be noted that the above-mentioned file fragmentation configuration table may be a table for configuring fragmentation information of a file.

[0086] It can be understood that the above file segmentation method can be the segmentation method used for the file to be downloaded.

[0087] In this embodiment, refer to Figure 5 , Figure 5 This is a diagram showing the file sharding configuration table in the file download method of this application. The specific fields and meanings of the file sharding configuration table T_FILE_SHARDING_CONF are as follows: Figure 5 As shown, the fields in the file sharding configuration table may include: file type field FILE_TYPE, file sharding mode SHARDING_MODE, size of each shard SHARDING_SIZE, relative start position offset START_OFFSET, relative end position offset END_OFFSET. Among them, the file type field FILE_TYPE is a custom value, which is used to indicate the type of file, such as POLICY indicates the policy file, and PRODUCT indicates the insurance type file; the file sharding mode SHARDING_MODE may be the sharding mode used for downloading files, such as: NOT_SHARDING indicates not using sharding download, SHARDING_BY_LENGTH indicates sharding by size, and SHARDING_BY_LINE indicates sharding by number of lines; for the size of each shard SHARDING_SIZE, if sharding is by size, the unit is bit B, such as 104857600 indicates that each shard is 100MB in size; if sharding is by number of lines, the unit is line.

[0088] It should be noted that, in this embodiment, the field END_OFFSET is an extra part downloaded for the segment, which is very important for some scenarios, as it can prevent line breaks and missing numbers. For example, if there is no END_OFFSET for a line in a TXT file for a Java object, and the final position of the segment is not the last character of a line in the file, then the data of the last line obtained by the segment will be incomplete, and there will be problems with the Java object processing program. In this solution, adding END_OFFSET can ensure that the line where the target end position END_POSITION is located can be downloaded completely. As for the incompleteness of the extra line of the next line after this line, it does not matter, because the next segment is downloaded from the END_POSITION position of the current segment, but it is necessary to ensure that the program data processing can be idempotent, that is, the results of multiple processing of the same line of data are the same.

[0089] Step S02: If the file segmentation method is segmentation by size, the file size of the file to be downloaded is determined according to the file information table and the file download toolkit.

[0090] It should be understood that the above-mentioned segmentation method based on size may be a method of segmenting the file to be downloaded based on the size of the file.

[0091] It can be understood that the above-mentioned file information table can be a table for storing information about the file itself, which can record information such as the server SERVERNAME where each file is located, the file type FILE_TYPE, the file path PATHNAME, the file name FILENAME, and the file's unique ID (corresponding to the FILE_IND_ID of the file fragment table), and this embodiment does not impose any restrictions on this; the above-mentioned file download toolkit is the file download SDK (Software Development Kit).

[0092] Step S03: Based on the file segmentation configuration table and the file size, the file to be downloaded is segmented using a preset file segmentation algorithm to obtain a plurality of file segmentation tasks.

[0093] It should be noted that the above-mentioned preset file segmentation algorithm can be an algorithm for segmenting files to be downloaded. The purpose of the preset file segmentation algorithm in this embodiment can be to generate the segmentation sequence number SHARDING_SEQUENCE, the segmentation target start download position START_POSITION, and the segmentation target end download position END_POSITION of the file segmentation information table.

[0094] Specifically, the step S03 includes:

[0095] Step S031: determining the target fragment size corresponding to the file to be downloaded according to the file fragment configuration table.

[0096] It should be noted that the above target fragment size may be the size of each fragment in the file to be downloaded.

[0097] Step S032: Determine the total number of shards based on the target shard size and the file size.

[0098] It is understandable that the total number of fragments can be the number of fragments in the file to be downloaded. In this embodiment, the target fragment size corresponding to the file to be downloaded can be determined according to the target fragment size field DEST_SIZE of the file fragment configuration table, and the file size of the file to be downloaded is divided by the target fragment size to obtain the total number N of fragments of the file to be downloaded.

[0099] Step S033: segmenting the file to be downloaded based on the total number of segments using a preset file segmentation algorithm to obtain a plurality of file segmentation tasks.

[0100] Further, the step S033 includes: determining the fragment sequence number corresponding to each file fragment in the file to be downloaded based on the total number of fragments through a preset file fragmentation algorithm; determining the download start position and download end position of each file fragment based on the fragment sequence number and the target fragment size; determining the current fragment position of each file fragment according to the download start position and the download start position offset; generating fragment field information of each file fragment based on the fragment sequence number, the current fragment position, the download start position and the download end position; performing file segmentation on the file to be downloaded according to the fragment field information to obtain a number of file fragmentation tasks.

[0101] It should be understood that the above-mentioned fragment sequence number is the sequence number corresponding to each fragment in the file to be downloaded; the above-mentioned download start position and download end position are the download start position and download end position of each fragment in the file to be downloaded, respectively. In this embodiment, the device can integrate the fragment sequence number, current fragment position, download start position and download end position corresponding to each fragment in the file to be downloaded, and generate fragment field information corresponding to each fragment, so that the file to be downloaded can be directly segmented based on the fragment field information to obtain a number of file segmentation tasks.

[0102] In practical applications, the preset file fragmentation algorithm can first use the total file size to divide the target per-fragment size field DEST_SIZE of the file fragmentation configuration table to obtain the total number of fragments N of the file. The fragment sequence number calculation formula is (N-1), where N is the Nth fragment, for example, the first fragment is 0, the second fragment is 1, and the Nth fragment is N-1; the fragment target start download position calculation formula is (N-1)*DEST_SIZE, and the fragment target start and end download position calculation formula is (N*DEST_SIZE); in addition, the current fragment position can be START_POSITION+START_OFFSET, at which time each field of each fragment in the file to be downloaded has been determined and completed, and then this information is integrated to obtain the fragment field information corresponding to each fragment, and finally the file to be downloaded is segmented based on the fragment field information to obtain several file fragmentation tasks.

[0103] Step S04: storing the file slicing task into a file slicing information table, and initializing the file execution status corresponding to the file slicing task in the file slicing information table.

[0104] It should be understood that after the file to be downloaded is segmented to obtain several file sharding tasks, the sharding tasks can be stored in the file sharding information T_FILE_SHARDING_INFO table, and the file execution status corresponding to the file sharding task can be initialized to set the initial status of each shard in the file to be downloaded to EXEC_STATUS = 0 (i.e., unexecuted status).

[0105] Furthermore, after step S01, the method further includes: if the file segmentation method is segmentation by line number, determining the total number of file lines of the file to be downloaded; determining the target number of segments corresponding to the file to be downloaded according to the file segmentation configuration table; and determining the total number of segments based on the target number of segments and the total number of file lines.

[0106] It should be understood that the above-mentioned fragmentation method by number of lines can be a method of fragmenting the file to be downloaded by the number of lines of the file; the above-mentioned total number of lines of the file is the total number of lines of the file to be downloaded; and the above-mentioned target number of lines of fragments can be the number of lines of each fragment in the file to be downloaded. In this embodiment, if the fragmentation method of the file to be downloaded is fragmentation by the number of lines, the total number of lines of the file to be downloaded can be used by the preset file fragmentation algorithm and the total number of lines of the file to be downloaded is in the target number of lines per fragment field of the file fragmentation configuration table, so as to obtain the total number of fragments N of the file to be downloaded.

[0107] In the present embodiment, a file segmentation method corresponding to a file to be downloaded is determined according to a file segmentation configuration table; if the file segmentation method is a size-based segmentation method, the file size of the file to be downloaded is determined according to a file information table and a file download toolkit; based on the file segmentation configuration table and the file size, the file to be downloaded is segmented by a preset file segmentation algorithm to obtain a number of file segmentation tasks; the file segmentation tasks are stored in the file segmentation information table, and the file execution status corresponding to the file segmentation tasks in the file segmentation information table is initialized, so that the file to be downloaded can be segmented into multiple segmentation tasks for downloading. Since each segment in the present embodiment is processed by optimistic locking, the file segments can be processed concurrently by multiple instances, thereby improving the efficiency and success rate of file downloading.

[0108] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and will not be described in detail later. Figure 6 , Figure 6 A flowchart diagram of the third embodiment of the method for downloading documents of this application is provided.

[0109] In this embodiment, before step S40, the method further includes steps S60 to S80:

[0110] Step S60: when the download of the to-be-downloaded segment task is completed, the task execution status corresponding to the to-be-downloaded segment task is updated to the execution completion status.

[0111] It should be understood that the above task execution status may be the current execution status of each shard in the task to be downloaded, and the task execution status of the shard in this embodiment may include: execution completed, execution failed, and not executed. In this embodiment, the device can update the task execution status of the shard task to the execution completed state by updating the field EXEC_STATUS in the file shard information table to 1.

[0112] Step S70: When the status update is completed, the current fragment position field is updated based on the target fragment size corresponding to the file to be downloaded.

[0113] It should be noted that after the task execution status of the to-be-downloaded segment task is updated to the execution completion status, the current segment position field CURRENT_POSITION in the file segment information table may be updated, wherein CURRENT_POSITION = original CURRENT_POSITION + the size of the current download processing.

[0114] Step S80: When the field update is completed, return to the step of determining the target download start position and the target download end position of the to-be-downloaded segment task according to the current segment position field in the file segment information table.

[0115] Furthermore, the method also includes: detecting in real time whether there is a target slice task with a task execution status of failed execution in the file to be downloaded; if so, switching the task execution status to an executing status, and determining the execution interruption position and execution end position of the target slice task; and continuing to download the target slice task based on the execution interruption position and the execution end position.

[0116] It should be noted that if a network interruption or execution failure occurs during the download of a file, the download can be resumed from the interrupted position. At the same time, other instances are also executing other fragments without interfering with each other, thus achieving multiple instances downloading the same file in parallel.

[0117] In the specific implementation, refer to Figure 7 , Figure 7 Schematic diagram of the process of file downloading in the file downloading method of this application. Figure 7As shown, when downloading a file, first, a multi-instance application can scan the file sharding information table every 5 seconds, take out the sharding tasks whose execution status is not executed or failed in the file sharding information table, and change its execution status to being executed. At the same time, the sharding task is locked using an optimistic lock, so that other application instances cannot obtain and modify the sharding task. Then, the starting download position and the ending download position of the sharding task can be determined according to the file sharding configuration table, and the sharding task is downloaded based on the starting download position and the ending download position according to the file segment download method. Then, the application processes the downloaded data (for example, when downloading a TXT file, each downloaded line of data is converted into a Java object and then stored in the database). After each download is completed, the current sharding position field in the file sharding information table can be updated, and the task execution status can be updated to completed.

[0118] In this embodiment, it is disclosed that when the download of the to-be-downloaded segment task is completed, the task execution status corresponding to the to-be-downloaded segment task is updated to the execution completion status; when the status update is completed, the current segment position field is updated based on the target segment size corresponding to the to-be-downloaded file; when the field update is completed, the process returns to the step of determining the target download start position and the target download end position of the to-be-downloaded segment task according to the current segment position field in the file segment information table, so that even if a network interruption or execution failure occurs during the file download process, the download can be continued from the interruption position again, thereby reducing the dependence on network speed and stability and improving the success rate of file download.

[0119] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the method for downloading the documents of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0120] This application also provides a file download device, please refer to Figure 8 , the file downloading device comprises:

[0121] The task acquisition module 10 is used to obtain the to-be-downloaded fragment task from the file fragment information table corresponding to the to-be-downloaded file at regular intervals through a number of file download instances;

[0122] The task locking module 20 is used to lock the to-be-downloaded segment task through optimistic locking;

[0123] A download position determination module 30, for determining a target download start position and a target download end position of the to-be-downloaded segment task according to the current segment position field in the file segment information table when locking is successful;

[0124] The task download module 40 is used to download the to-be-downloaded segment task based on the target download start position and the target download end position;

[0125] The download file generation module 50 is used to generate a complete download file based on the completed download segment tasks.

[0126] The file download device provided by the present application adopts the file download method in the above embodiment, which can solve the technical problem that the file download solution in the prior art interrupts the file download when a failure occurs in the file download instance, resulting in the failure of the file download. Compared with the prior art, the beneficial effects of the file download device provided by the present application are the same as the beneficial effects of the file download method provided by the above embodiment, and other technical features in the file download device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0127] The present application provides a file downloading device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the file downloading method in the above-mentioned embodiment 1.

[0128] Reference below Fig. 9 , which shows a schematic diagram of the structure of a file download device suitable for implementing the embodiment of the present application. The file download device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The file downloading device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0129] like Fig. 9As shown, the file download device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the file download device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the file download device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a file download device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0130] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0131] The file download device provided by the present application adopts the file download method in the above embodiment to solve the technical problem of file downloading. Compared with the prior art, the beneficial effects of the file download device provided by the present application are the same as the beneficial effects of the file download method provided by the above embodiment, and other technical features in the file download device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0132] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0133] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0134] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the file downloading method in the above-mentioned embodiment.

[0135] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0136] The computer-readable storage medium may be included in the file downloading device; or may exist independently without being assembled into the file downloading device.

[0137] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a file downloading device, the file downloading device: obtains the to-be-downloaded fragment task from the file fragment information table corresponding to the to-be-downloaded file periodically through a number of file downloading instances; locks the to-be-downloaded fragment task through optimistic locking; when the locking is successful, determines the target download starting position and the target download ending position of the to-be-downloaded fragment task according to the current fragment position field in the file fragment information table; and downloads the to-be-downloaded fragment task based on the target download starting position and the target download ending position.

[0138] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0140] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0141] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned file download method, and can solve the technical problem that the file download solution in the prior art interrupts the file download when a failure occurs in the file download instance, resulting in the failure of the file download. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the file download method provided by the above-mentioned embodiment, and will not be repeated here.

[0142] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A file downloading method, characterized in that: The method includes: Obtain the to-be-downloaded fragment task from the file fragment information table corresponding to the to-be-downloaded file periodically through a number of file download instances; Lock the shard task to be downloaded through optimistic locking; When the locking is successful, the target download starting position and the target download ending position of the to-be-downloaded segment task are determined according to the current segment position field in the file segment information table; Downloading the to-be-downloaded segment task based on the target download starting position and the target download ending position; Generate a complete download file based on the downloaded fragment tasks.

2. The method according to claim 1, characterized in that Before the step of periodically acquiring the to-be-downloaded segment tasks from the file segment information table corresponding to the to-be-downloaded file through a plurality of file download instances, the method further includes: Determine the file segmentation method corresponding to the file to be downloaded according to the file segmentation configuration table; If the file segmentation method is a size-based segmentation method, determining the file size of the file to be downloaded according to the file information table and the file download toolkit; Based on the file segmentation configuration table and the file size, the file to be downloaded is segmented by a preset file segmentation algorithm to obtain a plurality of file segmentation tasks; The file segmentation task is stored in a file segmentation information table, and the file execution status corresponding to the file segmentation task in the file segmentation information table is initialized.

3. The method according to claim 2, characterized in that The step of performing file segmentation on the file to be downloaded by using a preset file segmentation algorithm based on the file segmentation configuration table and the file size to obtain a plurality of file segmentation tasks includes: Determine the target fragment size corresponding to the file to be downloaded according to the file fragment configuration table; Determining a total number of shards based on the target shard size and the file size; The file to be downloaded is segmented based on the total number of segments using a preset file segmentation algorithm to obtain a plurality of file segmentation tasks.

4. The method according to claim 3, characterized in that The step of performing file segmentation on the to-be-downloaded file based on the total number of segments by using a preset file segmentation algorithm to obtain a plurality of file segmentation tasks includes: Determine the fragment sequence number corresponding to each file fragment in the file to be downloaded based on the total number of fragments by using a preset file fragmentation algorithm; Determine the download start position and download end position of each file segment based on the segment sequence number and the target segment size; Determine the current fragment position of each file fragment according to the download starting position and the download starting position offset; Generate the fragment field information of each file fragment based on the fragment sequence number, the current fragment position, the download start position and the download end position; The file to be downloaded is segmented according to the segmentation field information to obtain a plurality of file segmentation tasks.

5. The method according to claim 2, characterized in that After the step of determining the file segmentation method corresponding to the file to be downloaded according to the file segmentation configuration table, the method further includes: If the file segmentation method is segmentation by number of lines, then determining the total number of lines of the file to be downloaded; Determine the number of target fragment rows corresponding to the file to be downloaded according to the file fragment configuration table; The total number of shards is determined based on the target number of shard lines and the total number of lines in the file.

6. The method according to any one of claims 1 to 5, characterized in that After the step of downloading the to-be-downloaded segment task based on the target download starting position and the target download ending position, the method further includes: When the download of the to-be-downloaded segment task is completed, updating the task execution status corresponding to the to-be-downloaded segment task to the execution completion status; When the status update is completed, the current fragment position field is updated based on the target fragment size corresponding to the file to be downloaded; When the field update is completed, return to the step of determining the target download start position and the target download end position of the to-be-downloaded segment task according to the current segment position field in the file segment information table.

7. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Real-time detection of whether there is a target shard task with a task execution status of execution failure in the file to be downloaded; If it exists, the task execution state is switched to the executing state, and the execution interruption position and execution end position of the target slice task are determined; The target segment task continues to be downloaded based on the execution interruption position and the execution end position.

8. A file downloading device, characterized in that: The device comprises: The task acquisition module is used to obtain the task of the to-be-downloaded fragment from the file fragment information table corresponding to the to-be-downloaded file at regular intervals through a number of file download instances; A task locking module, used to lock the shard task to be downloaded through optimistic locking; A download position determination module, used to determine the target download start position and target download end position of the to-be-downloaded segment task according to the current segment position field in the file segment information table when locking is successful; A task download module, used for downloading the to-be-downloaded segment task based on the target download starting position and the target download ending position; The download file generation module is used to generate a complete download file based on the downloaded fragment tasks to be downloaded.

9. A file downloading device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the file downloading method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the file downloading method according to any one of claims 1 to 7 are implemented.

Images