Data processing method, electronic device and computer readable storage medium

By obtaining and filtering file lists and task lists, and persisting sub-task information, the problem of file transfer interruption in the existing technology is solved, and incremental collection and multiple transmissions of files are realized to ensure the continuity of transmission.

CN113672307BActive Publication Date: 2024-11-08ZHEJIANG DAHUA TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202110763923.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-06
Publication Date
2024-11-08
Estimated Expiration
2041-07-06

AI Technical Summary

Technical Problem

Existing data processing tools cannot support incremental collection and transmission of files, resulting in the inability to restore the previous progress after file transfer interruption, and the inability to achieve multiple transfers and continuity of files.

Method used

By obtaining the file list and task list, filter out the subtask list to be generated, persist the subtask information, execute the subtask and delete the completed subtasks, ensuring the recovery of task progress and the continuity of file transfer.

Benefits of technology

Incremental collection and multiple transmissions of files are realized, ensuring that the original progress can be restored after the transmission process is interrupted and the continuity of file transfer is maintained.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113672307B_ABST
    Figure CN113672307B_ABST
Patent Text Reader

Abstract

The present application discloses a data processing method, an electronic device, and a computer-readable storage medium, the method comprising: in response to obtaining a first file list from a data source, obtaining a current total task list and a subtask list; determining whether the subtask list includes unfinished subtasks; if included, filtering out a second file list of subtasks to be generated in the first file list based on the subtask information in the total task list; otherwise, using the first file list as the second file list; generating subtasks corresponding to the second file list, persisting the subtasks in the subtask list, and persisting the subtask information corresponding to the subtask in the total task list; executing subtasks based on the subtask list, and deleting the completed subtasks from the subtask list in response to the completion of any subtask. In the above manner, the present application can support multiple file transfers and maintain the continuity of file transfers.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the advent of the information age, higher requirements have been placed on the processing of file data. The process of extracting, converting, and loading files from the data source to the destination has become a new research direction.

[0003] Existing data processing tools usually only support full file collection, but not incremental file collection, that is, they cannot support multiple file collections. When the file collection process is interrupted due to unexpected reasons, the previous progress cannot be restored, resulting in the inability to continuously collect files. In view of this, how to support multiple file transfers and maintain the continuity of file transfers has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a data processing method, an electronic device and a computer-readable storage medium, which can support multiple file transmissions and maintain the continuity of file transmission.

[0005] To solve the above-mentioned technical problems, the first aspect of the present application provides a data processing method, which includes: in response to obtaining a first file list from a data source, obtaining a current total task list and subtask list; judging whether the subtask list includes unfinished subtasks; if included, filtering out a second file list of subtasks to be generated in the first file list based on the subtask information in the total task list; otherwise, using the first file list as the second file list; generating the subtasks corresponding to the second file list, persisting the subtasks in the subtask list, and persisting the subtask information corresponding to the subtask in the total task list; executing the subtasks based on the subtask list, and deleting the completed subtasks from the subtask list in response to the completion of any of the subtasks.

[0006] To solve the above technical problem, the second aspect of the present application provides an electronic device, which includes: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method described in the first aspect.

[0007] In order to solve the above technical problem, the third aspect of the present application provides a computer-readable storage medium on which program data is stored. When the program data is executed by a processor, the method described in the first aspect is implemented.

[0008] The beneficial effect of the present application is as follows: after obtaining the first file list, the present application obtains the current total task list and subtask list, wherein the subtask list includes persistent subtasks, and the total task list includes persistent subtask information. When the subtask list includes unfinished subtasks, the subtask information in the total task list is used to filter out the second file list of subtasks to be generated in the first file list, so that the progress of the task can be restored based on the total task list and the subtask list, and the continuity of the file transfer is maintained. When the subtask list does not include unfinished subtasks, the first file list is used as the second file list to generate the subtasks corresponding to the second file list, and then the subtask list and the total task list are updated. When any subtask is completed, the completed subtask is deleted from the subtask list. Therefore, the entire process can support the incremental collection of files, so that files can be transferred multiple times, and the original progress can be restored after the transmission process is interrupted, so as to maintain the continuity of the file transfer. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0010] Figure 1 It is a flowchart of an implementation method of the data processing method of the present application;

[0011] Figure 2 It is a flowchart of another implementation method of the data processing method of the present application;

[0012] Figure 3 It is a topological diagram of an implementation method of the data processing method of the present application;

[0013] Figure 4 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0014] Figure 5 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0017] See also Figure 1 , Figure 1 It is a flow chart of an implementation method of the data processing method of the present application, the method comprising:

[0018] S101: In response to obtaining a first file list from a data source, obtaining a current total task list and a subtask list.

[0019] Specifically, after obtaining the file from the data source, a first file list is generated in a preset manner, and the current total task list and subtask list are obtained, wherein the subtask list includes persistent subtasks, and the total task list includes persistent subtask information.

[0020] In one application method, in response to obtaining multiple files from a data source, the multiple files are arranged in order from front to back according to the modification time of the files to obtain a first file list, and the current total task list and subtask list are obtained, wherein the subtask list includes the file path of the file storage corresponding to the subtask, and the total task list includes subtask information, wherein the subtask information includes the modification time corresponding to the subtask that has been generated.

[0021] In another application method, in response to obtaining multiple files from a data source, the multiple files are arranged in order from front to back according to the file names to obtain a first file list, wherein the file names are sorted according to the ASCII code priorities corresponding to the characters of the file names, and the current total task list and subtask list are obtained, wherein the subtask list includes the file path of the file storage corresponding to the subtask, and the total task list includes subtask information, wherein the subtask information includes the file name and file path corresponding to the generated subtask.

[0022] S102: Determine whether the subtask list includes unfinished subtasks.

[0023] Specifically, please combine with the subsequent steps. When the subtask is generated, it will be persisted in the subtask list. When the subtask is executed, it will be deleted from the subtask list. Furthermore, when the original task process is interrupted, there are still unfinished subtasks recorded in the subtask list. The above-mentioned incomplete means that it has not been fully executed by the processing unit. Therefore, the incomplete subtasks that have not been fully executed will be saved in the subtask list.

[0024] Furthermore, if the task process has been interrupted before, the unfinished subtasks are included in the subtask list, and the process proceeds to step S103. If all subtasks have been executed, the unfinished subtasks are not included in the subtask list, and the process proceeds to step S104.

[0025] S103: Based on the subtask information in the general task list, a second file list of subtasks to be generated in the first file list is screened out.

[0026] Specifically, when the subtask list includes unfinished subtasks, the files for which subtasks have not yet been generated are filtered out in the first file list according to the subtask information in the total task list to form a second file list. By filtering the first file list, repeated operations on files for which subtasks have already been generated can be avoided, which would cause confusion in file transfer and data processing.

[0027] In one application method, when the subtask list includes unfinished subtasks and the total task list records the modification time of the generated subtasks, based on the first modification time corresponding to the last generated subtask in the total task list, the files in the first file list whose modification time is earlier than the first modification time are eliminated, and then the files for which subtasks have not yet been generated are obtained to form the second file list.

[0028] In another application method, the subtask list includes unfinished subtasks, and the total task list records the file names and file paths of the generated subtasks. Based on the file names and file paths of the generated subtasks in the total task list, the files of the generated subtasks are removed from the first file list, and then the files of the subtasks that have not yet been generated are obtained to form the second file list.

[0029] S104: Use the first file list as the second file list.

[0030] Specifically, when the subtask list does not include unfinished subtasks, it means that all previous tasks have been completed, and there is no need to screen the first file list, and the first file list is directly used as the second file list.

[0031] S105: Generate a subtask corresponding to the second file list, persist the subtask in the subtask list, and persist the subtask information corresponding to the subtask in the general task list.

[0032] Specifically, corresponding subtasks are generated based on the second file list, where the number of subtasks can be one or more. After the subtasks are generated, they are persisted in the subtask list. When any subtask is persisted in the subtask list, the subtask information corresponding to the subtask is persisted in the total task list.

[0033] Among them, persistence refers to the mechanism of converting data between persistent state and transient state. That is to say, transient data (such as data in memory, which cannot be permanently saved) is persisted as persistent data (such as persisted to a database, which can be saved for a long time). Therefore, even if the original task progress is interrupted, the subtask can be stored in the subtask list, and the subtask information is saved in the total task list.

[0034] In one application method, multiple corresponding subtasks are generated based on the second file list, and the multiple subtasks are persisted in the subtask list. After the subtasks are persisted in the subtask list, the modification time and file name of the file corresponding to the last line of instructions in the subtask are persisted as subtask information in the total task list.

[0035] S106: Execute subtasks based on the subtask list, and in response to completion of execution of any subtask, delete the completed subtask from the subtask list.

[0036] Specifically, the subtasks in the subtask list are executed sequentially according to the order in the subtask list, or multiple subtasks are extracted from the subtask list and executed in parallel. When any subtask is completed, the completed subtask is deleted from the subtask list.

[0037] In one application method, multiple execution modules for executing subtasks are pre-set. After obtaining the subtask list, a corresponding number of subtasks are selected from the subtask list according to the number of execution modules and executed in parallel. When any subtask is completed, the completed subtask is deleted from the subtask list, and then other subtasks are selected from the subtask list to continue to be executed by the execution module.

[0038] The above scheme, after obtaining the first file list, obtains the current total task list and subtask list, wherein the subtask list includes persistent subtasks, and the total task list includes persistent subtask information. When the subtask list includes unfinished subtasks, the subtask information in the total task list is used to filter out the second file list of subtasks to be generated in the first file list, so that the progress of the task can be restored based on the total task list and the subtask list to maintain the continuity of the file transfer. When the subtask list does not include unfinished subtasks, the first file list is used as the second file list to generate the subtasks corresponding to the second file list, and then the subtask list and the total task list are updated. When any subtask is completed, the completed subtask is deleted from the subtask list. Therefore, the entire process can support the incremental collection of files, so that files can be transferred multiple times, and the original progress can be restored after the transmission process is interrupted to maintain the continuity of the file transfer.

[0039] See also Figure 2 , Figure 2 It is a flowchart of another embodiment of the data processing method of the present application, the method comprising:

[0040] S201: In response to obtaining a first file list from a data source, obtaining a current total task list and a subtask list.

[0041] Specifically, see Figure 3 , Figure 3 This is a topology diagram of an implementation method of the data processing method of the present application, wherein the data source can be one or more of FTP, HDFS and NFS, the overall task list is the Jobs table, and the subtask list is the Tasks table. NFS (Network File System) is a network file system that is supported by FreeBSD. It allows computers in the network to share resources through the TCP / IP network. FTP (File Transfer Protocol) is a server that provides file storage and access services on the Internet. They provide services in accordance with the FTP protocol. HDFS (Hadoop Distributed File System) is an implementation of Google File System (GFS) and is a core subproject of the Hadoop project.

[0042] Furthermore, after obtaining the files from the data source, the file list obtaining module arranges the files to obtain a first file list, and obtains the latest total task list and subtask list.

[0043] In one application, the step of obtaining a first file list from a data source includes: obtaining a plurality of files from the data source, and sorting the files according to modification time and file name to obtain a first file list.

[0044] Specifically, multiple files are obtained from the above data source, the monitoring directory is scanned, and the files are sorted in descending order according to the modification time and file name of the files in the monitoring directory. Files with the same modification time are sorted according to the size of the file name, so as to obtain a first file list after the files are sorted. After the files in the first file list are uniformly sorted, the files in the first file list can be subsequently screened to reduce the probability of repeated operations on the same file when processing the files.

[0045] S202: Determine whether the subtask list includes unfinished subtasks.

[0046] Specifically, when the subtask list still includes subtasks, the process proceeds to step S203 ; if the subtask list does not include subtasks, the process proceeds to step S204 .

[0047] S203: extract the first subtask information corresponding to the subtask added last in the total task list, and remove the files before the first subtask information in the first file list to obtain a second file list.

[0048] Specifically, the file filtering module can be used to extract the subtask information corresponding to the last subtask added in the total task list, and use it as the first subtask information to filter the first file list, thereby eliminating the files before the first subtask information in the first file list, and then excluding the files for which subtasks have been generated, so as to obtain a second file list consisting of files for which subtasks have not been generated.

[0049] Furthermore, based on the first subtask information corresponding to the last added subtask saved in the total task list, the files in the first file list are divided into files of generated and ungenerated subtasks, so that the task progress can be restored before the task needs to be executed again after an emergency such as a system crash or task suspension, thereby ensuring the integrity of the data and avoiding repeated operations on files, thereby improving data processing efficiency.

[0050] In one application method, the subtask information includes a modification time and a file name, and the first subtask information corresponding to the subtask last added to the total task list is extracted, and the files before the first subtask information in the first file list are removed to obtain a second file list. The steps include: extracting the first subtask information corresponding to the subtask last added to the total task list, wherein the first subtask information includes a first modification time and a first file name; removing the files in the first file list whose modification time is earlier than the first modification time, and whose modification time is equal to the first modification time and whose file name is sorted less than the first file name, to obtain a second file list.

[0051] Specifically, after adding the newly generated subtask to the subtask list, the subtask information corresponding to the most recently added subtask is updated in the total task list. When the first file list needs to be filtered, the first task information corresponding to the most recently added subtask is first obtained from the total task list. The first subtask information includes the first modification time and the first file name. The first modification time is used as the time boundary to eliminate files that are earlier than the first modification time in terms of time sequence. When the modification time is the same as the first modification time, files whose modification time is equal to the first modification time and whose file names are ranked less than the first file name are eliminated, so as to achieve accurate filtering of files for which subtasks have been generated in the first file list, thereby obtaining a second file list for which subtasks have not been generated.

[0052] S204: Use the first file list as the second file list.

[0053] Specifically, when the subtask list does not include unfinished subtasks, the first file list is directly used as the second file list to enable subsequent steps.

[0054] S205: Generate a subtask corresponding to the second file list, persist the subtask in the subtask list, and persist the subtask information corresponding to the subtask in the general task list.

[0055] Specifically, multiple subtasks corresponding to the second file list are generated based on the order of the files in the second file list. The files in the second file list have been sorted in the screening stage, and multiple subtasks corresponding to the second file list are generated according to the order of the files in the second file list, so that the files corresponding to the second file list can be decomposed into multiple subtasks for execution, thereby improving execution efficiency.

[0056] Furthermore, the plurality of subtasks are persisted in the subtask list in sequence; in response to each addition of a subtask to the subtask list, the subtask information corresponding to the newly added subtask is persisted in the total task list.

[0057] In one application mode, please refer to Table 1 and Table 2 as shown below. After a subtask is generated, it is persisted in a subtask list. The subtask list includes the ID of the subtask, wherein the ID of the subtask is generated based on the total task ID in the total task list. The subtask list also records the context information (context) of the instruction corresponding to the subtask, and the number of lines executed by the current task. When a new subtask is added to the subtask list, the total task list adds the file name and modification time corresponding to the first line of instruction in the newly added subtask to the progress (progress) in the total task list.

[0058] Furthermore, when a subtask is generated, it is persisted in the subtask list. The subtask list also stores the file path of the file storage corresponding to the subtask. When a subtask is generated and persisted in the subtask list, as long as the subtask has not been deleted, the corresponding file can be found according to the file name and file path to perform the corresponding operation of the subtask. The first file name and the first modification time corresponding to the last added subtask are continuously updated in the total task list, so that after the task is interrupted, the newly acquired first file list can be screened and filtered according to the stored first file name and the first modification time to ensure the continuity of the task and support incremental collection of files.

[0059] Table 1: Overall task list

[0060]

[0061] Table 2: Subtask List

[0062]

[0063] S206: Read the file corresponding to the subtask, and parse the file to obtain the data corresponding to the file.

[0064] For details, please refer to Figure 3 ,The file reading and parsing module reads the files corresponding to the subtasks according to the order in the subtask list, and extracts the required data from the files corresponding to the subtasks. Among them, multiple subtasks can be executed in parallel, thereby improving the efficiency of subtask execution.

[0065] S207: Transmit the processed data to the corresponding destination, and feed back the progress of the subtask to the progress saver.

[0066] For details, please refer to Figure 3The destination includes at least one of a database, a message system, and a data warehouse, pushes data to a data producer or performs streaming processing and pushes the processed data to the destination, and pushes the progress of the subtask to a progress saver, wherein the progress of the subtask includes the file name corresponding to the file that has been transferred in the subtask, and the number of read lines of the instruction corresponding to the subtask. The progress feedback is separated through the progress saver to reduce the resource consumption of the progress feedback and improve the processing efficiency.

[0067] In one application mode, the progress saver is set with a preset progress feedback cycle. The data is pushed to the destination after being streamed or passed through the data generator. After each progress feedback cycle, the file name of the subtask that has been recently transferred and the number of rows that the subtask has read are fed back to the progress saver, thereby reducing resource consumption caused by progress feedback.

[0068] It should be noted that the step of reading the file corresponding to the subtask includes: obtaining the progress corresponding to the subtask, and determining whether the number of read lines of the instruction corresponding to the subtask is 0; if so, based on the context information in the subtask list, executing the instructions after the number of read lines to read the corresponding file; otherwise, executing the instructions corresponding to the subtask in sequence to read the corresponding file.

[0069] Specifically, please refer to Table 2 again. When starting to execute a subtask based on the subtask list, obtain the progress corresponding to the subtask. When the number of read lines corresponding to the executed instructions in the progress corresponding to the subtask is not 0, combine the context information (context) in the subtask list and the number of read lines to start executing the subtask from the corresponding instructions after the number of read lines. When the number of read lines corresponding to the executed instructions in the progress corresponding to the subtask is 0, start executing from the first line of instructions of the subtask. Therefore, by judging the progress of the subtask, it is possible to avoid repeating the execution of the already executed instructions when executing the subtask, thereby improving the rationality of task execution.

[0070] S208: In response to any subtask being completed, the completed subtask is deleted from the subtask list.

[0071] Specifically, in response to the completion of reading a file in any subtask or the deletion of a file in any subtask, the task completion symbol corresponding to the subtask is fed back to the progress saver. When the subtask is executed or the file corresponding to the subtask is manually deleted, the task completion symbol corresponding to the subtask is fed back to the progress saver, and then the task completion information is fed back through the progress saver.

[0072] Furthermore, in response to obtaining the task completion symbol uploaded by the progress saver, the progress of the task corresponding to the task completion symbol is persisted in the database, and the subtask corresponding to the task completion symbol is deleted from the subtask list.

[0073] In one application, the progress saver is part of the storage space in the memory, which is used to receive the progress of the subtask and feedback the status of the subtask being executed after obtaining the task completion symbol. The progress of the task corresponding to the task completion symbol is stored in a database formed by an external memory, so that the progress of the completed subtask can still be saved after the storage medium loses power. Then, after obtaining the task completion symbol, the subtask corresponding to the task completion symbol is deleted from the subtask list, so that the subtask list only includes unfinished subtasks, so that after the task is interrupted, it can be judged whether the unfinished tasks are included according to the subtask list.

[0074] In this embodiment, whether it is incremental file collection or full file collection, it can be achieved through the process of this embodiment. When the task is executed, the progress of the subtask execution is fed back through the progress saver, and the completed subtask is deleted from the subtask list after the subtask is executed. Then, it can be judged whether there is an unfinished task through the subtask list. At the same time, the obtained first file list is filtered through the subtask information in the total task list, so that only the second file list that has not yet generated the subtask generates the corresponding subtask, and the subtask list and the total task list are updated so that the task can be executed continuously and completely.

[0075] See also Figure 4 , Figure 4 It is a structural diagram of an embodiment of an electronic device of the present application, wherein the electronic device 40 includes a memory 401 and a processor 402 coupled to each other, wherein the memory 401 stores program data (not shown), and the processor 402 calls the program data to implement the data processing method in any of the above embodiments. For descriptions of related contents, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0076] See also Figure 5 , Figure 5 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 50 stores program data 500. When the program data 500 is executed by a processor, the data processing method in any of the above embodiments is implemented. For descriptions of related contents, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0077] It should be noted that the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present implementation scheme.

[0078] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0079] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0080] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: In response to obtaining the first file list from the data source, obtaining a current total task list and a subtask list; wherein the total task list includes subtask information corresponding to the generated subtasks, and the subtask list includes subtasks that have been generated but not yet completed; Based on the subtask information in the total task list, a second file list of subtasks to be generated in the first file list is screened out; Generate the subtask corresponding to the second file list, persist the subtask in the subtask list, and persist the subtask information corresponding to the subtask in the total task list; The subtasks are executed based on the subtask list, and in response to completion of execution of any of the subtasks, the completed subtask is deleted from the subtask list.

2. The data processing method according to claim 1, characterized in that: The step of generating the subtask corresponding to the second file list, persisting the subtask in the subtask list, and persisting the subtask information corresponding to the subtask in the total task list includes: Generate a plurality of subtasks corresponding to the second file list based on the order of the files in the second file list; Persisting the plurality of subtasks in sequence into the subtask list; In response to each addition of a subtask to the subtask list, the subtask information corresponding to the newly added subtask is persisted in the general task list.

3. The data processing method according to claim 1, characterized in that: The step of filtering out the second file list of subtasks to be generated in the first file list based on the subtask information in the total task list includes: The first subtask information corresponding to the subtask last added in the total task list is extracted, and the files before the first subtask information in the first file list are removed to obtain the second file list.

4. The data processing method according to claim 3, characterized in that: The step of obtaining a first file list from a data source comprises: A plurality of files are obtained from the data source, and the files are sorted according to modification time and file name to obtain the first file list.

5. The data processing method according to claim 4, characterized in that: The subtask information includes the modification time and the file name; The step of extracting the first subtask information corresponding to the subtask last added in the total task list, and removing the files before the first subtask information in the first file list to obtain the second file list includes: Extracting first subtask information corresponding to the subtask last added in the total task list, wherein the first subtask information includes a first modification time and a first file name; Eliminate files in the first file list whose modification time is earlier than the first modification time, and whose modification time is equal to the first modification time and whose file name is ranked smaller than the first file name, to obtain the second file list.

6. The data processing method according to claim 1, characterized in that: The step of executing the subtask based on the subtask list includes: Read the file corresponding to the subtask, and parse the file to obtain the data corresponding to the file; Transmitting the processed data to the corresponding destination end, and feeding back the progress of the subtask to the progress saver; The progress of the subtask includes the number of read lines of the instruction corresponding to the subtask.

7. The data processing method according to claim 6, characterized in that: The step of reading the file corresponding to the subtask includes: Obtain the progress corresponding to the subtask, and determine whether the number of read lines of the instruction corresponding to the subtask is 0; If yes, then based on the context information in the subtask list, execute the read line number and after instruction to read the corresponding file; Otherwise, the instructions corresponding to the subtasks are executed in sequence to read the corresponding files.

8. The data processing method according to claim 6, characterized in that: Before the step of deleting the completed subtask from the subtask list in response to completion of execution of any of the subtasks, the method includes: In response to the completion of reading a file in any of the subtasks, or the deletion of a file in any of the subtasks, feeding back a task completion symbol corresponding to the subtask to the progress saver; In response to completion of execution of any of the subtasks, the step of deleting the completed subtask from the subtask list includes: In response to obtaining the task completion symbol uploaded by the progress saver, persisting the progress of the task corresponding to the task completion symbol in a database; The subtask corresponding to the task completion symbol is deleted from the subtask list.

9. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Citation Information

Patent Citations

  • File transmission method and equipment

    CN105208135A

  • File transmission method and device

    CN107689976A

  • A file generation method and apparatus

    CN109298941A

  • Data synchronization method, system and device based on stream processing and medium

    CN112667586A

  • File transmission method, system, computer device and storage medium

    WO2020029388A1