Method and device for asynchronous and concurrent scheduling of data archiving tasks and related products

Abstract

The invention provides a method and device for asynchronous and concurrent scheduling of data archiving tasks and related products, and the method for asynchronous and concurrent scheduling of the data archiving tasks comprises the following steps: S101, obtaining a configuration data archiving rule; s102, acquiring an archiving demand description file, analyzing the archiving demand description file to match the archiving rule, and generating an archiving task according to the archiving rule; s103, executing a scheduling program which is set in the archiving task directory and is written based on a shell program, so as to construct a task flow according to the archiving rule, the archiving task and set concurrent task parameters; and S104, archiving to-be-processed data based on the task process to output an archiving result. According to the method provided by the invention, the processing efficiency and flexibility can be improved, the resource utilization rate is improved, and the problems of long time consumption and serious resource waste in an archiving process are solved.

Description

Technical Field

[0001] The present invention belongs to the field of data archiving, and in particular relates to a method, a device and related products for asynchronous concurrent scheduling of data archiving tasks. Background Art

[0002] In the field of data processing, technology has made significant development and progress in recent years, bringing unprecedented convenience and efficiency to enterprises and organizations in information management. With the rapid iteration and widespread application of information technology, the amount of data has grown at an astonishing rate, promoting the continuous innovation and optimization of data storage, processing and management technologies.

[0003] In terms of data archiving, data archiving is an important means to ensure the safe storage of data. With the continuous development of corporate business, the amount of data has increased dramatically, and how to store this data safely and effectively has become a key issue. Traditional data archiving technology mostly uses single-threaded or sequential processing to perform archiving tasks.

[0004] However, traditional data archiving methods are limited to single-threaded execution mode, and have problems such as low processing efficiency, low resource utilization, and poor flexibility. These methods cannot effectively cope with the archiving needs of massive data, resulting in a long archiving process, serious waste of resources, and difficulty in adapting to changing archiving rules and needs. Summary of the invention

[0005] Based on the above problems, the embodiments of the present application provide a method, device and related products for asynchronous concurrent scheduling of data archiving tasks to solve the problems existing in the above-mentioned prior art.

[0006] The present application discloses a method for asynchronous concurrent scheduling of data archiving tasks, comprising: S101, obtaining configuration data archiving rules; S102, obtaining an archiving requirement description file, and analyzing the archiving requirement description file to match the archiving rules, and generating an archiving task accordingly; S103, executing a scheduling program written based on a shell program and set in the archiving task directory to construct a task flow according to the archiving rules, the archiving tasks and the set concurrent task parameters; S104, archiving the data to be processed based on the task flow to output the archiving results.

[0007] Optionally, the archiving rules are configured according to the following rules, including: obtaining the data archiving range to construct multiple basic tables; obtaining the corresponding batches for data archiving and the actions on the basic tables to establish an operation table; based on the basic tables and the set database table dictionary table attributes, generating the archiving type of each basic table to construct a rule table; based on the operation table and the rule table, generating atomic operation tasks and operation batches to construct a control table; configuring the archiving rules based on the basic table, the operation table, the rule table, and the control table.

[0008] Optionally, the process of obtaining an archiving requirement description file, analyzing the archiving requirement description to match the archiving rules, and generating an archiving task accordingly includes: obtaining an archiving requirement description file, analyzing the archiving requirement description to match the archiving rules; and creating a task batch and a task script based on the matching result to generate an archiving task.

[0009] Optionally, the matching results create task batches, including: grouping and sorting the matching results according to a set behavioral logic to form a task list; determining the number of tasks that each batch can contain based on system resources and time window limitations; and creating task batches based on the task list and the number of tasks and in accordance with the correspondence between tasks and batches.

[0010] Optionally, creating a task script according to the matching result includes: creating a script template based on the task batch, and writing a script generator according to generation logic to create the task script.

[0011] Optionally, archiving the data to be processed based on the task flow to output the archiving result includes: if the data to be processed meets the task node conditions in the task flow, the archiving result is positive and the archiving result is output; if the data to be processed does not meet the task node conditions in the task flow, the archiving result is negative and the task flow is stopped. Optionally, the method also includes: S105, building a monitoring system based on a monitoring tool to monitor the asynchronous concurrent scheduling operation status.

[0012] The present application also provides an asynchronous concurrent scheduling device for data archiving tasks, including: a configuration unit: obtaining configured archiving rules; a requirement unit: obtaining an archiving requirement description file, analyzing the archiving requirement description to match the archiving rules, and generating an archiving task; a framework unit: writing a scheduling program based on a shell program, and setting the scheduling program in the archiving task directory to construct a task flow according to the archiving rules, the archiving tasks, and the set concurrent task parameters; an archiving unit: archiving the data to be processed based on the task flow to output the archiving results; a monitoring unit: building a monitoring system based on a monitoring tool to monitor the running status of the asynchronous concurrent scheduling.

[0013] The present application also provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer executable program, and the processor is used to execute the computer executable program to implement any one of the methods described in the present application.

[0014] The present application also provides a computer storage medium, on which a computer executable program is stored, and when the computer executable program is executed, any method described in the present application is implemented.

[0015] The present application provides a method, device and related products for asynchronous concurrent scheduling of data archiving tasks, wherein a method for asynchronous concurrent scheduling of data archiving tasks includes: S101, obtaining configuration data archiving rules; S102, obtaining an archiving requirement description file, and analyzing the archiving requirement description file to match the archiving rules, and generating an archiving task accordingly; S103, executing a scheduling program written based on a shell program set in the archiving task directory to build a task flow according to the archiving rules, the archiving tasks and the set concurrent task parameters; S104, archiving the data to be processed based on the task flow to output the archiving results. In the present application, through asynchronous concurrent scheduling, the method can process multiple archiving tasks in parallel, thereby significantly improving the efficiency of data archiving. Compared with the traditional single-threaded or sequential processing method, the method can greatly shorten the total time consumption of the archiving process and improve the overall data processing performance. Moreover, the concurrent scheduling mechanism makes fuller use of system resources (such as CPU, memory, I / O, etc.). By reasonably allocating and scheduling tasks, system resources can be load balanced between tasks, reducing resource idleness and waste. In addition, the method supports flexible archiving rule configuration, and can build customized archiving tasks and processes according to different business needs and data characteristics. At the same time, the characteristics of concurrent scheduling also enable the method to better cope with challenges such as surges in data volume and changes in rules, and enhance the adaptability and stability of the system. In summary, the method provided by the present application can improve processing efficiency and flexibility, improve resource utilization, and solve the problem of long time consumption and serious waste of resources in the archiving process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0017] Figure 1 This is a flow chart of a method for asynchronous concurrent scheduling of data archiving tasks in Embodiment 1 of the present application;

[0018] Figure 2 This is a flowchart of concurrent scheduling and archiving tasks in Example 1 of the present application;

[0019] Figure 3 A schematic diagram of the structure of a device for asynchronous concurrent scheduling of data archiving tasks according to an embodiment of the present application;

[0020] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0021] Figure 5 This is a schematic diagram of the hardware structure of the electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The implementation of any technical solution of the embodiments of the present application does not necessarily require achieving all of the above advantages at the same time.

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] First embodiment

[0025] Figure 1 This is a flow chart of a method for asynchronous concurrent scheduling of data archiving tasks according to an embodiment of the present application; Figure 1 As shown, a method for asynchronous concurrent scheduling of data archiving tasks includes: S101, obtaining configuration data archiving rules; S102, obtaining an archiving requirement description file, and analyzing the archiving requirement description file to match the archiving rules, and generating an archiving task accordingly; S103, executing a scheduling program written based on a shell program set in the archiving task directory to build a task flow according to the archiving rules, the archiving tasks and the set concurrent task parameters; S104, archiving the data to be processed based on the task flow to output the archiving results. In this embodiment, through asynchronous concurrent scheduling, the method can process multiple archiving tasks in parallel, thereby significantly improving the efficiency of data archiving. Compared with the traditional single-threaded or sequential processing method, the method can greatly shorten the total time consumption of the archiving process and improve the overall data processing performance. Moreover, the concurrent scheduling mechanism makes system resources (such as CPU, memory, I / O, etc.) more fully utilized. By reasonably allocating and scheduling tasks, system resources can be load balanced between tasks, reducing resource idleness and waste. In addition, this method supports flexible archiving rule configuration, and can build customized archiving tasks and processes according to different business needs and data characteristics. At the same time, the characteristics of concurrent scheduling also enable this method to better cope with challenges such as data volume surge and rule changes, and enhance the adaptability and stability of the system.

[0026] Optionally, the archiving rules are configured according to the following rules, including: obtaining the data archiving range to construct multiple basic tables; obtaining the corresponding batches of data archiving and the actions on the basic tables to establish an operation table; generating the archiving type of each basic table based on the basic table and the set database table dictionary table attributes to construct a rule table; generating atomic operation tasks and operation batches based on the operation table and the rule table to construct a control table; configuring the archiving rules based on the basic table, the operation table, the rule table, and the control table. In this embodiment, by constructing the basic table, the operation table, the rule table, and the control table, the configuration of the archiving rules becomes very flexible. This hierarchical configuration method allows the rules to be dynamically adjusted according to actual needs, increases the scalability of the system, and enables the archiving process to easily adapt to changes in different scenarios and business needs. Moreover, each step (from the acquisition of the data archiving range to the generation of the archiving type, and then to the definition of the atomic operation tasks and batches) has been carefully designed to ensure the accuracy of the archiving process. By defining the role and attributes of each table in detail, archiving errors caused by unclear rules or configuration errors are reduced. In addition, the hierarchical configuration of archiving rules allows for more refined management of data. By classifying, batch processing and defining rules for different data sets, more accurate data archiving can be achieved, which helps to improve the efficiency of data management and utilization. Finally, by managing atomic operation tasks and operation batches through control tables, the system can allocate resources more intelligently, which helps to achieve balanced allocation of resources among multiple archiving tasks, improve the overall resource utilization of the system, and reduce resource waste. At the same time, the setting of concurrent task parameters can further optimize the execution efficiency of tasks and shorten archiving time.

[0027] Optionally, the obtaining of the archiving requirement description file, analyzing the archiving requirement description to match the archiving rules, and generating an archiving task accordingly, includes: obtaining the archiving requirement description file, analyzing the archiving requirement description to match the archiving rules; and creating a task batch and a task script according to the matching result to generate an archiving task. In an embodiment of the present application, by matching the archiving requirement description file and the archiving rules, it is possible to flexibly handle various complex archiving requirements and adapt to different business scenarios and data characteristics. Moreover, the archiving requirements are automatically analyzed, the archiving rules are matched, and the archiving tasks are generated accordingly, which reduces manual intervention and improves work efficiency and accuracy.

[0028] Optionally, the matching result creates a task batch, including: grouping and sorting the matching results according to the set behavior logic to form a task list; determining the number of tasks that can be included in each batch based on system resources and time window restrictions; creating a task batch based on the task list and the number of tasks according to the corresponding relationship between tasks and batches. In this embodiment, by determining the number of tasks that can be included in each batch based on system resources and time window restrictions, the reasonable allocation and efficient use of resources are ensured. This can prevent system resource tension or collapse caused by running too many tasks at the same time. Moreover, processing tasks in batches can reduce the delay caused by processing too much data at a time, and each batch independently processes a part of the data, which can complete the processing of the entire task faster. In addition, this embodiment provides high flexibility and can dynamically adjust the size of the task batch according to the current state and needs of the system. In addition, it also has good scalability and can adapt to data processing requirements of different scales and complexities. Finally, a task list is created by grouping and sorting, and tasks are assigned in batches, which makes the management and monitoring of tasks clearer and controllable. System administrators can more easily track the progress of tasks and intervene in any possible problems or delays in a timely manner.

[0029] For example, a task batch can be created based on the following code:

[0030] nohup sh setjobs.sh A>setjobs.out&

[0031] / unload / script / 20180117

[0032] / unload / script / 20180117

[0033] ------ / 0001_tbl1

[0034] ----------- / jobs.C

[0035] ----------- / jobs.A

[0036] Optionally, the task script is created according to the result of the matching, including: creating a script template based on the task batch, and writing a script generator according to the generation logic to create the task script. In the present embodiment, by writing a script generator, the corresponding task script can be automatically generated according to the task batch, which greatly improves the creation efficiency of the task script and reduces manual intervention and errors. Moreover, the script template and the generation logic can be adjusted and optimized according to different task requirements, so that the generated script can adapt to a variety of archiving task scenarios, improving the flexibility of the system. In addition, the centralized management of the script template and the generation logic makes subsequent maintenance and updating easier. When it is necessary to modify the archiving rules or task flow, only the template or logic needs to be modified and can be quickly applied to all relevant task scripts. Finally, through a unified script template and generation logic, the consistency of all task scripts in format, structure and execution logic can be ensured, which is conducive to subsequent monitoring, management and troubleshooting.

[0037] For example, a task script can be created based on the following code:

[0038] nohup sh unload_all.sh>unload_all.out&

[0039] / unload / script / 20180117

[0040] / unload / script /

[0041] ------ / 20180117

[0042] ----------- / 0001_tbl1

[0043] ------------------ / tbl1_002_OIL_RevokeDBO.sh

[0044] ------------------ / tbl1_002_OIL_RevokeDBO.sql

[0045] ------------------ / tbl1_003_OIL_SetRepFalse.sh

[0046] ------------------ / tbl1_003_OIL_SetRepFalse.sql

[0047] …

[0048] ------------------ / tbl1_021_OIL_SetRepTrue.sh

[0049] ------------------ / tbl1_021_OIL_SetRepTrue.sql

[0050] ----------- / 0002_tbl2

[0051] ------------------ / tbl2_002_OIL_RevokeDBO.sh

[0052] ------------------ / tbl2_002_OIL_RevokeDBO.sql

[0053] …

[0054] ------------------ / tbl2_004_OIL_SelCount.sh

[0055] ------------------ / tbl2_004_OIL_SelCount.sql

[0056] ------------------ / tbl2_006_BAKOIL_BcpOutV.sh

[0057] Optionally, the script generator is written according to the generation logic, including: looping through each task in the task batch to extract task information; filling the task information into the script template to write the script generator. In this embodiment, the script generator ensures that each task script follows the same template and format, which helps to maintain the standardization and consistency of the script and reduce human errors. Moreover, by using templates, the script generator can be easily reused, without the need to write a separate script for each task, saving development time and resources. In addition, when there are new task requirements or rule changes, only the template or generation logic needs to be updated to quickly generate a new script, without the need to modify each script one by one, thereby improving the flexibility and scalability of the system. Finally, the complex task script writing process is simplified to template filling and logic processing, which reduces the skill requirements for developers and reduces the complexity of script writing, so that more people can participate in the generation and maintenance of scripts.

[0058] Optionally, the execution of the scheduling program written based on the shell program set in the archiving task directory to construct the task flow according to the archiving rules, the archiving tasks and the set concurrent task parameters includes: executing the scheduling program written based on the shell program set in the archiving task directory to take each of the archiving rules as an archiving node, the archiving tasks as a logic line, and the concurrent task parameters as the number of concurrent processes to construct the task flow. In this embodiment, by setting the concurrent task parameters, the system resources can be fully utilized to achieve the simultaneous processing of multiple archiving tasks, which greatly improves the efficiency of data archiving. Moreover, the scheduling program based on the shell program is easy to modify and expand, and the concurrent task parameters and task flow can be flexibly adjusted according to different archiving requirements and system environments. In addition, the archiving rules, archiving tasks and concurrent task parameters are encapsulated in the shell program, so that the management of the entire task flow is more centralized and unified, and the maintenance cost is reduced. Finally, the task flow is automatically generated according to the archiving rules and archiving requirements, which reduces manual intervention, reduces the error rate, and improves the accuracy and reliability of data archiving.

[0059] Optionally, the concurrent task parameter is connected in series with at least one of the archiving nodes. In this embodiment, by connecting the archiving nodes in series with the concurrent task parameter, multiple archiving tasks can be executed simultaneously or in parallel, thereby significantly improving the overall processing efficiency of data archiving. This concurrent processing method can better cope with the archiving needs of large-scale data than the traditional single-threaded or sequential processing method.

[0060] Optionally, the processing data is archived based on the task flow to output the archiving result, including: if the processing data meets the task node condition in the task flow, the archiving result is positive, and the archiving result is output; if the processing data does not meet the task node condition in the task flow, the archiving result is negative, and the task flow is stopped from continuing to execute. In this embodiment, through strict task node condition judgment, it is ensured that only data that meets the conditions will be archived, thereby improving the accuracy and reliability of the archived data. Moreover, when the processing data does not meet the task node condition, the task flow is stopped immediately, avoiding unnecessary resource consumption and time waste, and improving the overall processing efficiency. In addition, this embodiment allows the task node condition to be flexibly set according to actual needs to adapt to different archiving scenarios and needs, thereby improving the flexibility and adaptability of the system. Finally, when the archiving result is negative, the task node that does not meet the conditions can be quickly located according to the task log, which is convenient for troubleshooting and problem solving, and improves the maintainability of the system.

[0061] Figure 2 This is a flowchart of concurrent scheduling archiving tasks in Example 1 of this application, such as Figure 2 As shown:

[0062] Scheduling starts: This is the starting point of the entire task execution process, indicating that the system has started and is ready to execute the data archiving task. Read the task sequence: The system reads the pre-defined data archiving rules and executes them one by one in a certain order. Get the number of concurrent tasks N (3): The system determines the number of tasks that can be executed in parallel. In this embodiment, 3 tasks are executed simultaneously as an example. Are there any tasks in the task sequence: The system checks whether there are any unexecuted tasks in the task sequence. If the task sequence is empty, the entire process ends; if the task sequence is not empty, proceed to the next step. Take the next task: Take the next task from the task sequence for execution. The example is three main tasks: "Task 1 (0001_tbl1)", "Task 2 (0002_tbl2)" and "Task 3 (0003_tbl3)". Task execution: For each task, the system executes a series of task node conditions. These task node conditions involve operations and status checks on specific database tables. Task 1 (0001_tbl1): includes at least three task node conditions, each of which may involve checking the activity status of different database tables (such as tbl1_001_db1_act1) and making conditional judgments based on these statuses. Task 2 (0002_tbl2) and Task 3 (0003_tbl3): follow a similar pattern, each task has a series of task node conditions, each of which checks the activity status of different database tables. Condition judgment: After each task node condition is executed, the system will make a conditional judgment to determine whether the task node condition is successfully executed. If successful, it will continue to execute the next task node condition or task; if it fails, error handling or rollback operations may be required. Task node condition execution: Depending on the specific requirements of the task, each task may contain multiple task node conditions, which may be executed sequentially or in parallel. Task end: When all tasks and their task node conditions are successfully executed, the entire process ends.

[0063] Optionally, the archiving of the data to be processed based on the task flow to output the archiving results also includes: recording the archiving results based on the task log. In the present embodiment, by recording the archiving results in a detailed task log, when a problem occurs during the data archiving process, the specific link where the problem occurs can be quickly located, thereby speeding up the problem solving process. In addition, the task log provides a complete record for the archiving process, including key information such as the start time, end time, and operating status, so that the processing process of the data can be traced back, enhancing the transparency and credibility of data management. In addition, by analyzing the task log, the execution efficiency of the archiving task, resource consumption and other information can be understood, thereby optimizing the task scheduling strategy, rationally allocating system resources, and improving overall performance.

[0064] Optionally, the task log includes: a first task log and a second task log; the first task log is based on a file format and is stored in a hard disk; the second task log is based on a structured data format and is stored in a database. In this embodiment, the first task log is stored in a hard disk in a file format, and the second task log is stored in a database in a structured data format. This diversified storage method can effectively prevent data loss and improve data security. Moreover, the log in the form of a file is convenient for direct viewing and editing, and is suitable for rapid location and troubleshooting; while the log in the form of a database supports complex queries and statistics, which is convenient for data analysis and mining. In addition, by dividing the log into different types and storage methods, log management and maintenance can be more convenient. Finally, by adopting this log design method, the storage method and content of the log can be flexibly increased or modified according to actual needs, thereby enhancing the scalability and maintainability of the system. At the same time, it also provides more possibilities for subsequent upgrades and optimizations of the system.

[0065] Optionally, the first task log is named with script task name.batch name.log. In this embodiment, by using "script task name.batch name.log" as the naming rule for log files, the task and batch to which the log belongs can be intuitively understood from the file name, making it easy to quickly locate the required log in a large number of log files.

[0066] Optionally, the second task log records the start time, end time and running status of the task flow execution. In this embodiment, recording the start time, end time and running status of the task flow execution helps to quickly track the occurrence time and specific links of the problem when there is a problem in the task execution, thereby speeding up the problem location and resolution speed.

[0067] Optionally, the method further includes: S105, building a monitoring system based on a monitoring tool to monitor the running state of the asynchronous concurrent scheduling. In this embodiment, the monitoring system can track the running state of the asynchronous concurrent scheduling in real time, and once an abnormality or potential problem is found, an early warning can be issued immediately to help the operation and maintenance personnel to intervene in time to avoid the expansion of the problem. Moreover, through continuous monitoring, potential loopholes or errors in the system can be discovered and repaired in time, reducing business interruptions caused by risks such as system crashes or data loss, thereby improving the overall stability of the system. In addition, the monitoring system can collect and analyze the execution data of the scheduling tasks, such as task execution time, resource consumption, etc., to provide data support for optimizing resource allocation. By reasonably adjusting the concurrent task parameters, resource utilization can be further improved and operating costs can be reduced. In addition, the monitoring system provides a visual operation interface, and the operation and maintenance personnel can intuitively understand the running state and performance indicators of the system. This helps to simplify the operation and maintenance process, improve operation and maintenance efficiency, and reduce operation and maintenance costs. At the same time, the monitoring data can also be used as a basis for system performance evaluation and improvement, providing strong support for the continuous optimization of the system.

[0068] Optionally, build a monitoring system based on tools such as zabbix and grafana.

[0069] The embodiment of the present application also provides an asynchronous concurrent scheduling device for data archiving tasks, including: a configuration unit: obtaining configured archiving rules; a requirement unit: obtaining an archiving requirement description file, analyzing the archiving requirement description to match the archiving rules, and generating an archiving task; a framework unit: writing a scheduling program based on a shell program, and setting the scheduling program in the archiving task directory to construct a task flow according to the archiving rules, the archiving tasks and the set concurrent task parameters; an archiving unit: archiving the data to be processed based on the task flow to output the archiving results; a monitoring unit: building a monitoring system based on a monitoring tool to monitor the running status of the asynchronous concurrent scheduling.

[0070] Figure 3 Schematic diagram of the structure of a device for asynchronous concurrent scheduling of data archiving tasks according to an embodiment of the present application; Figure 3 As shown, it includes: a memory and a processor, the memory stores a computer executable program, and the processor is used to execute the computer executable program to implement any one of the methods in this embodiment.

[0071] A computer storage medium having a computer executable program stored thereon, wherein the computer executable program can implement any of the methods described in the present embodiment when executed.

[0072] Figure 4 Schematic diagram of the electronic device structure of the present application embodiment; Figure 4As shown, it includes: a memory and a processor, the memory stores a computer executable program, and the processor is used to execute the computer executable program to implement the method described in any embodiment of the present application.

[0073] Figure 5 Schematic diagram of the hardware structure of the electronic device of the present application embodiment; Figure 5 As shown, the hardware structure of the electronic device may include: a processor, a communication interface, a computer-readable medium and a communication bus;

[0074] Wherein, the processor, the communication interface, and the computer-readable medium communicate with each other via a communication bus;

[0075] Optionally, the communication interface may be an interface of a communication module, such as an interface of a GSM module;

[0076] The processor may be specifically configured to run an executable program stored in the memory, thereby executing all or part of the processing steps of any of the above method embodiments.

[0077] The processor 401 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0078] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to:

[0079] (1) Mobile communication devices: These devices are characterized by their mobile communication functions and their main purpose is to provide voice and data communications. These terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.

[0080] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, such as iPad.

[0081] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players (such as iPods), handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

[0082] (4) Server: A device that provides computing services. The server consists of a processor 710, a hard disk, memory, a system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0083] (5) Other electronic devices with data interaction functions.

[0084] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0085] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the verification code generation method described herein is implemented. In addition, when a general-purpose computer accesses a code for implementing the verification code generation method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the verification code generation method shown here.

[0086] It should be noted that the same and similar parts between the various embodiments in this specification can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, wherein the modules described as separate components may or may not be physically separated, and the components indicated as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0087] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for asynchronous concurrent scheduling of data archiving tasks, characterized in that: include: S101, obtaining configuration data archiving rules; S102, obtaining an archiving requirement description file, analyzing the archiving requirement description file to match the archiving rule, and generating an archiving task accordingly; S103, executing a scheduling program written based on a shell program and set in the archiving task directory to construct a task flow according to the archiving rule, the archiving task and the set concurrent task parameters; S104: Archiving the data to be processed based on the task flow to output an archiving result.

2. The method for asynchronous concurrent scheduling of data archiving tasks according to claim 1, characterized in that: The archiving rules are configured according to the following rules, including: Get the data archiving range to build multiple basic tables; Obtaining the batch corresponding to the data archive and the actions on the basic table to establish an operation table; Based on the basic table and the set database table dictionary table attributes, generate the filing type of each basic table to construct a rule table; Based on the operation table and the rule table, atomic operation tasks and operation batches are generated to construct a control table; The archiving rules are configured based on the basic table, the operation table, the rule table, and the control table.

3. The method for asynchronous concurrent scheduling of data archiving tasks according to claim 1, characterized in that: The step of obtaining the archiving requirement description file, analyzing the archiving requirement description to match the archiving rule, and generating an archiving task accordingly includes: Obtaining an archiving requirement description file, and analyzing the archiving requirement description to match the archiving rule; According to the matching result, a task batch and a task script are created to generate an archiving task.

4. The method for asynchronous concurrent scheduling of data archiving tasks according to claim 3, characterized in that: The matching result creates a task batch, including: Grouping and sorting the matching results according to the set behavior logic to form a task list; Determine the number of tasks that can be included in each batch based on system resources and time window constraints; Based on the task list and the task quantity, a task batch is created according to the correspondence between the tasks and the batches.

5. The method for asynchronous concurrent scheduling of data archiving tasks according to claim 4, characterized in that: The step of creating a task script according to the matching result includes: A script template is created based on the task batch, and a script generator is written according to the generation logic to create a task script.

6. The method for asynchronous concurrent scheduling of data archiving tasks according to claim 1, characterized in that: Archiving the data to be processed based on the task flow to output the archiving result includes: If the data to be processed meets the task node condition in the task flow, the archiving result is positive, and the archiving result is output; If the data to be processed does not meet the task node condition in the task flow, the archiving result is negative, and the task flow is stopped from being executed.

7. The method for asynchronous concurrent scheduling of data archiving tasks according to claim 1, characterized in that: The method further comprises: S105. Building a monitoring system based on monitoring tools to monitor the running status of the asynchronous concurrent scheduling.

8. A data archiving task asynchronous concurrent scheduling device, characterized in that: include: Configuration unit: Get the configured archiving rules; Requirement unit: obtains an archiving requirement description file, analyzes the archiving requirement description to match the archiving rule, and generates an archiving task; Framework unit: write a scheduling program based on the shell program, and set the scheduling program in the archiving task directory to construct a task flow according to the archiving rule, the archiving task and the set concurrent task parameters; Archiving unit: archiving the data to be processed based on the task flow to output the archiving result; Monitoring unit: Building a monitoring system based on monitoring tools to monitor the running status of the asynchronous concurrent scheduling.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer executable program, and the processor is used to execute the computer executable program to implement the method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer executable program, and the computer executable program implements the method according to any one of claims 1 to 7 when executed.

Images