Task scheduling methods and systems

Abstract

This invention relates to the field of task scheduling technology and discloses a task scheduling method and system. The method includes: a task scheduler executor sending registration information to a task scheduling center to complete registration; upon completion of registration, the task scheduler executor updating component information and service information with the task scheduling center through a polling service; the task scheduler executor receiving the service information to obtain task scheduling information; the task scheduler executor updating component information to the task scheduling center through an information conversion service; and the task scheduler executor performing task scheduling according to the task scheduling information. The technical solution provided by this invention solves the problems of task scheduling failure, abnormal task execution status, and loss of task execution results due to network anomalies, thereby improving the success rate of task debugging and execution, ensuring the continuity of task scheduling execution, the consistency of task status, and the accuracy of task results.

Description

Technical Field

[0001] This invention relates to the field of task scheduling technology, and in particular to a task scheduling method and system. Background Technology

[0002] Traditional task scheduling is based on customized development and unified management of business requirements and scenario needs of each system. When the network is interrupted, the tasks being executed cannot obtain database connection information, and the task status and task results cannot be stored in time, resulting in task data loss and untimely data processing. Some projects add built-in task scheduling to the project software to meet personalized needs. When the requirements change, the functionality of the project itself needs to be modified. Summary of the Invention

[0003] The main objective of this invention is to provide a task scheduling method and system that aims to solve the task scheduling problem when the network is interrupted.

[0004] To achieve the above objectives, the present invention provides a task scheduling method, the method comprising:

[0005] Step S10: The task scheduler executor completes registration by sending registration information to the task scheduling center through the polling service of the network component and using the session key;

[0006] Step S20: When registration is complete, the task scheduler executor sends network status information and updates the session key through the polling service;

[0007] Step S30: Determine if the network status is interrupted; if the network is not interrupted, proceed to step S40; otherwise, proceed to step S80.

[0008] Step S40: The task scheduling center obtains the latest component information and sends the latest service information to the task scheduling executor using the session key based on the component information;

[0009] Step S50: After receiving the service information and decrypting it using the session key, the task scheduler processes it through the information conversion service to obtain task scheduling information;

[0010] Step S60: The task scheduling executor stores the task scheduling information in a local storage container and updates the component information to the task scheduling center through the information conversion service;

[0011] Step S70: The task scheduler executor obtains task scheduling information from the local storage container and performs task scheduling according to the task scheduling information;

[0012] Step S80: The task scheduler continuously polls the network status through the network component and executes step S70.

[0013] Further, step S10 includes:

[0014] Step S110: The task scheduler executor sends registration information to the task scheduler center through the polling service of the network component according to the preset network service interface information of the task scheduler center; the registration information includes the network service interface information of the task scheduler executor and a session key randomly generated during each polling, and the session key is recorded in the local cache of the task scheduler executor.

[0015] Step S120: The task scheduling center receives and records the registration information, and updates the session key to the local storage of the task scheduling center;

[0016] Step S130: The task scheduling center returns the registration result to the task scheduling executor based on the session key;

[0017] Step S140: The task scheduler executor receives the registration result and completes the registration.

[0018] Further, step S50 includes:

[0019] Step S510: The task scheduler executor receives the service information;

[0020] Step S520: Use the session key to decrypt and identify the management and control instructions in the service information via the information conversion service;

[0021] Step S530: Obtain task scheduling information according to the management and control instructions.

[0022] Further, step S60 includes:

[0023] Step S610: The task scheduler executor updates and stores the identified task scheduling information in the local storage container;

[0024] Step S620: The task scheduler executor sends the updated component information back to the task scheduling center through the information conversion service;

[0025] Step S630: The task scheduling center receives the updated component information to record the changed status and store the data.

[0026] Furthermore, the task scheduling method further includes a task scheduling exception recovery step, which includes:

[0027] The task scheduling exception information is obtained by searching the status information and execution exception points of the abnormal task scheduling within the local storage container.

[0028] The task scheduling exception information is encrypted using a session key and then uploaded to the task scheduling center.

[0029] The task executor executes exception task scheduling through the task engine.

[0030] In addition, the present invention also provides a task scheduling system, which includes a task scheduling center and several task scheduling executors. The several task scheduling executors are all connected to the task scheduling center through a network. The task scheduling center is used to customize and manage scheduling tasks, and the task scheduling executors are used to execute the scheduling tasks.

[0031] Preferably, the task scheduler executor includes a task engine, service components, a network component, an information conversion service, and a local storage container. The task engine is used to execute scheduling tasks. The service components include any one or more of ES search service components, cache service components, file service components, and built-in library service components. The network component is used to send registration information and poll the network status. The information conversion service is used to convert the latest service component information into scheduling tasks after identifying the scheduling tasks. The local storage container is used to store task scheduling center information, registration information, network status information, and scheduling task execution information.

[0032] Preferably, the ES search service component is used to perform distributed full-text search on the data in the local storage container; the caching service component is used to improve data reading efficiency.

[0033] Preferably, the file service component is used to form a status key or file key in an encrypted form to improve the security of the service; the built-in library service component is used to store the information generated during the scheduling and execution of each task.

[0034] Preferably, the information conversion service processes task scheduling information according to priority and timestamp algorithms, and uses the MD5 encryption algorithm to determine the consistency of task scheduling information.

[0035] The task scheduling method and system provided by this invention enable the task scheduler to interconnect with the task scheduling center via a network component. The startup of the task scheduler and the execution of task scheduling are independent of network interruptions with the task scheduling center. During task scheduling, the task scheduling data generated by the executor is stored in a local storage container through the fusion of Elasticsearch search service components, cache service components, file service components, and built-in library service components. A different session key is generated each time the network component polls the service; the previous session key automatically expires. Upon network recovery with the task scheduling center, registration information is proactively sent, containing the session key. When the latest component information is obtained, the information is automatically decrypted using this session key. Decryption failure indicates invalid information; successful decryption uses a priority algorithm, timestamp algorithm, and MD5 algorithm to determine whether to upload or download the task information. Simultaneously, during the network component's polling process, network status monitoring is performed on the task scheduling information, resolving issues such as task scheduling failure, abnormal task execution status, and lost task execution results due to network anomalies. This improves the success rate of task debugging and execution, ensuring the continuity of task scheduling execution, the consistency of task status, and the accuracy of task results. Meanwhile, this invention selects different service components according to various business needs to form different task scheduling modes, thereby reducing the maintenance cost of personalized task scheduling. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating a task scheduling method according to an embodiment of the present invention;

[0037] Figure 2 for Figure 1 A flowchart illustrating step S10 in the process;

[0038] Figure 3 for Figure 1 A flowchart illustrating step S50 in the process;

[0039] Figure 4 for Figure 1 A flowchart illustrating step S60 in the process;

[0040] Figure 5 This is a schematic diagram of the structure of a task scheduling system provided in an embodiment of the present invention;

[0041] Figure 6 This is a schematic diagram of the structure of a task scheduling executor provided in an embodiment of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0043] Please see Figure 1 The above is a flowchart illustrating a task scheduling method according to an embodiment of the present invention. The method includes:

[0044] Step S10: The task scheduler executor completes registration by sending registration information to the task scheduling center through the polling service of the network component and using the session key;

[0045] Step S20: When registration is complete, the task scheduler executor sends network status information and updates the session key through the polling service;

[0046] Step S30: Determine if the network status is interrupted; if the network is not interrupted, proceed to step S40; otherwise, proceed to step S80.

[0047] Step S40: The task scheduling center obtains the latest component information and sends the latest service information to the task scheduling executor using the session key based on the component information;

[0048] Step S50: After receiving the service information and decrypting it using the session key, the task scheduler processes it through the information conversion service to obtain task scheduling information;

[0049] Step S60: The task scheduling executor stores the task scheduling information in a local storage container and updates the component information to the task scheduling center through the information conversion service;

[0050] Step S70: The task scheduler executor obtains task scheduling information from the local storage container and performs task scheduling according to the task scheduling information;

[0051] Step S80: The task scheduler continuously polls the network status through the network component and executes step S70.

[0052] Specifically, the task scheduling center is started and initialized, and a network service interface is provided externally. This means the network service interface of the task scheduling center is exposed and in a listening state, allowing external devices to send information to the task scheduling center via this interface. When the task scheduler executor starts, it performs internal component self-checks and obtains available service component information through the service components, and simultaneously starts the task engine. The task engine is the core of the task scheduler executor's task scheduling processing; specifically, the task engine supports multi-threaded polling scheduling. Simultaneously, the task scheduler executor starts the service components and network components. The service components include any one or more of the following: ES search service component, cache service component, file service component, and built-in library service component.

[0053] The ES search service component is used to perform distributed full-text search on the data in the local storage container, thereby enhancing the response efficiency of the data in the local storage container.

[0054] The caching service component is used to improve data reading efficiency; by configuring the caching service component, multiple caching methods can be provided to load frequently used and infrequently changing data, thereby improving data reading speed.

[0055] The file service component is used to form state keys or file keys through encryption to improve service security; the file service component encrypts critical information and forms state keys and file keys through different encryption methods to improve service security.

[0056] The built-in library service component is used to store information generated during the scheduling and execution of each task.

[0057] After the task scheduler starts, it sends registration information to the task scheduling center through the polling service of the network component and uses the session key to complete the registration.

[0058] Please see Figure 2 Specifically, step S10 includes:

[0059] Step S110: The task scheduler executor sends registration information to the task scheduling center through the polling service of the network component based on the preset network service interface information of the task scheduling center. The registration information includes the network service interface information of the task scheduler executor and a session key randomly generated each time it is polled. The session key is recorded in the local cache of the task scheduler executor. The polling service of the network component sends polling data packets to the task scheduling center at regular intervals and waits for a response from the task scheduling center. If a response is received from the task scheduling center within a preset waiting time, it indicates that the network is normal; otherwise, it indicates that the network is abnormal. For example, a packet is sent every 2 seconds, and if no response data packet is received after 6 seconds, it indicates that the network is abnormal. The network component generates a new session key each time it polls, and the previous session key automatically expires. The session key ensures the security of information exchange in the network communication between the task scheduler executor and the task scheduling center. At the same time, since the task scheduler executor generates a different session key each time it uses the polling service of the network component, security is improved.

[0060] Step S120: The task scheduling center receives and records the registration information, and updates the session key to the local storage of the task scheduling center; since the registration information contains the network service interface information of the task scheduling executor, the task scheduling center stores the network service interface of the task scheduling executor locally, and records the session key in the local cache of the task scheduling executor. The local storage of the task scheduling center may already contain the previous session key, so the newly received session key is updated to the local storage.

[0061] Step S130: The task scheduling center returns the registration result to the task scheduling executor based on the session key; each time the task scheduling center sends data to the task scheduling executor, it uses the latest session key for encryption before transmission, ensuring data security during data transmission.

[0062] Step S140: The task scheduler executor receives the registration result and completes the registration.

[0063] Upon completion of registration, the task scheduler executor sends network status information and updates the session key through the polling service. The network status information also contains the newly generated session key. When the task scheduling center receives the newly generated session key, it updates the locally stored session key. Simultaneously, the network status information also contains component information of the task scheduler, which includes task scheduling information, namely the current task scheduling status, running process, and running result.

[0064] Specifically, the task scheduler executes task scheduling based on the task scheduling information stored in the local storage container. Regardless of whether the network status is normal, the task scheduler executes the task scheduling tasks in the local storage container normally. In one embodiment provided by the present invention, this may include data acquisition task scheduling, data quality scoring task scheduling, etc.

[0065] Specifically, in data acquisition task scheduling, instructions are sent to process, transform, and transmit source data to the target storage. The entire process must be accurate and consistent. After the task scheduling is started, data acquisition instructions are continuously sent. The acquired information (task running status, running process, and running results) is stored locally through the file component. After the network component identifies the network status, it pushes the locally stored information to the task scheduling center through the network service interface exposed by the task scheduling center via the information transformation service, and records the changed component information.

[0066] For example, a data quality scoring task scheduler sends instructions to score specified data according to certain rules. If conditions permit, it can continuously score incoming data. After starting the task scheduler, it continuously sends data quality scoring instructions and stores the acquired information (task running status, process, and results) locally using a file component.

[0067] Meanwhile, the task scheduler executor sends network status information through the polling service to detect whether the network is normal. When the network is normal, the task scheduling center obtains the latest component information and sends the latest service information to the task scheduler executor using the session key based on the component information. After receiving the service information and decrypting it using the session key, the task scheduler executor processes it through the information conversion service to obtain task scheduling information.

[0068] Please see Figure 3 Specifically, step S50 includes:

[0069] Step S510: The task scheduling executor receives the service information; the service information includes task scheduling management and control instructions issued by the task scheduling center.

[0070] Step S520: Use the session key to decrypt and identify the management and control instructions in the service information through the information conversion service; the information conversion service processes the task scheduling information according to the priority and timestamp algorithm, and judges the consistency of the task scheduling information through the MD5 encryption algorithm.

[0071] Specifically, the information conversion service uses a timestamp algorithm to determine whether task information needs to be reordered, an MD5 encryption algorithm to determine whether the task content is complete, and a priority algorithm to convert the upload / download order. The task scheduling center must encrypt the information it sends, along with the corresponding task information and content. The information conversion service prioritizes using the priority algorithm to determine the upload / download order of task information. For example, if the task scheduler has a higher priority, its information is synchronized to the task scheduling center; if the task scheduling center has a higher priority, its information is sent to the task scheduler. When the priority algorithm cannot determine the order, the timestamp algorithm determines the latest information and adjusts the upload / download order accordingly. When both algorithms are ineffective, the task scheduling center uses its MD5 encryption algorithm to determine consistency; if inconsistent, the task scheduling center sends the task information.

[0072] Step S530: Obtain task scheduling information according to the management and control instructions.

[0073] The task scheduler executor stores the task scheduling information in a local storage container and updates the component information to the task scheduling center through the information conversion service.

[0074] Please see Figure 4 Step S60 includes:

[0075] Step S610: The task scheduler executor updates and stores the identified task scheduling information in the local storage container;

[0076] Step S620: The task scheduler executor sends the updated component information back to the task scheduling center through the information conversion service;

[0077] Step S630: The task scheduling center receives the updated component information to record the changed status and store the data.

[0078] Specifically, the updated component information includes task scheduling information executed by the task scheduler, such as task running status, running process, and running results.

[0079] The task scheduler executor obtains task scheduling information from the local storage container and executes task scheduling according to the task scheduling information.

[0080] When the network is interrupted, the task scheduler continues to poll the network status through the network component, while the task scheduler continues to execute the scheduled tasks by reading the task scheduling information in the local storage container.

[0081] The task scheduling method further includes a task scheduling exception recovery step, which includes:

[0082] The task scheduling exception information is obtained by searching the status information and execution exception points of the abnormal task scheduling within the local storage container.

[0083] The task scheduling exception information is encrypted using a session key and then uploaded to the task scheduling center.

[0084] The task executor executes exception task scheduling through the task engine.

[0085] Please see Figure 5 In addition, the present invention also provides a task scheduling system, which includes a task scheduling center 10 and a plurality of task scheduling executors 20. The plurality of task scheduling executors 20 are all connected to the task scheduling center 10 via a network. The task scheduling center 10 is used to customize and manage scheduling tasks, and the task scheduling executors 20 are used to execute the scheduling tasks.

[0086] Please see Figure 6 The task scheduler executor 20 includes a task engine 21, a service component 22, a network component 23, an information conversion service 24, and a local storage container 25. The task engine 21 executes scheduling tasks. The service component 22 includes any one or more of an ES search service component, a cache service component, a file service component, and a built-in library service component. The network component 23 polls the network status. The information conversion service 24 converts the latest service component information into scheduling tasks after identifying them. The local storage container 25 stores task scheduling center information, registration information, network status information, and scheduling task execution information. The ES search service component performs distributed full-text search on the data in the local storage container. The cache service component improves data reading efficiency. The file service component forms a status key or file key using encryption to improve service security. The built-in library service component stores information generated during each task scheduling execution. The information conversion service processes task scheduling information based on priority and timestamp algorithms and uses an MD5 encryption algorithm to determine the consistency of the task scheduling information.

[0087] Specifically, through the connection between the task scheduling center and the task scheduling executor, the task scheduling executor's startup is independent of whether the task scheduling center's network is interrupted. The task scheduling data generated during the task scheduling process, such as the status, process, and execution results of the scheduled tasks, is stored in a local storage container through the integration of the service components of the task scheduling executor, such as the ES search service component, cache service component, file service component, and built-in library service component. When the network component polls and discovers that the task scheduling center is online (i.e., when the network is restored to normal), it actively pushes registration information and obtains the latest component information for updates. During the polling process of the network component, the task scheduling data of the task scheduling is also monitored for network status. If the network is normal, the task scheduling data during the network interruption will be returned.

[0088] Compared with existing technologies, the task scheduling method and system provided by this invention enable the task scheduler executor to interconnect with the task scheduling center through a network component. Furthermore, the startup of the task scheduler executor and the execution of task scheduling are independent of network interruption with the task scheduling center. During task scheduling execution, the task scheduling data generated by the executor is stored in a local storage container through the fusion of functions from ES search service components, cache service components, file service components, and built-in library service components. A different session key is generated each time the network component polls the service; the previous session key automatically expires. Upon detection of network recovery with the task scheduling center, the main... The system automatically sends registration information, which includes a session key. When acquiring the latest component information, this session key is used to automatically decrypt the acquired information. Decryption failure indicates invalid information; successful decryption uses a priority algorithm, timestamp algorithm, and MD5 algorithm to determine the task information for uploading or distribution. Simultaneously, during network component polling, network status monitoring is performed on task scheduling information, resolving issues such as task scheduling failures, abnormal task execution states, and lost task execution results due to network anomalies. This improves the success rate of task debugging and execution, ensuring the continuity of task scheduling execution, the consistency of task states, and the accuracy of task results. Furthermore, this invention selects different service components based on various business needs, forming different task scheduling modes, reducing the maintenance cost of personalized task scheduling.

[0089] The above description is merely an embodiment of the present invention. It should be noted that those skilled in the art can make improvements without departing from the inventive concept of the present invention, but these improvements all fall within the protection scope of the present invention.

Claims

1. A task scheduling method, characterized in that, The task scheduling method includes: Step S10: The task scheduler executor completes registration by sending registration information to the task scheduling center through the polling service of the network component and using the session key; Step S20: When registration is complete, the task scheduler executor sends network status information and updates the session key through the polling service; Step S30: Determine if the network status is interrupted; if the network is not interrupted, proceed to step S40; otherwise, proceed to step S80. Step S40: The task scheduling center obtains the latest component information and sends the latest service information to the task scheduling executor using the session key based on the component information; Step S50: After receiving the service information and decrypting it using the session key, the task scheduler processes it through the information conversion service to obtain task scheduling information; Step S60: The task scheduling executor stores the task scheduling information in a local storage container and updates the component information to the task scheduling center through the information conversion service; Step S70: The task scheduler executor obtains task scheduling information from the local storage container and performs task scheduling according to the task scheduling information; Step S80: The task scheduler continuously polls the network status through the network component and executes step S70.

2. The task scheduling method according to claim 1, characterized in that, Step S10 includes: Step S110: The task scheduler executor sends registration information to the task scheduler center through the polling service of the network component according to the preset network service interface information of the task scheduler center; the registration information includes the network service interface information of the task scheduler executor and a session key randomly generated during each polling, and the session key is recorded in the local cache of the task scheduler executor. Step S120: The task scheduling center receives and records the registration information, and updates the session key to the local storage of the task scheduling center; Step S130: The task scheduling center returns the registration result to the task scheduling executor based on the session key; Step S140: The task scheduler executor receives the registration result and completes the registration.

3. The task scheduling method according to claim 1, characterized in that, Step S50 includes: Step S510: The task scheduler executor receives the service information; Step S520: Use the session key to decrypt and identify the management and control instructions in the service information via the information conversion service; Step S530: Obtain task scheduling information according to the management and control instructions.

4. The task scheduling method according to claim 1, characterized in that, Step S60 includes: Step S610: The task scheduler executor updates and stores the identified task scheduling information in the local storage container; Step S620: The task scheduler executor sends the updated component information back to the task scheduling center through the information conversion service; Step S630: The task scheduling center receives the updated component information to record the changed status and store the data.

5. The task scheduling method according to claim 1, characterized in that, The task scheduling method further includes a task scheduling exception recovery step, which includes: The task scheduling exception information is obtained by searching the status information and execution exception points of the abnormal task scheduling within the local storage container. The task scheduling exception information is encrypted using a session key and then uploaded to the task scheduling center. The task executor executes exception task scheduling through the task engine.

Images