Distributed anti-concurrency task execution system and method

Abstract

The present invention discloses a distributed anti-concurrency task execution system and method, which belongs to the field of task scheduling. The distributed anti-concurrency task execution system of the present invention includes a task scheduling unit, a task execution unit and a real-time monitoring unit. The present invention solves the problem that the existing single program cannot be upgraded and maintained locally. The present invention is based on the distributed microservice and can realize the timed triggering of tasks. At the same time, relying on the flexible support of cron for the calendar and cooperating with redis, it realizes the characteristics of 7x24 hours business separation and dynamic update of local distribution. Adding, modifying or removing any logical task does not affect other logical tasks, so that the execution program can achieve high-availability deployment, and by parsing the cron expression, calculating the next trigger time, and then sleeping with this time interval as a parameter, the execution cycle is flexible and configurable, and the operation can be monitored.

Description

Technical Field

[0001] The present invention relates to the technical field of task scheduling, and in particular to a distributed anti-concurrency task execution system and method. Background Art

[0002] With the expansion of business systems, simple linear operations can no longer achieve long-term processing of key business data. For example, when each container gate in and out of the terminal has to generate, sign and send dozens of different types of messages, and at the same time query more than ten peripheral systems to supplement relevant data, the program execution cycle (including storage IO cycle, network request cycle, processing operation cycle, etc.) will exceed the actual time of the container leaving the door in the real world. At this time, the more traditional way is to establish a single program, through its own sleep interrupt to wake up the thread every few seconds to perform a check to know whether there is a container leaving the door, and then perform related message generation operations until there are no containers waiting to generate messages and then go into sleep again, waiting to be woken up after the next interval; or through the operating system's crontab or scheduled tasks, set a fixed cycle to wake up this single program, but at this time the single program must be locked, otherwise it is very easy for the previous program to be woken up again before the next program is executed, resulting in repeated data execution. Moreover, usually, there are hundreds of logical tasks, and it is not possible to make a separate application program for each logical task and execute it independently. Therefore, a single program may contain dozens or hundreds of logical tasks waiting to be executed.

[0003] Although the above traditional methods have met the needs and are widely used in various industries, they still have the following problems:

[0004] 1. It is difficult for a single program to achieve high availability. Process crashes or server crashes will lead to business interruptions, message queue explosions, or loss of key data.

[0005] 2. The timing tasks implemented by the operating system are very complicated in the implementation of their locks, and in abnormal circumstances, the deadlock problem cannot be solved;

[0006] 3. The periodic task system implemented by its own thread scheduling concentrates all different types of businesses into a single program, which is highly coupled and contradicts the concept of microservices.

[0007] 4. The server needs to be regularly patched and restarted. The increase in new task requirements requires the redeployment of task programs, which makes it impossible for the task program itself to meet the 7x24 specification requirements. For automatic task programs, they may be processing critical tasks at all times. Any interruption or shutdown may have an impact on business data that is difficult to repair;

[0008] 5. A single task program cannot be manually intervened to directly execute a task at a certain moment as needed, because it contains all the tasks. Once it is manually run again, all tasks will be executed.

[0009] Therefore, the existing needs are not met, and we propose a distributed anti-concurrency task execution system and method. Summary of the invention

[0010] The purpose of the present invention is to provide a distributed anti-concurrency task execution system and method, which implements a set of timed trigger task system through a distributed microservice-based system. Relying on cron's flexible support for the calendar and cooperating with redis, it realizes the characteristics of 7x24 hours, business separation, and dynamic update of local distribution, solving the problems raised in the above background technology.

[0011] To achieve the above object, the present invention provides the following technical solution: a distributed anti-concurrency task execution system, comprising:

[0012] The task scheduling unit is used to receive and schedule tasks from the task manager, identify user login information according to the login request, and grant permissions according to the user login information. At the same time, it creates an executor and configures the scheduling task according to the user login information. The scheduling task is configured to select an executor that meets the configuration conditions;

[0013] The task execution unit is used to distribute the configured scheduling tasks to the executors that meet the configuration conditions. The executors execute the distributed scheduling tasks. The task execution unit can realize the timed triggering tasks based on the distribution of microservices. The executors select the scheduling tasks in the task list. The executors execute the selected scheduling tasks regularly and have task execution status feedback;

[0014] The real-time monitoring unit is used to monitor the task execution system status and task execution status in real time, feedback the task execution system status and task execution status to the user, and record and analyze system logs to provide task execution system failure warnings.

[0015] Furthermore, the task manager is connected to the task scheduling unit. The task manager is used to add, modify and delete tasks. The user adds tasks in the task manager. The scheduler is used to schedule tasks and assign tasks to reasonable executors. The scheduler is used to determine the execution order and time of tasks, and the executor is used to execute the assigned scheduled tasks.

[0016] Among them, the task manager adds a new task, and the new task is written into the database through the API server. The API server uses TaskJob to query all tasks, and queries all tasks whose update time is greater than the last task on the timeline. TaskJob searches for the newly obtained task in the existing tasks, removes the old tasks and adds the new tasks, and then parses the cron expression, obtains the next trigger time of the new task and calculates the time interval, so that the thread enters the sleep state with this time interval as a parameter.

[0017] Furthermore, the task scheduling unit includes:

[0018] The authority authentication module is used to identify the user login information according to the login request and grant permissions according to the user login information. The user can modify and delete the tasks of the task manager according to the granted permissions, or add new tasks in the task manager, and then configure the scheduling tasks;

[0019] The task scheduling module is used to create an executor according to the user login information, configure one or more executors matching the user login information, and schedule the task configuration to select an executor that meets the configuration conditions.

[0020] Furthermore, the task scheduling unit further includes:

[0021] A modification count real-time monitoring module is used to monitor in real time the number of modifications made by users to the tasks of the task manager according to the granted permissions;

[0022] A modification information content extraction module is used to extract the modification information content corresponding to each modification made by the user;

[0023] The task modification parameter acquisition module is used to acquire the task modification parameter according to the number of modifications made by the user to the task manager according to the granted authority and the modification information content, wherein the task modification parameter is acquired by the following formula:

[0024]

[0025] Where G represents the task modification parameter; n represents the number of times the user modifies the task of the task manager according to the granted permissions; g i The weight coefficient of the information content modified for the i-th time; C i Indicates the number of characters modified during the i-th modification of the information content; C 0i Indicates the total number of characters corresponding to the total content of the task in which the information content is modified for the i-th time; x represents the adjustment coefficient, and the adjustment coefficient is obtained by the following formula;

[0026]

[0027] Wherein, m represents the total number of unit time contained in the duration experienced after the current user is granted permission, and the unit time is 24 hours; f i represents the frequency of modification occurring in the i-th unit time;

[0028] The modification abnormality alarm module is used to issue a task modification abnormality alarm when the task modification parameter exceeds a preset modification parameter threshold.

[0029] Furthermore, the task scheduling unit further includes:

[0030] A real-time monitoring module for the number of deletion frequencies is used to monitor in real time the frequency of information deletion performed by users on the tasks of the task manager according to the permissions granted;

[0031] A deletion frequency comparison module, used to compare the deletion frequency with a preset deletion frequency threshold;

[0032] A deleted information content extraction module, used for extracting deleted information data corresponding to each information deletion when the deletion frequency exceeds a preset deletion frequency threshold;

[0033] The task deletion parameter acquisition module is used to acquire the task deletion parameter according to the deletion frequency and deletion information content of the task manager performed by the user according to the granted authority, wherein the task deletion parameter is acquired by the following formula:

[0034]

[0035] Among them, S represents the task deletion parameter; f si represents the frequency of information deletion in the i-th unit time; K represents the compensation coefficient; f sp Indicates the overall deletion frequency of users; f p Indicates the overall modification frequency of users;

[0036] The deletion abnormality alarm module is used to issue a task deletion abnormality alarm when the task deletion parameter exceeds a preset deletion parameter threshold.

[0037] Furthermore, the task execution unit includes:

[0038] The scheduling execution module implements timed triggering tasks based on the distribution of microservices. The scheduler first distributes the configured scheduling tasks to the executors that meet the configuration conditions, and then the scheduler executors execute the distributed scheduling tasks;

[0039] The execution feedback module is used to obtain the task execution results and feed back the task execution results to the user so that the user can understand the task execution status.

[0040] Furthermore, the task execution result is the execution status of each scheduled task, the execution status includes the task completion date, and whether the scheduled task is overdue is determined based on the execution status. If the scheduled task is not overdue, a message of successfully triggering the task execution is fed back to the user.

[0041] Furthermore, the task execution unit specifically performs the following operations:

[0042] The execution feedback module implements timed triggering tasks based on the distributed implementation of microservices. Tasks are scheduled through the scheduler. After the task is triggered, the scheduled task is assigned to a reasonable executor.

[0043] The executor executes the assigned tasks assigned to the executor according to various scheduling algorithms based on the type, waiting time and priority information of the task;

[0044] The task execution result is fed back through the execution feedback module, and the message that successfully triggered the task execution is fed back.

[0045] Furthermore, the real-time monitoring unit includes:

[0046] A monitoring module is used to monitor the status of the task execution system in real time, and to monitor the task execution status in real time according to the task execution status feedback of the task execution unit;

[0047] The analysis module is used to record and analyze system logs. Based on the monitored task execution status, users can check the cause of the error and provide early warning for task execution system failures.

[0048] The implementation method of the distributed anti-concurrency task execution system includes the following steps:

[0049] S1: The task scheduling unit receives and schedules tasks. The task execution unit triggers tasks based on the distributed implementation of microservices and feeds back messages that successfully trigger task execution.

[0050] S2: The user adds a new task in the task manager, and the update time is the current latest time. The new task is written to the database through the API server. TaskJob queries all tasks through the API server every minute for tasks whose update time is greater than the last task on the timeline. TaskJob then searches for the newly obtained task in the existing tasks, removes the old task and adds the new task.

[0051] S3: When the sleep time of the new task ends, it is automatically awakened and calls the corresponding http request configured in the task, carrying Authorization as the identity credential. After the call is completed, the last execution status and time information of the task are modified through the API server, and then the cron expression is parsed and the next trigger time is calculated, and the task is put into sleep with this time interval as the parameter;

[0052] S4: Monitor the task execution system status and task execution status in real time through the real-time monitoring unit. At the same time, record and analyze the system logs and check the causes of errors to provide early warning for task execution system failures.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] The present invention receives and schedules tasks through a task scheduling unit, and the task execution unit can realize timed triggering of tasks based on the distribution of microservices. At the same time, relying on the flexible support of cron for the calendar and cooperating with redis, the characteristics of 7x24 hours business separation and dynamic update of local distribution are realized, and adding, modifying or removing any logical task will not affect other logical tasks, so that the execution program can realize high-availability deployment and will not be unable to execute due to the restart or downtime of a certain machine. By parsing the cron expression, the next trigger time is calculated, and then the time interval is used as a parameter for sleep, so that the execution cycle is flexible and configurable, and the task execution unit can feedback the message of successful triggering of task execution, and the real-time monitoring unit can monitor the system status, task execution status and record and analyze system logs in real time, so that the operation can be monitored. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of the microservice concept of the distributed anti-concurrency task execution system of the present invention;

[0056] Figure 2 A schematic diagram of concurrent execution of a distributed anti-concurrency task execution system of the present invention;

[0057] Figure 3 It is a flow chart of the distributed anti-concurrency task execution system of the present invention requesting two loads on the same machine;

[0058] Figure 4 A schematic diagram of a flow chart of a distributed anti-concurrency task execution system of the present invention requesting loads twice on different machines;

[0059] Figure 5 It is a schematic diagram of the overall structure of the distributed anti-concurrency task execution system of the present invention;

[0060] Figure 6It is a structural diagram of the task scheduling unit of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] To solve the traditional timed triggering problem, either you rely on the scheduled tasks of the operating system, and each application only performs one business, or you schedule multiple businesses yourself and couple different business logics into a single program. However, this single program leads to the technical problem that it is impossible to upgrade and maintain the local deployment. Please refer to Figure 1-Figure 6 , this embodiment provides the following technical solutions:

[0063] Distributed anti-concurrency task execution system, including:

[0064] The task scheduling unit is used to receive and schedule tasks from the task manager, identify user login information according to the login request, and grant permissions according to the user login information. At the same time, it creates an executor and configures the scheduling task according to the user login information. The scheduling task is configured to select an executor that meets the configuration conditions;

[0065] The task execution unit is used to distribute the configured scheduling tasks to the executors that meet the configuration conditions. The executors execute the distributed scheduling tasks. The task execution unit can realize the timed triggering tasks based on the distribution of microservices. The executors select the scheduling tasks in the task list. The executors execute the selected scheduling tasks regularly and have task execution status feedback;

[0066] The real-time monitoring unit is used to monitor the task execution system status and task execution status in real time, feedback the task execution system status and task execution status to the user, and record and analyze system logs to provide task execution system failure warnings.

[0067] Task scheduling unit, including:

[0068] The authority authentication module is used to identify the user login information according to the login request and grant permissions according to the user login information. The user can modify and delete the tasks of the task manager according to the granted permissions, or add new tasks in the task manager, and then configure the scheduling tasks;

[0069] The task scheduling module is used to create an executor according to the user login information, configure one or more executors matching the user login information, and schedule the task configuration to select an executor that meets the configuration conditions.

[0070] The technical effect of the above content is: tasks are received and scheduled through the task scheduling unit, and the authority authentication module can identify the user login information according to the login request, and then grant the user authority according to the user login information, so that the user can add new tasks in the task manager, as well as modify and delete the tasks of the task manager, thereby realizing the configuration of scheduled tasks. Then, the task scheduling module can create an executor according to the user login information, and the number of executors is created according to the user login information. One or more executors can be configured, and the scheduling task is configured to select an executor that meets the configuration conditions, so that the scheduled tasks can be distributed to reasonable executors.

[0071] Specifically, the task scheduling unit further includes:

[0072] A modification count real-time monitoring module is used to monitor in real time the number of modifications made by users to the tasks of the task manager according to the granted permissions;

[0073] A modification information content extraction module is used to extract the modification information content corresponding to each modification made by the user;

[0074] The task modification parameter acquisition module is used to acquire the task modification parameter according to the number of modifications made by the user to the task manager according to the granted authority and the modification information content, wherein the task modification parameter is acquired by the following formula:

[0075]

[0076] Where G represents the task modification parameter; n represents the number of times the user modifies the task of the task manager according to the granted permissions; g i The weight coefficient of the information content modified for the i-th time; C i Indicates the number of characters modified during the i-th modification of the information content; C 0i Indicates the total number of characters corresponding to the total content of the task in which the information content is modified for the i-th time; x represents the adjustment coefficient, and the adjustment coefficient is obtained by the following formula;

[0077]

[0078] Wherein, m represents the total number of unit time contained in the duration experienced after the current user is granted permission, and the unit time is 24 hours; f i represents the frequency of modification occurring in the i-th unit time;

[0079] The modification abnormality alarm module is used to issue a task modification abnormality alarm when the task modification parameter exceeds a preset modification parameter threshold.

[0080] The technical effect of the above technical solution is: through the modification number real-time monitoring module, the system can track the number of times the user has modified the task of the task manager in real time. This provides a data basis for user behavior analysis and helps to discover abnormal modification behavior. The modification information content extraction module can extract the specific content of each modification, which helps to understand the motivation and purpose of the user to modify the task. At the same time, combined with the task modification parameter acquisition module, the degree of modification of the task by the user can be quantified, making the evaluation of the modification behavior more objective and accurate. In the calculation of the task modification parameters, weight coefficients and adjustment coefficients are introduced. The weight coefficient reflects the importance of each modification content, while the adjustment coefficient is dynamically adjusted according to the modification frequency after the user is granted permission, which makes the calculation of the task modification parameters more flexible and more adaptable. When the task modification parameter exceeds the preset modification parameter threshold, the modification abnormality alarm module will trigger an alarm to promptly notify the management personnel of the abnormal modification behavior. This helps to promptly discover and respond to potential security risks or improper operations. Through real-time monitoring of user modification behavior and abnormal alarm, the technical solution can enhance the security of the task management system and prevent unauthorized modification or malicious tampering with task settings. By analyzing the number of user modifications, modification content, and modification parameters, we can gain an in-depth understanding of user usage habits and needs, and provide valuable feedback for further optimization and improvement of the system.

[0081] In summary, this technical solution realizes comprehensive monitoring and analysis of user modification task behavior through the collaborative work of modules such as real-time monitoring, information extraction, parameter calculation and abnormal alarm, thereby improving the security of the task management system and the accuracy of user behavior analysis.

[0082] Specifically, the task scheduling unit further includes:

[0083] A real-time monitoring module for the number of deletion frequencies is used to monitor in real time the frequency of information deletion performed by users on the tasks of the task manager according to the permissions granted;

[0084] A deletion frequency comparison module, used to compare the deletion frequency with a preset deletion frequency threshold;

[0085] A deleted information content extraction module, used for extracting deleted information data corresponding to each information deletion when the deletion frequency exceeds a preset deletion frequency threshold;

[0086] The task deletion parameter acquisition module is used to acquire the task deletion parameter according to the deletion frequency and deletion information content of the task manager performed by the user according to the granted authority, wherein the task deletion parameter is acquired by the following formula:

[0087]

[0088] Among them, S represents the task deletion parameter; f si represents the frequency of information deletion in the i-th unit time; K represents the compensation coefficient; f sp Indicates the overall deletion frequency of users; f p Indicates the overall modification frequency of users;

[0089] The deletion abnormality alarm module is used to issue a task deletion abnormality alarm when the task deletion parameter exceeds a preset deletion parameter threshold.

[0090] The technical effect of the above technical solution is: through the real-time monitoring module of the deletion frequency, the system can monitor the frequency of users deleting tasks in the task manager in real time. This real-time monitoring mechanism helps to detect abnormal deletion behavior in a timely manner and provide data support for subsequent early warning and response measures. The deletion frequency comparison module compares the deletion frequency monitored in real time with the preset deletion frequency threshold. Once the threshold is exceeded, the system will consider that there may be an abnormality, thereby triggering the subsequent information extraction and parameter acquisition process. This mechanism helps to identify abnormal deletion behavior in a timely manner.

[0091] When the deletion frequency exceeds the preset threshold, the deletion information content extraction module extracts the information data of each deletion, which helps to analyze the user's deletion behavior, understand the motivation and reasons behind it, and then evaluate whether it is compliant and normal. The task deletion parameter acquisition module calculates a task deletion parameter using a specific formula by comprehensively considering the deletion frequency, compensation coefficient, and the overall deletion and modification frequency of the user. This parameter provides a basis for objectively and quantitatively evaluating the user's deletion behavior. When the task deletion parameter exceeds the preset deletion parameter threshold, the deletion anomaly alarm module triggers an alarm to notify the management personnel to intervene in time. This mechanism helps to discover and respond to potential deletion anomalies in the first place, ensuring the security of the task manager and the integrity of the data. The introduction of the compensation coefficient K enables the solution to dynamically adjust the calculation of the task deletion parameter according to the overall deletion and modification frequency of the user, enhancing the flexibility of the solution and its adaptability to different user behaviors.

[0092] In summary, this technical solution achieves comprehensive monitoring and early warning of user task deletion behavior through real-time monitoring of deletion frequency, comparison with preset thresholds, extraction of deletion information, calculation of task deletion parameters, and triggering of abnormal alarms. This not only improves the security of the task management system, but also provides an effective technical means for timely detection and handling of abnormal deletion behavior.

[0093] Task execution unit, including:

[0094] The scheduling execution module implements timed triggering tasks based on the distribution of microservices. The scheduler first distributes the configured scheduling tasks to the executors that meet the configuration conditions, and then the scheduler executors execute the distributed scheduling tasks;

[0095] The execution feedback module is used to obtain the task execution results and feed back the task execution results to the user so that the user can understand the task execution status.

[0096] Among them, the task execution result is the execution status of each scheduled task, which includes the task completion date. Whether the scheduled task is overdue is determined based on the execution status. If the scheduled task is not overdue, the message of successfully triggering the task execution will be fed back to the user.

[0097] The technical effect of the above content is as follows: the task execution unit implements timed triggering tasks based on the distribution of microservices. The scheduling execution module first distributes the configured scheduling tasks to the executors that meet the configuration conditions through the scheduler, and then the scheduler executor executes the distributed scheduling tasks. The executor can execute the assigned tasks according to a variety of scheduling algorithms based on the type of task, waiting time and priority information, and the execution results will be collected by the execution feedback module, so that the task execution results can be fed back to the user so that the user can understand the execution status of each task.

[0098] Real-time monitoring unit, including:

[0099] A monitoring module is used to monitor the status of the task execution system in real time, and to monitor the task execution status in real time according to the task execution status feedback of the task execution unit;

[0100] The analysis module is used to record and analyze system logs. Based on the monitored task execution status, users can check the cause of the error and provide early warning for task execution system failures.

[0101] The technical effect of the above content is: the real-time monitoring unit can monitor the task execution system status in real time through the monitoring module, and monitor the task execution status in real time according to the task execution status feedback of the task execution unit. The analysis module then records and analyzes the task execution system log, and based on the monitored task execution status, the user can check the cause of the error, thereby providing an early warning for the task execution system failure, thereby achieving the purpose of operational monitoring.

[0102] The task manager is connected to the task scheduling unit. The task manager is used to add, modify and delete tasks. Users add tasks in the task manager; the scheduler is used to schedule tasks and assign them to reasonable executors. The scheduler is used to determine the execution order and time of tasks, and the executor is used to execute the assigned scheduled tasks.

[0103] The task manager adds a new task, which is written into the database via the API server. The API server uses TaskJob to query all tasks, and queries all tasks whose update time is greater than the last task on the timeline. TaskJob searches for the newly acquired task in the existing tasks, removes the old tasks and adds the new tasks, then parses the cron expression, obtains the next trigger time of the new task and calculates the time interval, so that the thread enters the dormant state with this time interval as a parameter.

[0104] Technical effect of the above content: The present invention can realize timed triggering tasks based on the distributed microservices. At the same time, relying on cron's flexible support for the calendar and cooperating with redis, it realizes the characteristics of 7x24 hours business separation and dynamic update of local distribution. Adding, modifying or removing any logical task will not affect other logical tasks, so that the execution program can be deployed with high availability and will not be unable to execute due to the restart or downtime of a machine. By parsing the cron expression, the next trigger time is calculated, and then the time interval is used as a parameter for sleep, so that the execution cycle is flexible and configurable.

[0105] Specifically, this embodiment also proposes an implementation method of a distributed anti-concurrency task execution system, including the following steps:

[0106] S1: The task scheduling unit receives and schedules tasks. The task execution unit triggers tasks based on the distributed implementation of microservices and feeds back messages that successfully trigger task execution.

[0107] S2: The user adds a new task in the task manager, and the update time is the current latest time. The new task is written to the database through the API server. TaskJob queries all tasks through the API server every minute for tasks whose update time is greater than the last task on the timeline. TaskJob then searches for the newly obtained task in the existing tasks, removes the old task and adds the new task.

[0108] S3: When the sleep time of the new task ends, it is automatically awakened and calls the corresponding http request configured in the task, carrying Authorization as the identity credential. After the call is completed, the last execution status and time information of the task are modified through the API server, and then the cron expression is parsed and the next trigger time is calculated, and the task is put into sleep with this time interval as the parameter;

[0109] S4: Monitor the task execution system status and task execution status in real time through the real-time monitoring unit. At the same time, record and analyze the system logs and check the causes of errors to provide early warning for task execution system failures.

[0110] Specifically, Figure 1As shown, the task management interface of the task manager is operated by the operator, who can configure the task's running cycle, availability, and the microservice address and permission information corresponding to the task. The task metadata is as follows:

[0111] TNAME is the task name;

[0112] CRON is the expression that triggers Cron;

[0113] UPDATETIME is the modification time;

[0114] POST URL is the URL address for executing the request;

[0115] AUTHORIZATION is the content of the AUTHORIZATION header in the request;

[0116] ENABLE is whether authorization is available. The content in the authorization header is enabled when requesting.

[0117] NEXT PLAN TIME is the time for the next plan execution;

[0118] LAST RUN TIME is the time when the last task was executed;

[0119] LAST USED TIME is the time used for the last task execution;

[0120] LAST STATE is the status of the last task execution;

[0121] Each of the above task metadata can form an http request and has a corresponding cron expression. The detailed business process is shown in steps S1-S4. It should be noted that steps S1-S4 require asynchronous encapsulation of the task on the microservice side, so that it can start another thread after receiving the request to process the business logic of the task. At the same time, the message of successfully triggering the task execution is fed back to TaskJob, after which TaskJob can rest and wait for the next wake-up.

[0122] like Figure 2 As shown in Table 1, in the case of tasks, there is at least one machine serving, so nginx is used to handle multi-machine loads. This situation is as follows:

[0123]

[0124] Table 1

[0125] Task A is triggered at 00:00:00, and then every 10 seconds;

[0126] Due to the peak period of the task, the amount of data for each execution increases, and the time required for a task execution reaches 60 seconds;

[0127] At this time, at 00:00:10, the first task of task A is still executing, which triggers the second task. At 50-60 seconds, there are already as many as 6 tasks executing the same code at the same time.

[0128] The above content solves the problem that functional codes are executed concurrently at the same time in microservice programs.

[0129] like Figure 3-4 As shown in the figure, if two requests are loaded by nginx to two different machines, the problem of data being executed repeatedly will still occur. In order to solve this problem, a global distributed lock must be introduced, so redis can be used. Its application method is as follows:

[0130] The request enters a certain logic code through http. Where the logic code starts, write the "task id" as the key to redis and execute the Redis command:

[0131] SET key value [EX seconds] [PX milliseconds] NX. This command is to write if it does not exist, and not write if it exists. If the return value is successful, it means that it can be executed, which is the N of "Whether locked" in the figure. The code continues to execute. If the write fails, it means that the lock already exists. At this time, it returns directly without executing the business code.

[0132] If the business code is executed, the lock must be released after the business code is executed, that is, the Redis command executed must be deleted.

[0133] If the business code encounters an exception or error during execution, or is shut down and redeployed, or is stuck due to an exception, the lock written in redis will not be released. Therefore, it is necessary to set the EX seconds or PX milliseconds in SET keyvalue [EX seconds] [PX milliseconds] NX to ensure that it can automatically release the lock when it times out, so that new requests can come in and execute the business again to ensure uninterrupted business.

[0134] Among them, EX seconds is to set the expiration time of the key in seconds; PX milliseconds is to set the expiration time of the key in milliseconds; NX sets the key only when the key does not exist.

[0135] In this embodiment, the timeout period for automatic lock release is set to 5 minutes. Therefore, when a task is closed or fails during execution and the lock cannot be released in time, the lock will be automatically released after 5 minutes.

[0136] The reasons why the lock cannot be released correctly are as follows:

[0137] The business code is abnormal;

[0138] The entire application or pod or management container or server is shut down.

[0139] For the first case, you can use try..catch...finally and release the lock in finally to solve it. That is, try is used to capture exceptions, catch is used to handle exceptions captured in try, and finally, regardless of whether the exception is captured or handled, the code in the finally block will be executed.

[0140] For the second case, you can set the lock's expiration time to 20 seconds. Then, start a new thread to monitor the lock. If the lock is about to expire in 10 seconds, renew it, that is, reset its expiration time to 20 seconds. As long as the process is still there, the lock will be renewed by the monitoring thread unless it is explicitly released. If the process is closed or terminated unexpectedly, the lock will be released within the next 20 seconds due to expiration because there is no thread to renew it.

[0141] Working principle: The distributed system based on microservices can realize timed triggering tasks. At the same time, relying on cron's flexible support for calendars and cooperating with redis, it realizes the characteristics of 7x24 hours business separation and dynamic update of local layout. Adding, modifying or removing any logical task will not affect other logical tasks, so that the execution program can be deployed with high availability and will not be unable to execute due to the restart or downtime of a machine. By parsing the cron expression, the next trigger time is calculated, and then the time interval is used as a parameter for sleep, so that the execution cycle is flexible and configurable, and the task execution unit can feedback the message of successful triggering of task execution, and the real-time monitoring unit can monitor the system status, task execution status and record and analyze system logs in real time, so that the operation can be monitored.

[0142] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0143] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A distributed anti-concurrency task execution system, characterized in that: include: The task scheduling unit is used to receive and schedule tasks from the task manager, identify user login information according to the login request, and grant permissions according to the user login information. At the same time, it creates an executor and configures the scheduling task according to the user login information. The scheduling task is configured to select an executor that meets the configuration conditions; The task execution unit is used to distribute the configured scheduling tasks to the executors that meet the configuration conditions. The executors execute the distributed scheduling tasks. The task execution unit can realize the timed triggering tasks based on the distribution of microservices. The executors select the scheduling tasks in the task list. The executors execute the selected scheduling tasks regularly and have task execution status feedback; A real-time monitoring unit is used to monitor the status of the task execution system and the task execution status in real time, feed back the status of the task execution system and the task execution status to the user, and record and analyze system logs to provide early warning of task execution system failures; The task scheduling unit includes: The authority authentication module is used to identify the user login information according to the login request and grant permissions according to the user login information. The user can modify and delete the tasks of the task manager according to the granted permissions, or add new tasks in the task manager, and then configure the scheduling tasks; The task scheduling module is used to create an executor according to the user login information, configure one or more executors matching the user login information, and schedule the task to select an executor that meets the configuration conditions; The task scheduling unit further includes: A modification count real-time monitoring module is used to monitor in real time the number of modifications made by users to the tasks of the task manager according to the granted permissions; A modification information content extraction module is used to extract the modification information content corresponding to each modification made by the user; The task modification parameter acquisition module is used to acquire the task modification parameter according to the number of modifications made by the user to the task manager according to the granted authority and the modification information content, wherein the task modification parameter is acquired by the following formula: ; Where G represents the task modification parameter; n represents the number of times the user modifies the task of the task manager according to the granted permissions; g i The weight coefficient of the information content modified for the i-th time; C i Indicates the number of characters modified during the i-th modification of the information content; C 0i Indicates the total number of characters corresponding to the total content of the task in which the information content is modified for the i-th time; x represents the adjustment coefficient, and the adjustment coefficient is obtained by the following formula; ; Wherein, m represents the total number of unit time contained in the duration experienced after the current user is granted permission, and the unit time is 24 hours; f i represents the frequency of modification occurring in the i-th unit time; The modification abnormality alarm module is used to issue a task modification abnormality alarm when the task modification parameter exceeds a preset modification parameter threshold.

2. The distributed anti-concurrency task execution system according to claim 1, characterized in that: The task manager is connected to the task scheduling unit. The task manager is used to add, modify and delete tasks. The user adds tasks in the task manager; the scheduler is used to schedule tasks and assign tasks to reasonable executors. The scheduler is used to determine the execution order and time of tasks, and the executor is used to execute the assigned scheduled tasks.

3. The distributed anti-concurrency task execution system according to claim 2 is characterized in that: The task manager adds a new task, which is written into the database via the API server. The API server uses TaskJob to query all tasks, and queries all tasks whose update time is greater than the last task on the timeline. TaskJob searches for the newly acquired task in the existing tasks, removes the old task and adds the new task, then parses the cron expression, obtains the next trigger time of the new task and calculates the time interval, so that the thread enters a dormant state with this time interval as a parameter.

4. The distributed anti-concurrency task execution system according to claim 1, characterized in that: The task scheduling unit further includes: A real-time monitoring module for the number of deletion frequencies is used to monitor in real time the frequency of information deletion performed by users on the tasks of the task manager according to the permissions granted; A deletion frequency comparison module, used to compare the deletion frequency with a preset deletion frequency threshold; A deleted information content extraction module, used for extracting deleted information data corresponding to each information deletion when the deletion frequency exceeds a preset deletion frequency threshold; The task deletion parameter acquisition module is used to acquire the task deletion parameter according to the deletion frequency and deletion information content of the task manager performed by the user according to the granted authority, wherein the task deletion parameter is acquired by the following formula: ; Among them, S represents the task deletion parameter; f si represents the frequency of information deletion in the i-th unit time; K represents the compensation coefficient; f sp Indicates the overall deletion frequency of users; f p Indicates the overall modification frequency of users; The deletion abnormality alarm module is used to issue a task deletion abnormality alarm when the task deletion parameter exceeds a preset deletion parameter threshold.

5. The distributed anti-concurrency task execution system according to claim 1, characterized in that: The task execution unit comprises: The scheduling execution module implements timed triggering tasks based on the distribution of microservices. The scheduler first distributes the configured scheduling tasks to the executors that meet the configuration conditions, and then the scheduler executors execute the distributed scheduling tasks; The execution feedback module is used to obtain the task execution results and feed back the task execution results to the user so that the user can understand the task execution status.

6. The distributed anti-concurrency task execution system according to claim 5, characterized in that: The task execution result is the execution status of each scheduled task, including the task completion date. Whether the scheduled task is overdue is determined based on the execution status. If the scheduled task is not overdue, a message that the task execution is successfully triggered is fed back to the user.

7. The distributed anti-concurrency task execution system according to claim 5, characterized in that: The task execution unit specifically performs the following operations: The execution feedback module implements timed triggering tasks based on the distributed implementation of microservices. Tasks are scheduled through the scheduler. After the task is triggered, the scheduled task is assigned to a reasonable executor. The executor executes the assigned tasks assigned to the executor according to various scheduling algorithms based on the type, waiting time and priority information of the task; The task execution result is fed back through the execution feedback module, and the message that successfully triggered the task execution is fed back.

8. The distributed anti-concurrency task execution system according to claim 1, characterized in that: The real-time monitoring unit comprises: A monitoring module is used to monitor the status of the task execution system in real time, and to monitor the task execution status in real time according to the task execution status feedback of the task execution unit; The analysis module is used to record and analyze system logs. Based on the monitored task execution status, users can check the cause of the error and provide early warning for task execution system failures.

9. The implementation method of the distributed anti-concurrency task execution system according to any one of claims 1 to 8, characterized in that: The following steps are involved: S1: The task scheduling unit receives and schedules tasks. The task execution unit triggers tasks based on the distributed implementation of microservices and feeds back messages that successfully trigger task execution. S2: The user adds a new task in the task manager, and the update time is the current latest time. The new task is written to the database through the API server. TaskJob queries all tasks through the API server every minute for tasks whose update time is greater than the last task on the timeline. TaskJob then searches for the newly obtained task in the existing tasks, removes the old task and adds the new task. S3: When the sleep time of the new task ends, it is automatically awakened and calls the corresponding http request configured in the task, carrying Authorization as the identity credential. After the call is completed, the last execution status and time information of the task are modified through the API server, and then the cron expression is parsed and the next trigger time is calculated, and the task is put into sleep with this time interval as the parameter; S4: Monitor the task execution system status and task execution status in real time through the real-time monitoring unit. At the same time, record and analyze the system logs and check the causes of errors to provide early warning for task execution system failures.

Images