A task processing method and system, an electronic device, and a storage medium

Abstract

The disclosure provides a task processing method, system, electronic device and storage medium, relates to the technical field of data processing, and particularly relates to the technical field of task processing. The specific implementation scheme is as follows: configuration information of a target to-be-executed task and a task execution graph are acquired; based on the configuration information of the target to-be-executed task and the task execution graph, a first executable task corresponding to the target to-be-executed task is generated; the first executable task is converted into a second executable task that can be executed by a target task execution platform; the second executable task is pushed to the target task execution platform; and a task execution result obtained by the target task execution platform executing the second executable task is acquired, the task execution result of the second executable task representing a task execution result of the target to-be-executed task, so that the custom configuration execution of the task is realized, and the business scenario of the task processing is expanded.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, further relates to the technical field of task processing, task scheduling and the like, and in particular relates to a task processing method and system, an electronic device and a storage medium. BACKGROUND

[0002] With the continuous expansion of the business scale of enterprises and the continuous increase of the number of users, the enterprises pay more and more attention to the security of user identity. The number of corresponding tasks such as security management of the enterprises themselves and behavior analysis of users is increasing, and the business scenarios of task processing are expanding, and the execution efficiency of tasks is improved, which can enable the enterprises to better achieve the security management of themselves and improve the timeliness of the behavior analysis of users. SUMMARY

[0003] The present disclosure provides a task processing method and system, an electronic device and a storage medium.

[0004] According to an aspect of the present disclosure, a task processing method is provided, comprising:

[0005] obtaining configuration information and a task execution graph of a target to-be-executed task; wherein the configuration information and the task execution graph of the target to-be-executed task are set in advance for the target to-be-executed task; the task execution graph includes a target task execution platform for executing the target to-be-executed task;

[0006] generating a first executable task corresponding to the target to-be-executed task based on the configuration information and the task execution graph of the target to-be-executed task;

[0007] converting the first executable task into a second executable task executable by the target task execution platform;

[0008] pushing the second executable task to the target task execution platform;

[0009] obtaining a task execution result of the second executable task executed by the target task execution platform, the task execution result of the second executable task representing a task execution result of the target to-be-executed task.

[0010] According to another aspect of the present disclosure, a task processing system is provided, comprising:

[0011] an information obtaining module, configured to obtain configuration information and a task execution graph of a target to-be-executed task; wherein the configuration information and the task execution graph of the target to-be-executed task are set in advance for the target to-be-executed task; the task execution graph includes a target task execution platform for executing the target to-be-executed task;

[0012] a task generation module, configured to generate a first executable task corresponding to the target to-be-executed task based on the configuration information of the target to-be-executed task and the task execution graph;

[0013] a task conversion module, configured to convert the first executable task into a second executable task that can be executed by the target task execution platform;

[0014] a task allocation module, configured to push the second executable task to the target task execution platform;

[0015] a result acquisition module, configured to acquire a task execution result of the second executable task obtained by the target task execution platform executing the second executable task, the task execution result of the second executable task representing a task execution result of the target to-be-executed task.

[0016] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0017] at least one processor; and

[0018] a memory connected to the at least one processor in communication; wherein

[0019] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of the present disclosure.

[0020] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the method of any one of the present disclosure.

[0021] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method of any one of the present disclosure.

[0022] The embodiments of the present disclosure realize the customized configuration and execution of tasks, and expand the business scenarios of task processing.

[0023] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them:

[0025] Figure 1is a system architecture diagram for implementing a task processing method according to an embodiment of the present disclosure;

[0026] Figure 2 is a schematic diagram of a task processing method according to the present disclosure;

[0027] Figure 3 is a schematic diagram of a task execution diagram according to the present disclosure;

[0028] Figure 4 is a schematic diagram of a task execution driver according to the present disclosure;

[0029] Figure 5 is a schematic diagram of task generation and distribution execution according to the present disclosure;

[0030] Figure 6 is another schematic diagram of a task processing method according to the present disclosure;

[0031] Figure 7 is a schematic diagram of determining whether an instance node reaches an upper limit of load balancing according to the present disclosure;

[0032] Figure 8 is a schematic diagram of task data display according to the present disclosure;

[0033] Figure 9 is a schematic diagram of locking task recovery according to the present disclosure;

[0034] Figure 10 is a schematic diagram of determining whether a task is executed normally according to the present disclosure;

[0035] Figure 11 is a schematic diagram of a task processing system according to the present disclosure;

[0036] Figure 12 is a block diagram of an electronic device for implementing a task processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help the understanding of the present disclosure. These should be considered in the context of the overall description and should not be considered limiting in any way. Thus, it will be apparent to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of brevity and clarity, descriptions of well-known functions and constructions are omitted from the following description.

[0038] Identity and Access Management (IAM) of an enterprise has entered a relatively mature stage in the basic account / permission management function layer after multiple product iterations and evolution. However, as the business scale of the enterprise continues to expand and the number of users continues to increase, the enterprise pays more and more attention to user identity security. Enterprises still have various identity security related problems, making it necessary for enterprises to expand higher-level security management and analysis functions of identity recognition. The most advanced user identity security management and analysis function is adaptive identity authentication.

[0039] User Entity Behavior Analysis (UEBA) technology is the basis of adaptive identity authentication. The main function of UEBA is to analyze whether a user (including natural persons and device entities, collectively referred to as users) has abnormal behavior and security risks based on the behavior patterns of the user in the enterprise system, and feed back the analysis results to the system or prompt the system administrator, so that the system administrator can give appropriate disposal.

[0040] However, as the business scale of the enterprise continues to expand and the number of users continues to increase, small enterprises may have several hundred user behavior records per day, while large enterprises and public clouds may have user behavior data reaching billions or even hundreds of billions per day, resulting in a huge amount of data to be analyzed for user behavior, and accordingly, a large number of tasks to be analyzed for user behavior. Moreover, different enterprises often have different business scenarios, making it a problem to be solved how to process data tasks in different business scenarios.

[0041] To solve the above problems, the present disclosure provides a task processing method, which acquires configuration information and a task execution graph of a target to-be-executed task; wherein the configuration information and the task execution graph of the target to-be-executed task are customized and set in advance for the target to-be-executed task, and the task execution graph includes a target task execution platform that executes the target to-be-executed task; based on the configuration information and the task execution graph of the target to-be-executed task, a first executable task corresponding to the target to-be-executed task is generated; the first executable task is converted into a second executable task that can be executed by the target task execution platform; the second executable task is pushed to the target task execution platform; a task execution result of the second executable task obtained by the target task execution platform executing the second executable task is acquired, and the task execution result of the second executable task represents a task execution result of the target to-be-executed task.

[0042] In the embodiments of the present disclosure, the configuration information and the task execution graph of the target to-be-executed task can be customized in advance for different target to-be-executed tasks, and then the first executable task corresponding to the target to-be-executed task is generated based on the customized configuration information and the task execution graph, thereby realizing the customized configuration and generation of the task and expanding the business scenarios of the task processing. Further, the first executable task is converted into a second executable task that can be executed by the target task execution platform, and the converted second executable task is pushed to the target task execution platform for execution, thereby ensuring the flexible scalability of the task computing capability in the task execution process and enabling the processing of the task to adapt to the internal execution environment of different enterprises.

[0043] The task processing method provided by the embodiments of the present disclosure can be applied to the UEBA-based user abnormal behavior analysis scenario, and accordingly, the task in the embodiments of the present disclosure can be a user abnormal behavior analysis task. Of course, the task processing method of the embodiments of the present disclosure can also be applied to any scenario that needs to execute a task. The embodiments of the present disclosure take the UEBA-based user abnormal behavior analysis task as an example for illustration. In one example, the system architecture for implementing the task processing method of the embodiments of the present disclosure is as shown in FIG. 1. Figure 1

[0044] Figure 1 The system is a UEBA-based user abnormal behavior analysis system, and the task control module in the system is used to implement the task processing process in the embodiments of the present disclosure. Specifically, the task control module is responsible for controlling the entire life cycle of the task, including the starting, distribution, assembly, conversion, retry, termination and completion of the task. According to the different life cycles of the task, the task control module can be divided into a task retry submodule, a task load balancing submodule, a task assembly submodule and a task conversion submodule.

[0045] ​To prevent the repeated execution of tasks, the tasks are locked during execution, but the system may have power failure, downtime and other sudden situations, so that the tasks are interrupted and cannot be unlocked. The task retry submodule can unlock and recover the tasks that are interrupted or failed but still in the locked state, and try to execute again. The task load balancing submodule is responsible for balancing the distribution of tasks to different instance nodes for execution to ensure the load balancing of each instance node. For example, an instance node can correspond to a machine device. The task assembly submodule can integrate and assemble the user-defined task configuration information and the task execution graph (TaskGraph) into a complete executable task in memory. The assembled executable task contains a series of task execution logic (or operator), and the assembled executable task can be executed, terminated, and completed. The task conversion submodule can convert the executable task assembled locally by the instance node into an executable task under different execution platforms through syntax conversion, so that the task can be pushed to different execution platforms for execution to ensure the flexible expansion of the system computing capacity.

[0046] Figure 1 The control execution in the system is to control and execute the assembled executable task, which contains a series of task execution logic (or operator) and is executed by various components in the system, such as data access components, data modeling components, risk engine components, and external service components. The data access component can include data collection, data filtering, data conversion, and event view processing. The data modeling component can include baseline modeling, entity association, and entity behavior modeling. The risk engine component can include baseline analysis, clustering analysis, static rules, and risk scoring. The external service component can include configuration interface, console, and risk data.

[0047] Different data sources usually have great differences in data acquisition methods, data meanings, data formats, and data integrity. The data access component eliminates the differences between different data sources by performing data collection, data filtering, data conversion, and event view operations on the data obtained from different data sources to realize data visualization.

[0048] The data access component can obtain data from external dependencies, which represent different data sources, such as device log data, audit log data, interface log data, database log data, network traffic data, and the like. The data access component can collect data in the form of files, through an API (Application Programming Interface), or a kafka queue, and the like, from different data sources. After data collection, the collected data is filtered using pre-set rules, such as a task of counting user login frequency, and the corresponding rule can be an expression of a login interface path. Further, for the filtered data, the keywords (or values) in the data are corresponded to pre-set fields, converted into key-value pairs, and the user identity in the filtered data is parsed to obtain entity information corresponding to the user, and an event view is generated based on the obtained key-value pairs and entity information. For example, the obtained data is {“name”:“zhangsan”,“age”:20,“height”:180}, the subscribed view is [“name”,“age”], and the extracted data (i.e., the generated event view) is {“name”:“zhangsan”,“age”:20}, which can be transmitted to the data modeling component in the form of a data stream. The event view can also be stored, such as in a Mysql database, a Redis (Remote dictionary server) database, or a graph database.

[0049] Figure 1 The data modeling component in the system generates a series of description data of user behavior based on the data provided by the data access component, and then transmits the data to the risk engine component in the form of a data stream. The data modeling component generates baseline modeling information based on the data provided by the data access component, such as entity behavior information including account login frequency and access location, entity association behavior information including account and account, account and device, and abnormal information including determination of whether to exceed system rules and thresholds, and the like, wherein the system rules and thresholds can be generated based on existing user access environment, access behavior experience, and the like.

[0050] The risk engine component in the graph system 1 analyzes whether the user event is abnormal based on the series of description data generated by the data modeling component, and the main analysis methods include baseline analysis, clustering analysis, static rules, and risk scoring. In the process of analyzing the data, the risk engine component can call other distributed computing engines, such as a Spark computing engine.

[0051] Figure 1The results of the risk engine component's analysis are transmitted to the external service component in the form of a data stream. The external service component provides configuration interfaces and a console accessible to administrators, who can then configure and operate tasks through these interfaces and the console. The external service component also provides query interfaces for risk data, enabling IdaaS (Alibaba Cloud Identity as a Service), IAM, and other applications to query risk data through the external service component.

[0052] The task processing method provided in the embodiments of this disclosure will be described in detail below.

[0053] The task processing method provided in this disclosure can be applied to electronic devices, such as server devices, cluster devices, and cloud service devices. Preferably, the execution entity for executing the task processing method can be any instance node, with one instance node corresponding to one machine device. The application scenario of the task processing method provided in this disclosure can be user anomaly behavior analysis scenarios based on UEBA. User anomaly behavior analysis scenarios supported by UEBA can include: short-term login from different locations, login from uncommon regions, login from uncommon IP (Internet Protocol) addresses, login at uncommon times, frequent logins within a short period, login from non-whitelisted IP addresses, abnormal key access frequency, and monitoring of unused keys.

[0054] See Figure 2 , Figure 2 A flowchart illustrating a task processing method provided in this disclosure includes the following steps:

[0055] S201, Obtain the configuration information and task execution graph of the target task to be executed.

[0056] The configuration information and task execution graph of the target task to be executed are pre-defined for the target task to be executed. The task execution graph includes the target task execution platform that executes the target task to be executed.

[0057] In this embodiment of the disclosure, the above-mentioned Figure 1 The system shown is an example of a UEBA-based user abnormal behavior analysis system, which includes a fully customizable task execution model. Figure 1 (Not shown in the image), this task execution model allows users to develop and configure complete task execution processes, enabling the system to support various task analysis scenarios. Specifically, users can... Figure 1The configuration interface and the console in the external service component shown are used to customize the configuration information of the task and the task execution graph in the task execution model, so as to dynamically expand the business scenarios of the task.

[0058] Different enterprises have different magnitudes of user behavior data generated internally. For small and medium-sized enterprises, the daily user behavior data may only have several thousand to tens of thousands of pieces. Such magnitude of data can usually be analyzed in the local physical machine / virtual machine memory. For large enterprises, the daily user behavior data may reach the level of hundreds of thousands or even millions of pieces. Such magnitude of data cannot be calculated by the enterprise by providing corresponding physical machines / virtual machines as computing resources, and thus requires a dedicated computing engine to achieve.

[0059] When the target task to be executed is customized and configured, the target task execution platform for executing the target task to be executed can be configured according to the task amount of the target task to be executed, the resources required to be consumed, and the like. In an example, whether the local physical machine / virtual machine can complete the target task to be executed can be estimated according to the size of the computing resources occupied by the target task to be executed, the complexity of the calculation, and the frequency of the calculation, and the like. If the local physical machine / virtual machine can complete the target task to be executed, the target task execution platform of the target task to be executed is configured as the local computing engine. If the local computing engine is insufficient to support the completion of the target task to be executed, the target task execution platform of the target task to be executed is configured as the dedicated computing engine. The target task execution platform for executing the target task to be executed included in the task execution graph may be, for example, the name identifier or address information of the target task execution platform.

[0060] In a possible implementation, the target task execution platform described above can include a local computing engine, a Spark computing engine, and a Flink computing engine.

[0061] In the embodiments of the present disclosure, the task is simultaneously supported to be executed in the local computing engine (local physical machine / virtual machine), the Spark computing engine, and the Flink computing engine, which guarantees the flexible scalability of the task computing capability in the task execution process, so that the processing of the task can adapt to the internal execution environment of different enterprises.

[0062] When the instance node of the UEBA-based user abnormal behavior analysis system detects that the task needs to be processed, the configuration information of the target task to be executed and the task execution graph are acquired.

[0063] S202, a first executable task corresponding to the target task to be executed is generated based on the configuration information of the target task to be executed and the task execution graph.

[0064] In an example, the instance node uses the target task execution platform corresponding to the target task to be executed to generate the first executable task corresponding to the target task to be executed. Figure 1The task assembly submodule in the illustrated system assembles the configuration information of the target to-be-executed task and the task execution graph to instantiate a first executable task corresponding to the target to-be-executed task.

[0065] In a possible implementation, the task execution graph of the target to-be-executed task can further include information corresponding to each task execution node that executes the target to-be-executed task, and execution order and connection relationship information of each task execution node; and the configuration information of the target to-be-executed task includes attribute information used by each task execution node when executing the target to-be-executed task. The information corresponding to the task execution node can be an identifier, a name, or the like of the task execution node.

[0066] A complete executable task includes a task execution graph, and the task execution graph includes task execution nodes (TaskExecuteNode) that execute the task, and each task execution node can be a specified execution logic or operator (TaskExecutor). The task execution graph is a directed acyclic graph, and the execution order and connection relationship between the task execution nodes are defined in the graph.

[0067] For example, a task execution graph is illustrated in FIG. 1, which includes task execution nodes A, B, C, D, and E that execute the task, and the execution order and connection relationship between the task execution nodes A, B, C, D, and E, that is, after the task execution node A is executed, the task execution nodes B and C are executed, after the task execution nodes B and C are executed, the task execution node D is executed, and after the task execution node D is executed, the task execution node E is executed. Figure 3

[0068] Each task execution node has corresponding task execution configuration information, and the task execution configuration information defines various configurable parameters used by the task execution node in the process of executing the task, that is, attribute information used by the task execution node when executing the target to-be-executed task.

[0069] For example, the operator corresponding to the task execution node is a statistical remote login event, and the operator defines a minimum time interval and an analysis logic that, when a remote login occurs within the minimum time interval, the login is counted as a remote login exception event. Correspondingly, the task execution configuration information of the task execution node includes the minimum time interval, the determination of the remote login event, and the determination method of the remote login exception event (that is, a remote login occurs within the minimum time interval).

[0070] ​For example, the operator corresponding to the task execution node is a statistical access frequency event. The operator defines a set time period and the analysis logic that, when the number of accesses occurring within the set time period exceeds a set threshold, this access is counted as an access frequency abnormal event. Accordingly, the task execution configuration information of this task execution node includes: the set time period, the set threshold, and the method for determining access frequency abnormal events (i.e., the number of accesses occurring within the set time period exceeds the set threshold).

[0071] In one example, each task execution node corresponds to a type. Different types of task execution nodes have different execution logic, and the type for each task execution node can be customized. For instance, the type corresponding to each task execution node could be as described above. Figure 1 The system shown includes pre-embedded common analyzer types (SystemTaskExecutor), or analyzers (CustomTaskExecutor) implemented according to the interface standards provided by UEBA and uploaded to the UEBA service. These data analysis types can be named by the user. This enables custom configuration of task execution nodes and expands the business scenarios for task analysis.

[0072] Accordingly, based on the configuration information of the target task to be executed and the task execution graph, a first executable task corresponding to the target task to be executed is generated, including: instantiating and generating the first executable task corresponding to the target task to be executed based on the information corresponding to each task execution node, the execution order and connection relationship information of each task execution node, and the attribute information used by each task execution node when executing the target task to be executed.

[0073] The process involves assembling each task execution node, its execution order and connection relationships, and the attribute information used by each task execution node when executing the target task. This assembly then instantiates the first executable task corresponding to the target task. Specifically, the process assembles each task execution node, its execution order and connection relationships, and the attribute information used by each task execution node when executing the target task into an executable task object in memory. This generates an executable instantiated program object, which specifies the relationships between the task execution nodes (such as the execution order, start, and termination of the task execution nodes).

[0074] In the embodiments of the present disclosure, the task execution graph of the target to-be-executed task includes information corresponding to each task execution node for executing the target to-be-executed task, and execution order and connection relationship information of each task execution node, and the configuration information of the target to-be-executed task includes attribute information used by each task execution node when executing the target to-be-executed task, so that a first executable task corresponding to the target to-be-executed task can be instantiated based on the information corresponding to each task execution node, the execution order and connection relationship information of each task execution node, and the attribute information used by each task execution node when executing the target to-be-executed task, thereby realizing self-definable setting of the task, dynamically expanding the task analysis scene, and enabling the user's analysis demand for various business scene tasks to be met in a highly scalable manner.

[0075] Referring to Figure 2 , S203, the first executable task is converted into a second executable task executable by the target task execution platform.

[0076] In one example, the instance node utilizes the task conversion submodule in the system shown in Figure 1 to convert the first executable task into a second executable task executable by the target task execution platform through syntax conversion.

[0077] In a possible implementation, a task execution driver (TaskGraphDriver) is defined in the UEBA system (i.e., the above-mentioned UEBA-based user abnormal behavior analysis system), as shown in Figure 4 , the task execution driver can include a local task execution driver (LocalTaskGraphDriver), a Spark task execution driver (SparkTaskGraphDriver), and a Flink task execution driver (FlinkTaskGraphDriver) corresponding to the above-mentioned target task execution platform.

[0078] The function of the task execution driver is to convert the task execution graph that is pre-defined and configured into an executable task (TaskWrapper) executable on the corresponding task execution platform. For example, as shown in Figure 5 , the task execution driver can convert the task execution graph that is pre-defined and configured into an executable task (TaskWrapper) executable on the corresponding task execution platform.

[0079] For example, the target task execution platform that executes the target task to be executed is a Spark computing engine, and the first executable task is converted into a second executable task that can be executed by the Spark computing engine by using a parseTaskGraph method, such as a task defined by an RDD (Resilient Distributed Datasets) of the Spark computing engine. The parseTaskGraph method implements different syntax conversions according to different computing engines.

[0080] In S204, the second executable task is pushed to the target task execution platform.

[0081] The converted second executable task is pushed to the target task execution platform, so that the target task execution platform executes the second executable task, and returns a task execution result of the second executable task after the second executable task is completed.

[0082] In S205, a task execution result obtained by executing the second executable task by the target task execution platform is acquired.

[0083] In one example, the task execution result returned by the target task execution platform after the second executable task is completed can be received, or the task execution result obtained by executing the second executable task by the target task execution platform can be read from the target task execution platform.

[0084] The task execution result of the second executable task represents a task execution result of the target task to be executed.

[0085] In the embodiments of the present disclosure, the configuration information and the task execution graph of the target task to be executed can be customized in advance for different target tasks to be executed, and then the first executable task corresponding to the target task to be executed is generated based on the customized configuration information and the task execution graph, so as to realize the customized configuration and generation of the task and expand the business scenarios of the task processing. Further, the first executable task is converted into the second executable task that can be executed by the target task execution platform, and the converted second executable task is pushed to the target task execution platform for execution, so as to ensure the flexible scalability of the task computing capability in the task execution process, and make the task processing adapt to different internal execution environments of enterprises.

[0086] Referring to Figure 6 , Figure 6 Another flowchart of a task processing method provided by the embodiments of the present disclosure includes the following steps:

[0087] In S601, it is determined whether a current instance node reaches a load balancing upper limit condition.

[0088] The current instance node determines whether it reaches the load balancing upper limit condition. In an example, the load balancing upper limit condition can be whether the number of tasks being executed by the current instance node reaches a preset value, or whether the memory utilization of the current instance node reaches a set utilization, and the like.

[0089] In the embodiments of the present disclosure, each current instance node is stateless.

[0090] S602, in the case where the current instance node does not reach the load balancing upper limit condition, taking one to-be-executed task from the to-be-executed task list as a target to-be-executed task.

[0091] In the case where the current instance node reaches the load balancing upper limit condition, the current instance node ends the execution of the task in step S609. In the case where the current instance node does not reach the load balancing upper limit condition, the to-be-executed task list of all to-be-executed tasks in the state of the database is queried, and further in the case where there is a to-be-executed task in the to-be-executed task list, one to-be-executed task is taken from the to-be-executed task list as a target to-be-executed task. In an example, the to-be-executed tasks in the to-be-executed task list can be traversed, and the to-be-executed tasks in the to-be-executed task list are taken one by one.

[0092] S603, locking the target to-be-executed task.

[0093] In order to ensure that one to-be-executed task can be executed by only one instance node at any time, in the case where the current instance node does not reach the load balancing upper limit condition, one to-be-executed task is taken from the to-be-executed task list as a target to-be-executed task, and the target to-be-executed task is pre-empted, that is, locking.

[0094] In a possible implementation, the locking of the target to-be-executed task includes: in the case where the state of the target to-be-executed task is in the unlocked state, updating the state of the target to-be-executed task to the locked state.

[0095] For example, the statement "update task set status='locked' where status='unlocked' where id=1" can be used to lock the target to-be-executed task, which indicates that the target to-be-executed task with the identifier of 1 is set to the locked state if its state is in the unlocked state.

[0096] Since only one instance node can successfully execute the statement at the same time, the rest of the instance nodes will fail to execute due to the task being updated to the locked state, thus achieving the purpose that only one instance node can execute a task at any time, and only the instance node that successfully locks can execute the task.

[0097] In the embodiments of the present disclosure, in the case that the state of the target to-be-executed task is in the unlocked state, the state of the target to-be-executed task is updated to the locked state, and the target to-be-executed task is preempted, so that only one to-be-executed task can be executed by one instance node at any time, and the repeated execution of the task is avoided.

[0098] In one example, the implementation of steps S601-S603 can be completed by the task load balancing submodule in the system shown in FIG. 1. Figure 1

[0099] S604, in the case that the target to-be-executed task is successfully locked, the configuration information and the task execution graph of the target to-be-executed task are obtained.

[0100] The configuration information and the task execution graph of the target to-be-executed task are set in advance for the target to-be-executed task, and the task execution graph includes a target task execution platform for executing the target to-be-executed task.

[0101] In the case that the target to-be-executed task is successfully locked, the target to-be-executed task is executed. In the case that the target to-be-executed task is not successfully locked, a to-be-executed task is taken out from the to-be-executed task list as the target to-be-executed task in step S602 until there is no to-be-executed task in the to-be-executed task list.

[0102] In the embodiments of the present disclosure, in the case that the current instance node does not reach the load balancing upper limit condition and the target to-be-executed task is not successfully locked, a to-be-executed task is taken out from the to-be-executed task list as the target to-be-executed task until there is no to-be-executed task in the to-be-executed task list, and the execution of the task on the basis of load balancing is realized.

[0103] S605, based on the configuration information and the task execution graph of the target to-be-executed task, a first executable task corresponding to the target to-be-executed task is generated.

[0104] S606, the first executable task is converted into a second executable task that can be executed by the target task execution platform.

[0105] S607, the second executable task is pushed to the target task execution platform.

[0106] ​S608, Obtain the task execution result obtained by the target task execution platform from executing the second executable task.

[0107] The second executable task's execution result indicates the execution result of the target task to be executed.

[0108] The implementation process of step S604, which obtains the configuration information and task execution graph of the target task to be executed, and steps S605-S608, can be referred to the implementation process of steps S201-S205 above. The embodiments disclosed herein will not be repeated here.

[0109] S609, End.

[0110] In this embodiment, stateless instance nodes are used for automatic task load balancing, enabling arbitrary scaling of instance nodes. This increases the scalability of task processing while ensuring each instance node can handle all tasks. Instance nodes lock and preempt tasks, ensuring that a task can only be executed by one instance node at any given time, preventing duplicate execution and improving real-time performance and accuracy. Furthermore, configuration information and execution graphs for different target tasks can be pre-defined. Based on these customized configurations and execution graphs, a first executable task is generated, enabling custom task configuration and generation and expanding the business scenarios for task processing. The first executable task is then converted into a second executable task that the target task execution platform can execute. This second executable task is then pushed to the target task execution platform for execution, ensuring flexible scalability of task computing power during execution and allowing task processing to adapt to different enterprise internal execution environments.

[0111] In one possible implementation, such as Figure 7 As shown, the implementation process of step S601 above, which determines whether the current instance node has reached the load balancing upper limit, includes:

[0112] S701, based on the target instance list, determines the number of currently valid instance nodes.

[0113] The target instance list is obtained by updating its own information and corresponding time information to the instance list at preset time intervals for each instance node; the currently valid instance node is indicated by the time difference between the time information of the instance node in the target instance list and the current system time being within the preset time range.

[0114] Each instance node can update its own information and corresponding time information into an instance list collection (uebaInstanceList) of a database at a preset time interval, and store the information as a target instance list. The target instance list stores the latest own information and corresponding time information of each instance node. The own information can be an identifier or name of the instance node, and the corresponding time information is time stamp information when the own information is updated to the database. The database can be a Redis database or a Mysql database, and the preset time interval can be configured according to requirements, for example, 5 seconds, 10 seconds, or 20 seconds.

[0115] For example, when the preset time interval is t, the instance nodes whose time information stored in the target instance list and the current system time have a time difference within 2t can be determined as the current effective instance nodes.

[0116] S702, determine the total number of tasks, including the number of tasks to be executed and the number of tasks being executed.

[0117] Query the task table stored in the database to determine the total number of tasks from the task table, including the number of tasks to be executed and the number of tasks being executed.

[0118] For example, the instance node represents a UEBA instance, as shown in Figure 8 As shown, each UEBA instance (i.e., instance node, Figure 8 three are shown) updates its own information and corresponding time information into a target instance list of a Redis database at a preset time interval, and then determines the number of current effective instance nodes by querying the target instance list. The task table in the Mysql database is queried to determine the total number of tasks.

[0119] S703, based on the number of current effective instance nodes and the total number of tasks, determine whether the current instance node reaches the load balancing upper limit condition.

[0120] In one example, according to the determined number of current effective instance nodes and the total number of tasks, the average number of tasks processed by each instance node can be calculated to determine whether the instance node reaches the load balancing upper limit condition.

[0121] In the embodiments of the present disclosure, according to the determined number of current effective instance nodes and the total number of tasks, load balancing is achieved according to the number of tasks, so that tasks can be evenly distributed to all instance nodes for execution.

[0122] In a possible implementation, the step S703 determines whether the current instance node reaches the load balance upper limit condition based on the current number of effective instance nodes and the current total number of tasks, including:

[0123] determining a quotient of the current total number of tasks and the current number of effective instance nodes and a sum of 1 as a load upper limit threshold value;

[0124] determining the number of tasks currently executed by the instance node;

[0125] determining that the current instance node does not reach the load balance upper limit condition in a case where the number of tasks currently executed by the instance node is less than the load upper limit threshold value;

[0126] determining that the current instance node reaches the load balance upper limit condition in a case where the number of tasks currently executed by the instance node is not less than the load upper limit threshold value.

[0127] For example, the current total number of tasks is represented as m, the current number of effective instance nodes is represented as n, and the load upper limit threshold value is represented as m / n+1.

[0128] Each instance node maintains a list of tasks currently executed by the instance node, and the number of tasks currently executed by the instance node can be known by querying the task execution list of the current instance node (current_execute_tasks).

[0129] determining that the current instance node does not reach the load balance upper limit condition in a case where the number of tasks currently executed by the instance node is less than the load upper limit threshold value, and otherwise, determining that the current instance node reaches the load balance upper limit condition.

[0130] In the embodiments of the present disclosure, the tasks are evenly distributed in the instance nodes according to the number of tasks, so that the tasks can be evenly distributed to all instance nodes for execution.

[0131] In a possible implementation, after obtaining the task execution result of the second executable task, the step S703 can further include: unlocking the target to-be-executed task and deleting the target to-be-executed task from the to-be-executed task list.

[0132] In the embodiments of the present disclosure, the tasks are locked during execution, the locked tasks are unlocked after execution, and the executed tasks are deleted from the to-be-executed task list to complete the tasks.

[0133] In the process of executing a task, the instance node can directly stop running due to system power failure, system crash, CPU or memory occupation, and the like, so that the task execution fails. The task that fails to execute cannot be unlocked because it is still in execution, and thus is always in a locked state and cannot be executed again.

[0134] To handle such tasks that fail to execute due to the instance node directly stopping running due to system power failure, system crash, CPU or memory occupation, and the like, in the embodiments of the present disclosure, the instance node uses the task retry sub-module in the system shown in the above Figure 1 to automatically recover and execute the task that fails to execute.

[0135] In a possible implementation, as shown in Figure 9 on the basis of the above embodiments, the following steps can also be performed. Preferably, the following steps can be performed before determining whether the current instance node reaches the upper limit condition of load balancing. The steps include:

[0136] S901, determining whether there is a locked task in a locked state.

[0137] The instance node queries whether there is a locked task in a locked state. If there is, steps S902-S903 are performed to recover the execution of the locked task. If there is not, there is no need to recover the locked task.

[0138] S902, in the case where there is a locked task in a locked state, determining, for each locked task, whether the locked task is normally executed.

[0139] In the case where there is a locked task in a locked state, each locked task in a locked state is traversed to determine whether each locked task is normally executed. The locked task that is not normally executed is recovered.

[0140] In one example, for each locked task, it is determined whether the locked task is executed by the instance node. If yes, it indicates that the locked task is normally executed. Otherwise, it indicates that the locked task is not normally executed.

[0141] S903, in the case where the locked task is not normally executed, unlocking the locked task, and adding the unlocked task obtained by the unlocking to a to-be-executed task list.

[0142] In the embodiments of the present disclosure, it is determined whether there is a locking task in a locking state, in the case that there is a locking task in a locking state, it is further determined whether the locking task is normally executed for each locking task, in the case that the locking task is not normally executed, the locking task is unlocked, and the unlocked task obtained by the unlocking is added to the to-be-executed task list, so that the automatic recovery and execution of the task not normally executed are realized, and the reliability of task processing is improved.

[0143] In a possible implementation, the locking task contains a target identifier of an instance node executing the locking task. Figure 10 As shown in the embodiment of determining whether the locking task is normally executed, the embodiment process includes:

[0144] S1001, judging whether the instance node executing the locking task is the current instance node based on the target identifier.

[0145] In the locking process of the target to-be-executed task, the identifier of the instance node executing the locking process can also be added to the locking task, the instance node executing the locking process of the target to-be-executed task is the instance node executing the locking task obtained after the target to-be-executed task is locked, and the locking task after the locking success contains the target identifier of the instance node executing the locking task. For example, the target identifier can be the name or IP address of the instance node.

[0146] For example, the locking process of the target to-be-executed task can use the statement "update task set status='locked', execute_node='current node host' where status='unlocked' and id='1'", which means that if the target to-be-executed task with the identifier 1 is in the unlocked state, the state of the target to-be-executed task is set to the locked state, and the instance node executing the target to-be-executed task is set to the current node host (the name of the instance node).

[0147] The target identifier contained in the locking task is matched with the identifier of the current instance node, in the case that the target identifier is the same as the identifier of the current instance node, it is indicated that the instance node executing the locking task is the current instance node, otherwise, it is indicated that the instance node executing the locking task is not the current instance node.

[0148] S1002, in the case that the instance node executing the locking task is the current instance node, it is determined whether the task execution list of the current instance node contains the locking task.

[0149] Each instance node maintains a list of tasks that it is currently executing, and in the case that the instance node executing the locked task is the current instance node, the task execution list of the current instance node is further queried to determine whether the locked task exists in the task execution list of the current instance node. If the locked task exists, it indicates that the locked task is normally executed in the current instance node, and if the locked task does not exist, it indicates that the locked task is not executed but is locked, and at this time, the locked task needs to be unlocked and recovered.

[0150] In the case that the locked task exists in the task execution list of the current instance node, S1003, it is determined that the locked task is normally executed.

[0151] In the case that the locked task does not exist in the task execution list of the current instance node, S1004, it is determined that the locked task is not normally executed.

[0152] In the case that the instance node executing the locked task is not the current instance node, S1005, it is determined whether the instance node executing the locked task is in the target instance list.

[0153] In the case that the instance node executing the locked task is not the current instance node, the target instance list of the database is further queried to determine whether the instance node executing the locked task exists. If the instance node executing the locked task is in the target instance list, it indicates that the locked task is normally executed in the instance node, and if the instance node executing the locked task is not in the target instance list, it indicates that the instance node executing the locked task no longer exists, and currently no instance node executes the locked task, and at this time, the locked task needs to be unlocked and recovered.

[0154] In the case that the instance node executing the locked task is in the target instance list, S1006, it is determined that the locked task is normally executed.

[0155] In the case that the instance node executing the locked task is not in the target instance list, S1007, it is determined that the locked task is not normally executed.

[0156] In the embodiments of the present disclosure, it is first determined whether the instance node executing the locked task is the current instance node, and in the case that it is, it is further queried whether the locked task exists in the task execution list of the current instance node to determine whether the locked task is normally executed, and in the case that it is not, it is further queried whether the instance node executing the locked task is in the target instance list to determine whether the locked task is normally executed, and then it is accurately determined whether the task in the locked state is normally executed, so that in the case that the task in the locked state is not normally executed, the locked task is unlocked and recovered, thereby improving the reliability of task processing.

[0157] Exemplarily, the task processing method provided by the embodiment of the present disclosure comprises the following steps:

[0158] Step 1), determining whether there is a locked task in a locked state;

[0159] Step 2), in the case that there is a locked task in a locked state, determining, for each locked task, whether the locked task is executed normally;

[0160] Step 3), in the case that the locked task is not executed normally, performing unlocking processing on the locked task, and adding the unlocked task obtained by the unlocking processing to a to-be-executed task list;

[0161] Step 4), in the case that the locked task is executed normally, performing Step 5);

[0162] Step 5), determining whether the current instance node reaches the load balancing upper limit condition;

[0163] Step 6), in the case that the current instance node does not reach the load balancing upper limit condition, taking one to-be-executed task from the to-be-executed task list as a target to-be-executed task;

[0164] Step 7), in the case that the current instance node reaches the load balancing upper limit condition, performing no processing.

[0165] Step 8), performing locking processing on the target to-be-executed task;

[0166] Step 9), in the case that the locking on the target to-be-executed task is successful, obtaining configuration information and a task execution graph of the target to-be-executed task; wherein the configuration information and the task execution graph of the target to-be-executed task are set in advance for the target to-be-executed task, and the task execution graph comprises a target task execution platform for executing the target to-be-executed task;

[0167] Step 10), in the case that the locking on the target to-be-executed task is not successful, returning to Step 6) to take one to-be-executed task from the to-be-executed task list as a target to-be-executed task;

[0168] Step 11), generating a first executable task corresponding to the target to-be-executed task based on the configuration information and the task execution graph of the target to-be-executed task;

[0169] Step 12), converting the first executable task into a second executable task that can be executed by the target task execution platform; wherein the task execution platform comprises a local computing engine, a Spark computing engine and a Flink computing engine;

[0170] Step 13), pushing the second executable task to the target task execution platform;

[0171] Step 14), obtaining a task execution result obtained by executing the second executable task by the target task execution platform.

[0172] In the embodiments of the present disclosure, in the process of executing a task by an instance node, the instance node may directly stop running due to system power failure, system crash, CPU or memory occupation being too high and other factors, so that the task execution fails. The task that fails to execute cannot be unlocked because it is still in execution, and thus is always in a locked state and cannot be executed again. In the case that the locked task is not executed normally, the locked task is unlocked, and the unlocked task obtained by the unlocking is added to a to-be-executed task list, so as to realize automatic recovery and execution of the task and improve the reliability of task processing.

[0173] The stateless instance node automatically performs load balancing of the task, can realize arbitrary scaling of the instance node, and increases the scalability of the task processing while ensuring that all tasks can be loaded by the instance nodes. The instance node ensures that only one instance node can execute one to-be-executed task at any time by locking and preoccupying the task, avoids repeated execution of the task, and thus improves the real-time performance and accuracy of the task execution.

[0174] The configuration information and the task execution graph of the target to-be-executed task are customized in advance, and then the first executable task corresponding to the target to-be-executed task is generated based on the customized configuration information and the task execution graph, so as to realize customized configuration and generation of the task and expand the business scenarios of the task processing. Further, the first executable task is converted into a second executable task that can be executed by the target task execution platform, and the converted second executable task is pushed to the target task execution platform for execution, so as to ensure the flexible scalability of the task computing capability in the task execution process and make the task processing adaptable to different internal execution environments of enterprises.

[0175] Meanwhile, the task is supported to be executed in a local computing engine (local physical machine / virtual machine), a Spark computing engine and a Flink computing engine, so as to ensure the flexible scalability of the task computing capability in the task execution process and make the task processing adaptable to different internal execution environments of enterprises.

[0176] The embodiments of the present disclosure also provide a task processing system, referring to Figure 11 The system comprises:

[0177] The information obtaining module 1101 obtains configuration information of a target to-be-executed task and a task execution graph; the configuration information of the target to-be-executed task and the task execution graph are set in advance for the target to-be-executed task; the task execution graph includes a target task execution platform that executes the target to-be-executed task;

[0178] The task generation module 1102 generates a first executable task corresponding to the target to-be-executed task based on the configuration information of the target to-be-executed task and the task execution graph.

[0179] The task conversion module 1103 converts the first executable task into a second executable task that can be executed by the target task execution platform.

[0180] The task allocation module 1104 pushes the second executable task to the target task execution platform.

[0181] The result obtaining module 1105 obtains a task execution result of the second executable task obtained by the target task execution platform, and the task execution result of the second executable task represents a task execution result of the target to-be-executed task.

[0182] In the embodiments of the present disclosure, the configuration information of the target to-be-executed task and the task execution graph are set in advance for different target to-be-executed tasks, and then the first executable task corresponding to the target to-be-executed task is generated based on the set configuration information and the task execution graph, thereby realizing the custom configuration and generation of the task and expanding the business scenarios of the task processing. Further, the first executable task is converted into the second executable task that can be executed by the target task execution platform, and the converted second executable task is pushed to the target task execution platform for execution, thereby ensuring the flexible scalability of the task computing capability in the task execution process and making the task processing adaptable to different internal execution environments of enterprises.

[0183] In a possible implementation, the system further includes:

[0184] The load balancing module is configured to determine whether a current instance node reaches a load balancing upper limit condition.

[0185] The task determination module is configured to, in a case where the load balancing module determines that the current instance node does not reach the load balancing upper limit condition, take a to-be-executed task from a to-be-executed task list as the target to-be-executed task.

[0186] The task locking module is configured to perform a locking process on the target to-be-executed task, and in a case where the target to-be-executed task is successfully locked, trigger the information obtaining module to perform the obtaining of the configuration information of the target to-be-executed task and the task execution graph.

[0187] In a possible implementation, the load balancing module comprises:

[0188] The first determining sub-module is configured to determine the number of currently effective instance nodes based on a target instance list, wherein the target instance list is obtained by updating self information and corresponding time information of each instance node to the instance list at intervals of a preset time period, and the currently effective instance node indicates that a time difference between the time information of the instance node in the target instance list and a current system time is within a preset time range.

[0189] The second determining sub-module is configured to determine a current total number of tasks, wherein the total number of tasks comprises a number of to-be-executed tasks and a number of tasks being executed.

[0190] The load balancing sub-module is configured to determine whether the current instance node reaches a load balancing upper limit condition based on the number of currently effective instance nodes and the current total number of tasks.

[0191] In a possible implementation, the load balancing sub-module is specifically configured to:

[0192] determine a load upper limit threshold value as a sum of a quotient of the current total number of tasks and the number of currently effective instance nodes and 1;

[0193] determine the number of tasks being executed by the current instance node;

[0194] in a case where the number of tasks being executed by the current instance node is less than the load upper limit threshold value, determine that the current instance node does not reach the load balancing upper limit condition;

[0195] in a case where the number of tasks being executed by the current instance node is not less than the load upper limit threshold value, determine that the current instance node reaches the load balancing upper limit condition.

[0196] In a possible implementation, the task locking module is specifically configured to:

[0197] in a case where a state of a target to-be-executed task is in an unlocked state, update the state of the target to-be-executed task to a locked state.

[0198] In a possible implementation, the system further comprises:

[0199] The task obtaining module is configured to, in a case where the task locking module fails to lock the target to-be-executed task, trigger the task determining module to execute taking one to-be-executed task from the to-be-executed task list as the target to-be-executed task.

[0200] In a possible implementation, the task execution graph of the target to-be-executed task further includes information corresponding to each task execution node that executes the target to-be-executed task, and execution order and connection relationship information of each task execution node; the configuration information of the target to-be-executed task includes attribute information used by each task execution node when executing the target to-be-executed task; and the target task execution platform includes a local computing engine, a Spark computing engine, and a Flink computing engine.

[0201] The task generation module 1102 is specifically configured to: based on the information corresponding to each task execution node, the execution order and connection relationship information of each task execution node, and the attribute information used by each task execution node when executing the target to-be-executed task, instantiate and generate a first executable task corresponding to the target to-be-executed task.

[0202] In a possible implementation, the system further includes:

[0203] The task deletion module is configured to: after obtaining the task execution result of the second executable task, perform unlocking processing on the target to-be-executed task, and delete the target to-be-executed task from the to-be-executed task list.

[0204] In a possible implementation, the system further includes:

[0205] The task state determination module is configured to: determine whether there is a locked task in a locked state.

[0206] The task execution situation determination module is configured to: in a case where the task state determination module determines that there is a locked task in a locked state, for each locked task, determine whether the locked task is normally executed.

[0207] The task recovery module is configured to: in a case where the task execution situation determination module determines that the locked task is not normally executed, perform unlocking processing on the locked task, and add a locked task obtained through the unlocking processing to the to-be-executed task list.

[0208] In a possible implementation, the locked task includes a target identifier of an instance node that executes the locked task; and the determination of whether the locked task is normally executed includes:

[0209] Based on the target identifier, it is determined whether the instance node that executes the locked task is a current instance node.

[0210] In a case where the instance node that executes the locked task is the current instance node, it is determined whether the locked task exists in a task execution list of the current instance node.

[0211] In a case where the locking task exists in the task execution list of the current instance node, it is determined that the locking task is normally executed;

[0212] In a case where the locking task does not exist in the task execution list of the current instance node, it is determined that the locking task is not normally executed;

[0213] In a case where the instance node executing the locking task is not the current instance node, it is determined whether the instance node executing the locking task is in the target instance list;

[0214] In a case where the instance node executing the locking task is in the target instance list, it is determined that the locking task is normally executed;

[0215] In a case where the instance node executing the locking task is not in the target instance list, it is determined that the locking task is not normally executed.

[0216] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information comply with relevant laws and regulations and do not violate public order and good customs. It should be noted that the head model in the present embodiment is not a head model of a specific user and cannot reflect the personal information of a specific user. It should be noted that the two-dimensional face image in the present embodiment comes from a public data set.

[0217] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0218] The electronic device provided by the present disclosure comprises:

[0219] at least one processor; and

[0220] a memory in communication connection with the at least one processor; wherein

[0221] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of the present disclosure.

[0222] The present disclosure provides a non-transitory computer readable storage medium storing computer instructions, wherein the computer instructions are used to make a computer execute the method of any one of the present disclosure.

[0223] The present disclosure provides a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method of any one of the present disclosure.

[0224] Figure 12A schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0225] like Figure 12 As shown, device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1202 or a computer program loaded from storage unit 1208 into random access memory (RAM) 1203. The RAM 1203 may also store various programs and data required for the operation of device 1200. The computing unit 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. Input / output (I / O) interface 1205 is also connected to bus 1204.

[0226] Multiple components in device 1200 are connected to I / O interface 1205, including: input unit 1206, such as keyboard, mouse, etc.; output unit 1207, such as various types of monitors, speakers, etc.; storage unit 1208, such as disk, optical disk, etc.; and communication unit 1209, such as network card, modem, wireless transceiver, etc. Communication unit 1209 allows device 1200 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0227] The computing unit 1201 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, and the like. The computing unit 1201 performs various methods and processes described above, such as the task processing method. For example, in some embodiments, the task processing method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded onto the RAM 1203 and executed by the computing unit 1201, one or more steps of the task processing method described above can be performed. Alternatively, in other embodiments, the computing unit 1201 can be configured to perform the task processing method by any other suitable means, such as by means of firmware.

[0228] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0229] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0230] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0231] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0232] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0233] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0234] It should be understood that the various forms of flow shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure can be achieved, which is not limited herein.

[0235] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A task processing method, comprising: Determine whether the current instance node has reached the load balancing limit. If the current instance node has not reached the load balancing limit, a task to be executed is taken from the list of tasks to be executed as the target task to be executed; The target task to be executed is locked. If the target task to be executed is successfully locked, the configuration information and task execution graph of the target task to be executed are obtained; wherein, the configuration information and task execution graph of the target task to be executed are pre-defined for the target task to be executed; the task execution graph includes: the target task execution platform that executes the target task to be executed; Based on the configuration information of the target task to be executed and the task execution graph, a first executable task corresponding to the target task to be executed is generated; The first executable task is converted into a second executable task that the target task execution platform can execute; The second executable task is pushed to the target task execution platform; Obtain the task execution result obtained by the target task execution platform from executing the second executable task, wherein the task execution result of the second executable task represents the task execution result of the target task to be executed; The step of determining whether the current instance node has reached the load balancing limit includes: Based on the target instance list, the number of currently valid instance nodes is determined; the target instance list is obtained by each instance node updating its own information and corresponding time information to the instance list at preset time intervals; the currently valid instance node means that the time difference between the time information of the instance node in the target instance list and the current system time is within a preset time range. Determine the current total number of tasks, which includes the number of tasks to be executed and the number of tasks currently being executed; The sum of the quotient of the current total number of tasks and the current number of valid instance nodes, and 1, is determined as the load limit threshold. Determine the number of tasks currently being executed on the instance node; If the number of tasks being executed by the current instance node is less than the load limit threshold, it is determined that the current instance node has not reached the load balancing limit condition; If the number of tasks being executed by the current instance node is not less than the load limit threshold, it is determined that the current instance node has reached the load balancing limit condition.

2. The method according to claim 1, wherein, The locking process for the target task to be executed includes: If the target task to be executed is in an unlocked state, update the target task to be executed to a locked state.

3. The method according to claim 1, further comprising: If locking the target task fails, return to execution and retrieve a task from the list of tasks to be executed as the target task.

4. The method according to claim 1, wherein, The task execution graph of the target task to be executed also includes: information corresponding to each task execution node that executes the target task to be executed, as well as the execution order and connection relationship information of each task execution node; the configuration information of the target task to be executed includes: attribute information used by each task execution node when executing the target task to be executed; The step of generating a first executable task corresponding to the target task based on the configuration information and task execution graph includes: Based on the information corresponding to each task execution node, the execution order and connection relationship information of each task execution node, and the attribute information used by each task execution node when executing the target task to be executed, a first executable task corresponding to the target task to be executed is instantiated and generated.

5. The method according to claim 1, wherein, The target task execution platform includes: a local computing engine, a Spark computing engine, and a Flink computing engine.

6. The method according to claim 1, further comprising, after obtaining the task execution result of the second executable task: The target task to be executed is unlocked, and then removed from the list of tasks to be executed.

7. The method according to any one of claims 1-6, further comprising: Determine if there are any locking tasks in a locked state; In the case of locked tasks that are in a locked state, for each locked task, determine whether the locked task is executed normally; If the locking task is not executed normally, the locking task is unlocked, and the unlocked task is added to the list of tasks to be executed.

8. The method according to claim 7, wherein, The locking task includes the target identifier of the instance node that executes the locking task; Determining whether the locking task was executed normally includes: Based on the target identifier, determine whether the instance node executing the locking task is the current instance node; If the instance node executing the locking task is the current instance node, determine whether the locking task exists in the task execution list of the current instance node; If the locking task exists in the task execution list of the current instance node, it is determined that the locking task is executed normally; If the locking task does not exist in the task execution list of the current instance node, it is determined that the locking task has not been executed normally. If the instance node executing the locking task is not the current instance node, determine whether the instance node executing the locking task is in the target instance list; If the instance node executing the locking task is in the target instance list, the locking task is determined to be executed normally. If the instance node executing the locking task is not in the target instance list, it is determined that the locking task was not executed normally.

9. A task processing system, comprising: The load balancing module is used to determine whether the current instance node has reached the load balancing limit. The task determination module is used to select a task to be executed from the list of tasks to be executed as the target task to be executed when the load balancing module determines that the current instance node has not reached the load balancing upper limit condition. The task locking module is used to lock the target task to be executed. The information acquisition module is used to acquire the configuration information and task execution graph of the target task to be executed when the target task to be executed is successfully locked; wherein, the configuration information and task execution graph of the target task to be executed are pre-defined for the target task to be executed; the task execution graph includes: the target task execution platform that executes the target task to be executed; The task generation module is used to generate a first executable task corresponding to the target task based on the configuration information of the target task to be executed and the task execution graph; The task conversion module is used to convert the first executable task into a second executable task that the target task execution platform can execute; The task allocation module is used to push the second executable task to the target task execution platform; The result acquisition module is used to acquire the task execution result obtained by the target task execution platform from executing the second executable task. The task execution result of the second executable task represents the task execution result of the target task to be executed. The load balancing module includes: The first determining submodule is used to determine the number of currently valid instance nodes based on the target instance list; the target instance list is obtained by each instance node updating its own information and corresponding time information to the instance list at preset time intervals; the currently valid instance node means that the time difference between the time information of the instance node in the target instance list and the current system time is within a preset time range. The second determining submodule is used to determine the current total number of tasks, which includes the number of tasks to be executed and the number of tasks currently being executed; The load balancing submodule is used to determine the load limit threshold by summing the quotient of the current total number of tasks and the current number of valid instance nodes with 1; determine the number of tasks currently being executed by the current instance node; if the number of tasks currently being executed by the current instance node is less than the load limit threshold, determine that the current instance node has not reached the load balancing limit condition; if the number of tasks currently being executed by the current instance node is not less than the load limit threshold, determine that the current instance node has reached the load balancing limit condition.

10. The system according to claim 9, wherein, The task execution graph of the target task to be executed also includes: information corresponding to each task execution node that executes the target task to be executed, as well as the execution order and connection relationship information of each task execution node; the configuration information of the target task to be executed includes: attribute information used by each task execution node when executing the target task to be executed; the target task execution platform includes: local computing engine, Spark computing engine and Flink computing engine; The task generation module is specifically used to: instantiate and generate a first executable task corresponding to the target task based on the information corresponding to each task execution node, the execution order and connection relationship information of each task execution node, and the attribute information used by each task execution node when executing the target task.

11. The system according to any one of claims 9-10, further comprising: The task status determination module is used to determine whether there are any locked tasks in a locked state. The task execution status determination module is used to determine whether each locked task is executed normally when the task status determination module determines that there are locked tasks in a locked state. The task recovery module is used to unlock the locked task when the task execution status determination module determines that the locked task has not been executed normally, and to add the unlocked task to the list of tasks to be executed.

12. The system according to claim 11, wherein, The locking task includes the target identifier of the instance node that executes the locking task; Determining whether the locking task was executed normally includes: Based on the target identifier, determine whether the instance node executing the locking task is the current instance node; If the instance node executing the locking task is the current instance node, determine whether the locking task exists in the task execution list of the current instance node; If the locking task exists in the task execution list of the current instance node, it is determined that the locking task is executed normally; If the locking task does not exist in the task execution list of the current instance node, it is determined that the locking task has not been executed normally. If the instance node executing the locking task is not the current instance node, determine whether the instance node executing the locking task is in the target instance list; If the instance node executing the locking task is in the target instance list, the locking task is determined to be executed normally. If the instance node executing the locking task is not in the target instance list, it is determined that the locking task was not executed normally.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images