Task processing method and device, computer equipment and storage medium

Abstract

The invention provides a task processing method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring current time; judging whether the current time reaches preset task scheduling time or not; wherein the task scheduling time is task scheduling time in task configuration content corresponding to a target task pre-configured by a user; if yes, calling a preset main task management node, and judging whether the main task management node is in a normal operation state or not; if yes, task configuration content of the target task is obtained, and a task instance corresponding to the task configuration content is generated based on the main task management node; screening out a target task execution node from a plurality of preset task execution nodes; and sending the execution instruction of the task instance to a target task execution node so as to perform corresponding task execution processing according to the execution instruction of the task instance through the target task execution node. According to the invention, the rationality and intelligence of resource allocation are improved, and the risk of task processing failure is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a task processing method, device, computer equipment and storage medium. Background Art

[0002] With the rapid progress of digital modern information technology, the value of data is being deeply mined and utilized at an unprecedented speed. In the current wave of digitalization, whether in finance, medical care, education or entertainment, a large amount of offline and real-time data is inevitably generated. These data are not only large in scale, but also of various data types, including but not limited to structured data, semi-structured data and unstructured data, providing rich information resources and decision-making basis for all walks of life. In order to effectively respond to the challenges of big data and tap the potential value of data, big data platforms have emerged. These platforms have powerful data processing capabilities and can efficiently process, store and analyze massive data. However, with the continuous expansion of the scale and number of big data platforms, how to efficiently use resources, improve service quality and reduce operation and maintenance costs has become an important issue facing enterprises.

[0003] In order to improve resource utilization and reduce operation and maintenance costs, many companies have begun to adopt the public cluster approach, that is, to allow related business departments to use cluster resources together and manage and schedule these resources in a unified manner. This approach has improved resource utilization efficiency to a certain extent, but it has also brought new problems and challenges.

[0004] Traditional scheduling development platforms have obvious deficiencies in scalability, reliability, and resource utilization. Especially when processing large-scale tasks, these platforms often encounter bottlenecks. Specifically, traditional big data platforms usually integrate resource scheduling and task control functions in a master node. When the task volume reaches a peak, this master node will be under tremendous pressure, resulting in a large amount of resource consumption, low resource utilization, and even the risk of task processing failure. Summary of the invention

[0005] The main purpose of the present invention is to provide a task processing method, apparatus, computer equipment and storage medium, aiming to solve the technical problem of low resource utilization in traditional big data platforms in the prior art.

[0006] To achieve the above object, the present invention provides a task processing method, which comprises:

[0007] Get the current time;

[0008] Determine whether the current time reaches a preset task scheduling time; wherein the task scheduling time is the task scheduling time in the task configuration content corresponding to the target task pre-configured by the user;

[0009] If yes, call the preset main task management node and determine whether the main task management node is in a normal operating state;

[0010] If yes, obtain the task configuration content of the target task, and generate a task instance corresponding to the task configuration content based on the main task management node;

[0011] Filtering a target task execution node from multiple preset task execution nodes;

[0012] The execution instruction of the task instance is sent to the target task execution node, so that the target task execution node performs corresponding task execution processing according to the execution instruction of the task instance.

[0013] Optionally, after determining whether the main task management node is in a normal operating state, the method further includes:

[0014] If the main task management node is in a faulty state, a preset load balancing tool is called;

[0015] Filtering a designated task management node from a plurality of preset backup task management nodes based on a preset filtering strategy;

[0016] Based on the load balancing tool, using the designated task management node to perform failover processing on the main task management node, so as to take over the virtual IP address corresponding to the main task management node through the designated task management node;

[0017] Based on the designated task management node, a task dispatching process corresponding to the task configuration content of the target task is performed.

[0018] Optionally, the step of selecting a designated task management node from a plurality of preset standby task management nodes based on a preset screening strategy includes:

[0019] Obtaining a preset weight value of each of the standby task management nodes;

[0020] Filter out the designated weight value with the largest value from all the preset weight values;

[0021] Acquire a first standby task management node corresponding to the specified weight value from all the standby task management nodes;

[0022] Determine whether the number of the first standby task management nodes is greater than 1;

[0023] If the number of the first standby task management nodes is equal to 1, the first standby task management node is used as the designated task management node.

[0024] Optionally, after determining whether the number of the first standby task management nodes is greater than 1, the method further includes:

[0025] If the number of the first standby task management nodes is greater than 1, obtaining the IP address of each of the first standby task management nodes;

[0026] Compare the IP addresses of the first standby task management nodes and select the designated IP address with the largest value;

[0027] Acquire a second standby task management node corresponding to the designated IP from all the first standby task management nodes;

[0028] The second standby task management node is used as the designated task management node.

[0029] Optionally, before determining whether the current time reaches a preset task scheduling time, the method further includes:

[0030] Determining whether a task configuration request triggered by the user is received;

[0031] If yes, display the preset task configuration interface;

[0032] Receiving basic configuration data, alarm configuration data, and extended configuration data corresponding to the target task input by the user in the task configuration interface;

[0033] Generate task configuration content of the target task based on the basic configuration data, the alarm configuration data and the extended configuration data;

[0034] The task configuration content is associated with the target task and stored.

[0035] Optionally, after sending the execution instruction of the task instance to the target task execution node, the method further includes:

[0036] Determine whether a data query request corresponding to the target sensitive data triggered by a specified user is received;

[0037] If so, obtaining user information of the specified user;

[0038] Acquire a specified permission level of the specified user based on the user information;

[0039] Obtaining a query permission level interval corresponding to the target sensitive data;

[0040] Determining whether the specified authority level is within the query authority level range;

[0041] If so, obtain the target sensitive data and display the target sensitive data.

[0042] Optionally, after sending the execution instruction of the task instance to the target task execution node, the method further includes:

[0043] After the target task execution node completes the task execution processing, receiving task status information corresponding to the task instance reported by the target task execution node;

[0044] Based on the task status information, the task information in the preset database is updated;

[0045] Based on the task status information, a status update process corresponding to the task instance is performed in a preset target interface.

[0046] In addition, to achieve the above-mentioned purpose, the present invention further provides a task processing device, the task processing device comprising:

[0047] The first acquisition module is used to obtain the current time;

[0048] A first judgment module is used to judge whether the current time reaches a preset task scheduling time; wherein the task scheduling time is the task scheduling time in the task configuration content corresponding to the target task pre-configured by the user;

[0049] The second judgment module is used to call a preset main task management node and judge whether the main task management node is in a normal operating state;

[0050] A first generating module, configured to obtain the task configuration content of the target task and generate a task instance corresponding to the task configuration content based on the main task management node;

[0051] A first screening module, used to screen out a target task execution node from a plurality of preset task execution nodes;

[0052] The sending module is used to send the execution instruction of the task instance to the target task execution node, so that the target task execution node performs corresponding task execution processing according to the execution instruction of the task instance.

[0053] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0054] The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the task processing methods proposed in the embodiments of the present application are implemented.

[0055] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0056] The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of any one of the task processing methods proposed in the embodiments of the present application are implemented.

[0057] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0058] The present invention provides a task processing method, device, computer equipment and storage medium, the method comprising: first obtaining the current time; then judging whether the current time reaches the preset task scheduling time; wherein the task scheduling time is the task scheduling time in the task configuration content corresponding to the target task pre-configured by the user; if so, calling the preset main task management node, and judging whether the main task management node is in a normal operating state; if so, obtaining the task configuration content of the target task, and generating a task instance corresponding to the task configuration content based on the main task management node; subsequently filtering out the target task execution node from a plurality of preset task execution nodes; finally sending the execution instruction of the task instance to the target task execution node, so that the target task execution node performs corresponding task execution processing according to the execution instruction of the task instance. When the present invention detects that the current time reaches the task scheduling time in the task configuration content corresponding to the target task pre-configured by the user, and determines that the called main task management node is in a normal operating state, the task configuration content of the target task is automatically and intelligently obtained, and a task instance corresponding to the task configuration content is generated based on the main task management node, and then the target task execution node is screened out from the preset multiple task execution nodes, and then the execution instruction of the task instance is sent to the target task execution node, so that the target task execution node performs the corresponding task execution processing according to the execution instruction of the task instance. The present invention decouples the functions on the main node of the big data platform to separate them into a task management node and a task execution node, and then based on the combined use of the task management node and the task execution node, it can realize the efficient and accurate completion of the task processing of the target task configured by the user, effectively improves the intelligence of task processing, greatly reduces the resource consumption of the main node, improves resource utilization, improves the rationality and intelligence of resource allocation, and reduces the risk of task processing failure. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0061] Figure 2 is a flowchart of a task processing method provided by an embodiment of the present invention;

[0062] Figure 3 is a structural schematic diagram of an embodiment of a task processing device according to the present application;

[0063] Figure 4 This is a basic structural block diagram of the computer device in this embodiment. DETAILED DESCRIPTION

[0064] The task processing method provided by the embodiment of the present invention is applied to the task processing device. Unless otherwise defined, all technical and scientific terms used in this article have the same meaning as those generally understood by technicians in the technical field of this application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned drawings and any variations thereof are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0065] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0066] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0067] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0068] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social online platform software, etc.

[0069] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0070] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0071] It should be noted that the task processing method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the task processing device is generally set in the server / terminal device.

[0072] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0073] With the rapid progress of digital modern information technology, the value of data is being deeply mined and utilized at an unprecedented speed. In the current wave of digitalization, whether in finance, medical care, education or entertainment, a large amount of offline and real-time data is inevitably generated. These data are not only large in scale, but also of various data types, including but not limited to structured data, semi-structured data and unstructured data, providing rich information resources and decision-making basis for all walks of life. In order to effectively respond to the challenges of big data and tap the potential value of data, big data platforms have emerged. These platforms have powerful data processing capabilities and can efficiently process, store and analyze massive data. However, with the continuous expansion of the scale and number of big data platforms, how to efficiently use resources, improve service quality and reduce operation and maintenance costs has become an important issue facing enterprises.

[0074] In order to improve resource utilization and reduce operation and maintenance costs, many companies have begun to adopt the public cluster approach, that is, to allow related business departments to use cluster resources together and manage and schedule these resources in a unified manner. This approach has improved resource utilization efficiency to a certain extent, but it has also brought new problems and challenges.

[0075] Traditional scheduling development platforms have obvious deficiencies in scalability, reliability, and resource utilization. Especially when processing large-scale tasks, these platforms often encounter bottlenecks. Specifically, traditional big data platforms usually integrate resource scheduling and task control functions in a master node. When the task volume reaches a peak, this master node will be under tremendous pressure, resulting in a large amount of resource consumption, low resource utilization, and even the risk of task processing failure.

[0076] Continue to refer Figure 2 , shows a flow chart of an embodiment of the task processing method proposed in the present application. The embodiment of the present application can acquire and process relevant data based on artificial intelligence technology.

[0077] The task processing method provided by the embodiment of the present invention comprises the following steps:

[0078] S210, obtaining the current time.

[0079] In this step, the execution subject of the present invention may be an offline big data scheduling platform, or an offline big data system (referred to as a system). The current time may be obtained in the command prompt by using the time / T command.

[0080] S220, determining whether the current time reaches a preset task scheduling time.

[0081] In this step, the task scheduling time is the task scheduling time in the task configuration content corresponding to the target task pre-configured by the user. The current time obtained can be compared with the task scheduling time to determine whether the current time reaches the task scheduling time.

[0082] S230: If yes, call the preset main task management node and determine whether the main task management node is in normal operation.

[0083] In this step, the above-mentioned main task management node refers to falcon-base. The present invention decouples the functions on the main node (master node) of the traditional big data platform, that is, the responsibilities are separated into a resource management component, namely, the task management node falcon-base and the task scheduling component, namely, the task execution component falcon-runner. Falcon-base is mainly responsible for managing the calculation of resources and the distribution of tasks. The state of the child process is judged by the heartbeat of falcon-runner, which reduces the single point failure risk of the main node, reduces the resource consumption of the master, and the resource consumption is in units of memory and CPU, and the resource allocation is more reasonable. This design greatly reduces the resource consumption of the master node, and the resource allocation is also more reasonable. The basic unit of task scheduling is the instance of the task, and the task information created by the user can be understood as a template for instance execution. Among them, the above-mentioned main task management node can be monitored by using a load balancing tool to see if it is in a normal operating state. In addition, the task management node supports the issuance of tasks of different priorities, and gives priority to the issuance of core link tasks. One task type corresponds to a runner process.

[0084] S240: If yes, obtain the task configuration content of the target task, and generate a task instance corresponding to the task configuration content based on the main task management node.

[0085] In this step, when it is detected that the current time reaches the above-mentioned task scheduling time and the main task management node is in normal operation, the task management node (falcon-base) will create a corresponding task instance according to the task details of the target task configured by the user (i.e., the task configuration content). Among them, the task instance is the concretization of the task execution, which contains all the key information required to execute the task, such as task type, data source, target, SQL statement, etc., which are the basis for task execution. In addition, the creation time of the task instance is the first day of the scheduling cycle, such as the monthly task at 12:00 a.m. on the 1st of each month, and the weekly task at 12:00 a.m. on Sunday. The task instance with a status of ready is generated and waits until the start time set by the user, that is, the task scheduling time.

[0086] S250: Filter out a target task execution node from a plurality of preset task execution nodes.

[0087] In this step, according to the task dependency and priority of the target task, the main task management node (falcon-base) selects a node with sufficient resources (sufficient CPU and memory) from multiple preset task execution nodes (falcon-runner) through polling as the above-mentioned target task execution node. Among them, multiple task execution nodes are pre-built, and the task execution node is used to receive the task issued by the task management node, execute the task and feedback the task execution result.

[0088] Each task type is a runner process. For derivative tasks, falcon-runner generates the properties configuration file required for derivatives based on the derivative template and user configuration. The cheetah process is started from the falcon-runner process, and the template and properties configuration file are used as startup parameters, and cheetah executes the derivative task.

[0089] For Hive Task and Python Task, Falcon-runner generates Python scripts in the background, splices parameters through Falcon-runner, generates Python script execution commands, starts Python scripts, obtains Hive links from Zookeeper, and submits to Yarn to execute SQL. Shell Task is spliced ​​into shell to start execution. Shell Task is mainly used to execute Spark tasks.

[0090] S260, sending the execution instruction of the task instance to the target task execution node, so that the target task execution node performs corresponding task execution processing according to the execution instruction of the task instance.

[0091] In this step, after determining the target task execution node, the main task management node (falcon-base) will pass the specific execution instructions of the created task instance to the target task execution node. This process is a key link in task execution, which ensures that the information in the task instance can be correctly passed to the target task execution node, and the target task execution node executes the task according to this information. Among them, the target task execution node executes the task and feedbacks the task execution status by accepting the execution instructions of the task instance issued by the main task management node (falcon-base).

[0092] For derivative tasks, the target task execution node will start the Cheetah module, generate the configuration file required for the derivative according to the template and user configuration, and then execute the derivative task. By directly template-izing the extraction task configuration, Cheetah executes the logic plug-in. Users only need to configure parameters on the front end, including the data source, data target, and structured query statements for extracting data. Cheetah supports a variety of persistence methods, including HDFS, MySQL, HBase, Elasticsearch, etc. In addition, the Cheetah module that executes the derivative task. Deployed on the Falcon-Runner node and started by Falcon-Runner. The data source and target source of the derivative are determined according to the configuration file generated by Falcon-Runner. The data source and target source services support user-defined configuration.

[0093] In addition, during the task execution process, the target task execution node will periodically send heartbeat and task execution status information to the main task management node. After the target task execution node is started, it will register with the main task management node. The target task execution node will periodically communicate with the main task management node through sockets to transmit task status information during task execution and report resource information such as memory and CPU of the current node. The task information uses the protoBuf storage format, and finally the target task execution node starts the corresponding application to start executing the task.

[0094] In an embodiment of the present invention, the current time is first obtained; then it is determined whether the current time reaches the preset task scheduling time; wherein the task scheduling time is the task scheduling time in the task configuration content corresponding to the target task pre-configured by the user; if so, the preset main task management node is called, and it is determined whether the main task management node is in a normal operating state; if so, the task configuration content of the target task is obtained, and a task instance corresponding to the task configuration content is generated based on the main task management node; subsequently, the target task execution node is filtered out from the preset multiple task execution nodes; finally, the execution instruction of the task instance is sent to the target task execution node, so that the target task execution node performs corresponding task execution processing according to the execution instruction of the task instance. When the present invention detects that the current time reaches the task scheduling time in the task configuration content corresponding to the target task pre-configured by the user, and determines that the called main task management node is in a normal operating state, the task configuration content of the target task is automatically and intelligently obtained, and a task instance corresponding to the task configuration content is generated based on the main task management node, and then the target task execution node is screened out from the preset multiple task execution nodes, and then the execution instruction of the task instance is sent to the target task execution node, so that the target task execution node performs the corresponding task execution processing according to the execution instruction of the task instance. The present invention decouples the functions on the main node of the big data platform to separate them into a task management node and a task execution node, and then based on the combined use of the task management node and the task execution node, it can realize the efficient and accurate completion of the task processing of the target task configured by the user, effectively improves the intelligence of task processing, greatly reduces the resource consumption of the main node, improves resource utilization, improves the rationality and intelligence of resource allocation, and reduces the risk of task processing failure.

[0095] Optionally, after determining whether the main task management node is in a normal operating state, the method further includes:

[0096] If the main task management node is in a faulty state, a preset load balancing tool is called.

[0097] In this step, the above-mentioned load balancing tool can specifically use keepalived. Keepalived is a tool for achieving high availability and load balancing of servers. It uses VRRP (Virtual Router Redundancy Protocol) protocol to implement virtual routing between multiple physical routing devices. Keepalived can monitor the status of the server and automatically failover when a fault is detected. Based on keepalived, the high availability of falcon-base is achieved, which is out of business and has efficient switching speed, fully ensuring the stability of the offline big data scheduling platform.

[0098] Specifically, the system uses keepalived to achieve high availability of the task management node (falcon-base) through master-slave switching. Multiple physical routing devices are virtualized into a virtual router. Only the master provides services to the outside at the same time. When the master router fails, the backup router will re-elect and generate a new master to continue to provide services. Since users use the virtual IP of the virtual router, the entire switching process is imperceptible to users. Falcon-base is separated from the business layer to achieve high availability, with fast switching speed, effectively ensuring service continuity.

[0099] A designated task management node is selected from a plurality of preset standby task management nodes based on a preset screening strategy.

[0100] In this step, the specific implementation process of selecting the designated task management node from the preset multiple backup task management nodes based on the preset screening strategy will be further described in detail in the subsequent specific embodiments of the present invention and will not be elaborated on here.

[0101] Based on the load balancing tool, the designated task management node is used to perform failover processing on the main task management node, so that the virtual IP address corresponding to the main task management node is taken over by the designated task management node.

[0102] In this step, the load balancing tool is used to achieve high availability between task management nodes (including the main task management node and the backup task management node). It virtualizes multiple physical routing devices into a virtual router through the VRRP protocol, and ensures that it can quickly switch to the backup node, that is, the designated task management node, when the main task management node fails. Keepalived is also responsible for monitoring the status of the node and triggering the failover process when a failure is detected.

[0103] Specifically, the load balancing tool, keepalived, configures multiple physical routing devices (here, multiple falcon-base nodes, including the main task management node and the standby task management node) to virtualize into a virtual router. The virtual router has a virtual IP address, through which users access the system. Under normal circumstances, only the main task management node provides services to the outside world and holds a virtual IP address. When the main task management node fails, keepalived will detect this and trigger the failover process. During the failover process, the standby task management node will compete to become the new main task management node, that is, the designated task management node mentioned above, and take over the virtual IP address and provide services. The entire switching process is imperceptible to the user because the user always accesses the system through the virtual IP address.

[0104] In addition, the main task management node will periodically send broadcasts to other backup task management nodes, which is equivalent to sending heartbeats. If the backup does not receive the heartbeat from the main task management node within the specified time, the backup will compete for the new main task management node by comparing the weights. If the backup wins the election, the virtual IP will be bound to the new main task management node r. The user is not aware of it, it is separated from the business layer, and the switching process is fast.

[0105] Based on the designated task management node, a task dispatching process corresponding to the task configuration content of the target task is performed.

[0106] In this step, the task dispatching process corresponding to the task configuration content of the target task based on the designated task management node may refer to the processing steps of steps S240-S260, which will not be described in detail here.

[0107] In this embodiment, if it is detected that the main task management node is in a faulty state, a preset load balancing tool is called; then, based on a preset screening strategy, a designated task management node is screened out from a preset plurality of backup task management nodes; then, based on the load balancing tool, the designated task management node is used to perform a failover process on the main task management node, so that the designated task management node takes over the virtual IP address corresponding to the main task management node; and subsequently, based on the designated task management node, a task dispatch process corresponding to the task configuration content of the target task is performed. When the present invention detects that the main task management node is in a faulty state, the designated task management node is screened out from a preset plurality of backup task management nodes based on the use of a screening strategy, and then, based on the load balancing tool, the designated task management node is used to perform a failover process on the main task management node, so that the designated task management node takes over the virtual IP address corresponding to the main task management node, thereby achieving high availability between the main task management node and the backup task management node, and ensuring the stability and reliability of the system.

[0108] Optionally, the step of selecting a designated task management node from a plurality of preset standby task management nodes based on a preset screening strategy includes:

[0109] Obtaining a preset weight value of each of the standby task management nodes.

[0110] In this step, when each standby task management node is started, different weight values ​​of 0 to 255 are set for each standby task management node in advance according to actual business needs. The larger the weight value of the node, the more likely it is that the node will compete to become the new main task management node during the failover process.

[0111] A specified weight value with the largest value is selected from all the preset weight values.

[0112] In this step, the preset weight values ​​of all standby task management nodes may be numerically compared to select the designated weight value with the largest value.

[0113] A first standby task management node corresponding to the specified weight value is obtained from all the standby task management nodes.

[0114] In this step, the first standby task management node refers to a node that matches the specified weight value among all standby task management nodes.

[0115] Determine whether the number of the first standby task management nodes is greater than 1.

[0116] In this step, the number of the first standby task management nodes may be 1 or greater than 1.

[0117] If the number of the first standby task management nodes is equal to 1, the first standby task management node is used as the designated task management node.

[0118] In this step, if it is detected that the number of the first standby task management nodes is equal to 1, the first standby task management node is directly used as the designated task management node.

[0119] In this embodiment, the preset weight values ​​of each of the backup task management nodes are obtained; then the designated weight value with the largest value is screened out from all the preset weight values; then the first backup task management node corresponding to the designated weight value is obtained from all the backup task management nodes; then it is determined whether the number of the first backup task management nodes is greater than 1; if the number of the first backup task management nodes is equal to 1, the first backup task management node is used as the designated task management node. The present invention obtains the preset weight values ​​of each of the backup task management nodes, and screens out the designated weight value with the largest value from all the preset weight values, then obtains the first backup task management node corresponding to the designated weight value from all the backup task management nodes, and when it is detected that the number of the first backup task management nodes is equal to 1, the first backup task management node is automatically used as the designated task management node, effectively ensuring the accuracy of the designated task management nodes obtained by screening.

[0120] Optionally, after determining whether the number of the first standby task management nodes is greater than 1, the method further includes:

[0121] If the number of the first standby task management nodes is greater than 1, the IP address of each of the first standby task management nodes is obtained.

[0122] In this step, if the number of first backup task management nodes is greater than 1, that is, the preset weight values ​​of each first backup task management node are the same, then the node IP comparison needs to be continued. The larger the IP, the higher the weight. That is, the larger the node IP, the more likely it is that the node will compete to become the new main task management node during the failover process.

[0123] The IP addresses of the first standby task management nodes are compared to select the designated IP address with the largest value.

[0124] In this step, the IPs of the first standby task management nodes may be compared to select the designated IP with the largest value from the IPs of all the first standby task management nodes.

[0125] Acquire a second standby task management node corresponding to the designated IP from all the first standby task management nodes.

[0126] In this step, the second standby task management node refers to a node in the first standby task management node that matches the designated IP.

[0127] The second standby task management node is used as the designated task management node.

[0128] In this embodiment, if it is detected that the number of the first backup task management nodes is greater than 1, the IP of each of the first backup task management nodes is obtained; then the IPs of each of the first backup task management nodes are compared to screen out the designated IP with the largest value; then the second backup task management node corresponding to the designated IP is obtained from all of the first backup task management nodes; and the second backup task management node is subsequently used as the designated task management node. When the present invention detects that the number of the first backup task management nodes is greater than 1, the IP of each of the first backup task management nodes is further obtained, and the IP of each of the first backup task management nodes is compared to screen out the designated IP with the largest value, and the second backup task management node corresponding to the designated IP is subsequently obtained from all of the first backup task management nodes and used as the designated task management node, thereby effectively ensuring the accuracy and intelligence of the designated task management node obtained by screening.

[0129] Optionally, before determining whether the current time reaches a preset task scheduling time, the method further includes:

[0130] Determine whether the task configuration request triggered by the user is received.

[0131] In this step, the task configuration request is a request triggered by a user for performing task visualization configuration.

[0132] If yes, display the preset task configuration interface.

[0133] In this step, the task configuration interface is a pre-built web front-end interface for assisting users in visually configuring tasks. The system provides the function of configuring task attributes through the front-end web interface. Based on the web front-end interface, users can be provided with visual configuration of tasks, which is simple to operate, has low learning cost, and simple task configuration, greatly saving development cost and operation and maintenance cost.

[0134] The basic configuration data, the alarm configuration data and the extended configuration data corresponding to the target task input by the user in the task configuration interface are received.

[0135] In this step, the user can input the basic configuration data, alarm configuration data and extended configuration data corresponding to the target task in the above-mentioned task configuration interface according to the actual task configuration requirements. Among them, the above-mentioned basic configuration data may include data such as task scheduling time, scheduling cycle, task dependency, task responsible person, etc. The above-mentioned alarm configuration data may include notification methods (such as Feishu push, email, SMS, phone, etc.) when abnormal situations such as task execution failure or timeout occur. The above-mentioned extended configuration data may include supplementary configurations related to the task type, such as SQL statements for executing statistical tasks or data sources (source) and targets (target) for derivative tasks. Among them, users only need to configure tasks according to business needs and write core logic code (mostly SQL) to minimize coding costs.

[0136] In addition, the system also has an automatic verification function for task dependencies and a wealth of alarm functions, such as circular dependency alarms, task timeout alarms, etc. It supports multiple alarm prompt methods, not limited to Feishu push, email, SMS, phone calls, etc.

[0137] In addition, the system supports functions such as user-defined UDF functions to upload HDFS and Excel file data to Hive tables, etc. It also supports a variety of task types to fully meet user storage and statistical needs.

[0138] The task configuration content of the target task is generated based on the basic configuration data, the alarm configuration data and the extended configuration data.

[0139] In this step, the above basic configuration data, alarm configuration data and extended configuration data may be integrated to obtain corresponding integrated data, and the integrated data may be used as the task configuration content of the target task.

[0140] The task configuration content is associated with the target task and stored.

[0141] In this step, the generated task configuration content is stored in association with the target task to ensure the correspondence between the task configuration content and the target task, thereby completing the task configuration processing for the target task.

[0142] In this embodiment, it is determined whether the task configuration request triggered by the user is received; if so, a preset task configuration interface is displayed; then the basic configuration data, alarm configuration data and extended configuration data corresponding to the target task input by the user in the task configuration interface are received; then the task configuration content of the target task is generated based on the basic configuration data, the alarm configuration data and the extended configuration data; and the task configuration content is subsequently stored in association with the target task. After receiving the task configuration request triggered by the user, the present invention will intelligently receive the basic configuration data, alarm configuration data and extended configuration data corresponding to the target task input by the user based on the use of the task configuration interface, and then generate the task configuration content of the target task based on the basic configuration data, the alarm configuration data and the extended configuration data. Subsequently, by storing the task configuration content in association with the target task, it is possible to automatically and accurately complete the task configuration processing that adapts to the user's personal needs, thereby improving the processing intelligence and processing standardization of the task configuration, and is conducive to improving the user's experience.

[0143] Optionally, after sending the execution instruction of the task instance to the target task execution node, the method further includes:

[0144] Determine whether a data query request corresponding to the target sensitive data is received and triggered by a specified user.

[0145] In this step, the data query request is a request triggered by a specified user to query the target sensitive data in the offline big data system. The system can meet the interactive query needs of users and support multi-engine execution, including MapReduce, Tez, etc. It also provides data download and data permission management functions, and achieves permission isolation and download restrictions for sensitive data.

[0146] If so, obtain the user information of the specified user.

[0147] In this step, the user information may include the user name of the designated user.

[0148] A designated authority level of the designated user is obtained based on the user information.

[0149] In this step, the user information can be used to query the preset user authority data table to obtain the specified authority level of the specified user. The user authority data table is a pre-constructed data table that stores a one-to-one mapping relationship between the name information of each business person and the authority level.

[0150] Obtain the query permission level range corresponding to the target sensitive data.

[0151] In this step, according to the actual permission isolation requirements for sensitive data, query permission level intervals corresponding to different sensitive data are pre-set.

[0152] Determine whether the specified authority level is within the query authority level range.

[0153] In this step, the above-mentioned specified authority level can be compared with the two endpoint values ​​included in the above-mentioned query authority level range to obtain a corresponding numerical comparison result, and based on the numerical comparison result, it can be identified whether the above-mentioned specified authority level is within the above-mentioned query authority level range.

[0154] If so, obtain the target sensitive data and display the target sensitive data.

[0155] In this step, if the above-mentioned designated permission level is within the above-mentioned query permission level range, it is determined that the designated user has the permission to view the above-mentioned target sensitive data, and then the target sensitive data will be obtained, and the target sensitive data will be displayed for the user to view. In addition, if the above-mentioned designated permission level is not within the above-mentioned query permission level range, it is determined that the designated user does not have the permission to view the above-mentioned target sensitive data, and then the data query request corresponding to the target sensitive data triggered by the designated user will be refused to respond, and the corresponding content will be returned to the designated user as a reminder message that the relevant data cannot be viewed due to insufficient permissions.

[0156] In this embodiment, it is determined whether a data query request corresponding to the target sensitive data is received and triggered by a designated user; if so, the user information of the designated user is obtained; then the designated permission level of the designated user is obtained based on the user information; then the query permission level interval corresponding to the target sensitive data is obtained; subsequently, it is determined whether the designated permission level is within the query permission level interval; if so, the target sensitive data is obtained and the target sensitive data is displayed and processed. After receiving a data query request corresponding to the target sensitive data triggered by a designated user, the present invention will obtain the corresponding designated permission level according to the user information of the designated user, and then determine whether the designated permission level is within the query permission level interval corresponding to the target sensitive data. Only when it is detected that the designated permission level is within the query permission level interval will the data query request be responded to later. By obtaining the target sensitive data and displaying it, the processing standardization and processing accuracy of the data query request for sensitive data are effectively ensured, and the response security of the data query request for sensitive data is improved.

[0157] Optionally, after sending the execution instruction of the task instance to the target task execution node, the method further includes:

[0158] After the target task execution node completes the task execution process, task status information corresponding to the task instance reported by the target task execution node is received.

[0159] In this step, after the above task execution processing, that is, after the target task is successfully executed, the target task execution node will report the task status information corresponding to the above task instance to the main task management node. Exemplarily, the task status information may include the execution result (such as success), the execution time (such as 1 am every day to the specific time when the task is completed, and the task status (such as completed).

[0160] The task information in a preset database is updated based on the task status information.

[0161] In this step, the database is a pre-built library that records the task information of each task. According to the received task status information, the task information of the target task corresponding to the task instance in the database can be automatically updated accordingly, thereby ensuring the accuracy of the task information of the target task recorded in the database.

[0162] Based on the task status information, a status update process corresponding to the task instance is performed in a preset target interface.

[0163] In this step, the target interface may be a pre-built web interface that displays the task status of the target task corresponding to the task instance. The task status of the target task corresponding to the task instance in the target interface may be automatically updated according to the received task status information, that is, the task status of the target task is updated to completion, thereby ensuring the accuracy of the task status of the target task displayed in the target interface.

[0164] In this embodiment, after the target task execution node completes the task execution processing, the task status information corresponding to the task instance reported by the target task execution node is received; the task information in a preset database is subsequently updated based on the task status information; and based on the task status information, a status update processing corresponding to the task instance is performed in a preset target interface. After the target task execution node completes the task execution processing, when the task status information corresponding to the task instance reported by the target task execution node is received, the present invention automatically and intelligently updates the task information in a preset database according to the obtained task status information, and performs a status update processing corresponding to the task instance in a preset target interface, effectively ensuring the accuracy of the task information of the target task recorded in the database, and ensuring the accuracy of the task status of the target task displayed in the target interface.

[0165] In some optional implementations, the user information obtained is subject to the user's consent and complies with relevant laws and policies.

[0166] Further references Figure 3 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a task processing device 300, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0167] An embodiment of the present invention provides a task processing device 300, and the task processing device 300 includes:

[0168] A first acquisition module 310, used to acquire the current time;

[0169] The first judgment module 320 is used to judge whether the current time reaches the preset task scheduling time; wherein the task scheduling time is the task scheduling time in the task configuration content corresponding to the target task pre-configured by the user;

[0170] The second judgment module 330 is used to call a preset main task management node and judge whether the main task management node is in a normal operating state;

[0171] A first generating module 340 is used to obtain the task configuration content of the target task and generate a task instance corresponding to the task configuration content based on the main task management node;

[0172] A first screening module 350 is used to screen out a target task execution node from a plurality of preset task execution nodes;

[0173] The sending module 360 ​​is used to send the execution instruction of the task instance to the target task execution node, so that the target task execution node performs corresponding task execution processing according to the execution instruction of the task instance.

[0174] Optionally, the task processing device 300 further includes:

[0175] A calling module, used for calling a preset load balancing tool if the main task management node is in a fault state;

[0176] A second screening module, used to screen out a designated task management node from a plurality of preset standby task management nodes based on a preset screening strategy;

[0177] A first processing module is used to perform failover processing on the main task management node using the designated task management node based on the load balancing tool, so as to take over the virtual IP address corresponding to the main task management node through the designated task management node;

[0178] The second processing module is used to perform task issuance processing corresponding to the task configuration content of the target task based on the designated task management node.

[0179] Optionally, the second screening module includes:

[0180] A first acquisition submodule is used to acquire a preset weight value of each of the standby task management nodes;

[0181] A first screening submodule, used to screen out a designated weight value with the largest value from all the preset weight values;

[0182] A second acquisition submodule, used to acquire a first standby task management node corresponding to the specified weight value from all the standby task management nodes;

[0183] A judging submodule, used to judge whether the number of the first standby task management nodes is greater than 1;

[0184] The first determining submodule is configured to use the first backup task management node as the designated task management node if the number of the first backup task management nodes is equal to 1.

[0185] Optionally, the second screening module further includes:

[0186] a third acquisition submodule, configured to acquire an IP address of each of the first standby task management nodes if the number of the first standby task management nodes is greater than 1;

[0187] A second screening submodule is used to compare the IP addresses of the first standby task management nodes and screen out the designated IP address with the largest value;

[0188] A fourth acquisition submodule, used to acquire a second standby task management node corresponding to the designated IP from all the first standby task management nodes;

[0189] The second determining submodule is used to use the second standby task management node as the designated task management node.

[0190] Optionally, the task processing device 300 further includes:

[0191] A third judgment module is used to judge whether the task configuration request triggered by the user is received;

[0192] A display module, used to display the preset task configuration interface;

[0193] A first receiving module, configured to receive basic configuration data, alarm configuration data, and extended configuration data corresponding to a target task input by the user in the task configuration interface;

[0194] A second generating module, configured to generate task configuration content of the target task based on the basic configuration data, the alarm configuration data and the extended configuration data;

[0195] The storage module is used to associate and store the task configuration content with the target task.

[0196] Optionally, the task processing device 300 further includes:

[0197] A fourth determination module is used to determine whether a data query request corresponding to target sensitive data triggered by a designated user is received;

[0198] A second acquisition module, configured to acquire user information of the designated user;

[0199] A third acquisition module, used to acquire the specified permission level of the specified user based on the user information;

[0200] A fourth acquisition module, used to acquire a query permission level interval corresponding to the target sensitive data;

[0201] A fifth judgment module, used to judge whether the specified authority level is within the query authority level range;

[0202] The third processing module is used to obtain the target sensitive data and display the target sensitive data.

[0203] Optionally, the task processing device 300 further includes:

[0204] A second receiving module is used to receive task status information corresponding to the task instance reported by the target task execution node after the target task execution node completes the task execution processing;

[0205] A first updating module, used for updating the task information in a preset database based on the task status information;

[0206] The second updating module is used to perform a status update process corresponding to the task instance in a preset target interface based on the task status information.

[0207] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0208] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0209] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0210] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as the program code of the task processing method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0211] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the program code stored in the memory 41 or process data, such as running the program code of the task processing method.

[0212] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0213] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores the application crash processing program, and the application crash processing program can be executed by at least one processor to enable the at least one processor to perform the steps of the task processing method as described above.

[0214] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware online platform, and of course, by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0215] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0216] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A task processing method, characterized in that: include: Get the current time; Determine whether the current time reaches a preset task scheduling time; wherein the task scheduling time is the task scheduling time in the task configuration content corresponding to the target task pre-configured by the user; If yes, call the preset main task management node and determine whether the main task management node is in a normal operating state; If yes, obtain the task configuration content of the target task, and generate a task instance corresponding to the task configuration content based on the main task management node; Filtering a target task execution node from multiple preset task execution nodes; The execution instruction of the task instance is sent to the target task execution node, so that the target task execution node performs corresponding task execution processing according to the execution instruction of the task instance.

2. The method according to claim 1, characterized in that After determining whether the main task management node is in a normal operating state, the method further includes: If the main task management node is in a faulty state, a preset load balancing tool is called; Filtering a designated task management node from a plurality of preset backup task management nodes based on a preset filtering strategy; Based on the load balancing tool, using the designated task management node to perform failover processing on the main task management node, so as to take over the virtual IP address corresponding to the main task management node through the designated task management node; Based on the designated task management node, a task dispatching process corresponding to the task configuration content of the target task is performed.

3. The method according to claim 2, characterized in that The step of selecting a designated task management node from a plurality of preset standby task management nodes based on a preset screening strategy includes: Obtaining a preset weight value of each of the standby task management nodes; Filter out the designated weight value with the largest value from all the preset weight values; Acquire a first standby task management node corresponding to the specified weight value from all the standby task management nodes; Determine whether the number of the first standby task management nodes is greater than 1; If the number of the first standby task management nodes is equal to 1, the first standby task management node is used as the designated task management node.

4. The method according to claim 3, characterized in that: After determining whether the number of the first standby task management nodes is greater than 1, the method further includes: If the number of the first standby task management nodes is greater than 1, obtaining the IP address of each of the first standby task management nodes; Compare the IP addresses of the first standby task management nodes and select the designated IP address with the largest value; Acquire a second standby task management node corresponding to the designated IP from all the first standby task management nodes; The second standby task management node is used as the designated task management node.

5. The method according to claim 1, characterized in that Before determining whether the current time reaches the preset task scheduling time, the method further includes: Determining whether a task configuration request triggered by the user is received; If yes, display the preset task configuration interface; Receiving basic configuration data, alarm configuration data, and extended configuration data corresponding to the target task input by the user in the task configuration interface; Generate task configuration content of the target task based on the basic configuration data, the alarm configuration data and the extended configuration data; The task configuration content is associated with the target task and stored.

6. The method according to claim 1, characterized in that After sending the execution instruction of the task instance to the target task execution node, the method further includes: Determine whether a data query request corresponding to the target sensitive data triggered by a specified user is received; If so, obtaining user information of the specified user; Acquire a specified permission level of the specified user based on the user information; Obtaining a query permission level interval corresponding to the target sensitive data; Determining whether the specified authority level is within the query authority level range; If so, obtain the target sensitive data and display the target sensitive data.

7. The method according to claim 1, characterized in that After sending the execution instruction of the task instance to the target task execution node, the method further includes: After the target task execution node completes the task execution processing, receiving task status information corresponding to the task instance reported by the target task execution node; Based on the task status information, the task information in the preset database is updated; Based on the task status information, a status update process corresponding to the task instance is performed in a preset target interface.

8. A task processing device, characterized in that: include: The first acquisition module is used to obtain the current time; A first judgment module is used to judge whether the current time reaches a preset task scheduling time; wherein the task scheduling time is the task scheduling time in the task configuration content corresponding to the target task pre-configured by the user; The second judgment module is used to call a preset main task management node and judge whether the main task management node is in a normal operating state; A first generating module, configured to obtain the task configuration content of the target task and generate a task instance corresponding to the task configuration content based on the main task management node; A first screening module, used to screen out a target task execution node from a plurality of preset task execution nodes; The sending module is used to send the execution instruction of the task instance to the target task execution node, so that the target task execution node performs corresponding task execution processing according to the execution instruction of the task instance.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the task processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the task processing method according to any one of claims 1 to 7 are implemented.

Images