Task processing method of distributed system and related device

Abstract

The invention provides a task processing method of a distributed system and a related device. The method comprises the following steps: for any node in the distributed system, detecting a resource use data sequence of the node in a task execution process within a first preset duration before a first moment; encoding resource use data at different moments in the resource use data sequence to obtain a first feature vector sequence composed of feature vectors at different moments; performing recursive prediction based on the first feature vector sequence and a second preset duration to obtain a first thread number; based on the first thread number, adjusting a first task number of the node at the first moment to obtain a second task number of the node at the second moment; and controlling the node to execute the task at the second moment based on the second task quantity. According to the invention, the task load of the nodes in the distributed system can be balanced.

Description

Technical Field

[0001] The present application relates to computer technology, and in particular to a task processing method and related devices for a distributed system. Background Art

[0002] Computing clusters or distributed systems can be used to perform tasks related to business indicator statistics. Task scheduling plays a vital role in the operation process of the entire system. It has an extremely close and serious dependency on resources. Once there is a shortage of resources, the operation of the entire system will be seriously affected. For example: when the CPU processing power is insufficient, the execution speed of the task will be significantly slowed down, and the task may even be stuck or unresponsive for a long time; when the memory space is insufficient, the program may crash due to the inability to allocate enough memory. The task processing of distributed systems is dependent on resources. When some nodes are short of resources, other nodes may be idle, and there will be resource imbalance in the task processing process of distributed systems. Summary of the Invention

[0003] Embodiments of the present application provide a task processing method, device, electronic device, computer-readable storage medium, and computer program product for a distributed system, which can balance the task load of nodes in the distributed system.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present invention provides a method for processing tasks in a distributed system, the method comprising:

[0006] For any node in the distributed system, detecting a resource usage data sequence of the node during a task execution process within a first preset time period before a first moment;

[0007] Encoding the resource usage data at different moments in the resource usage data sequence to obtain a first feature vector sequence composed of feature vectors at different moments;

[0008] Recursively predicting based on the first feature vector sequence and a second preset duration to obtain a first number of threads, wherein the first number of threads is the number of threads occupied by the node when executing computing tasks after a second moment, and the second moment is separated from the first moment by the second preset duration and is later than the first moment;

[0009] Adjusting the number of first tasks of the node at the first moment based on the first number of threads to obtain the number of second tasks of the node at the second moment;

[0010] At the second moment, the node is controlled to execute tasks based on the second task quantity.

[0011] The present invention provides a distributed system task processing device, including:

[0012] A resource detection module is used to detect, for any node in the distributed system, a sequence of resource usage data during task execution of the node within a first preset time period before a first moment;

[0013] a resource prediction module, configured to encode resource usage data at different moments in the resource usage data sequence to obtain a first feature vector sequence composed of feature vectors at different moments; and perform recursive prediction based on the first feature vector sequence and a second preset duration to obtain a first number of threads, wherein the first number of threads is the number of threads occupied by the node when executing computing tasks after a second moment, the second moment being separated from the first moment by the second preset duration and being later than the first moment;

[0014] A task processing module is used to adjust the first task quantity of the node at the first moment based on the first thread quantity to obtain the second task quantity of the node at the second moment; and control the node to execute tasks based on the second task quantity at the second moment.

[0015] An embodiment of the present application provides an electronic device, comprising:

[0016] a memory for storing computer executable instructions or computer programs;

[0017] The processor is used to implement the task processing method of the distributed system provided in the embodiment of the present application when executing the computer executable instructions or computer programs stored in the memory.

[0018] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program, which is used to implement the task processing method of the distributed system provided in the embodiment of the present application when executed by a processor.

[0019] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the distributed system task processing method provided in the embodiment of the present application is implemented.

[0020] The embodiments of the present application have the following beneficial effects:

[0021] For any node in a distributed system, a resource usage data sequence of the node during task execution in a time interval before a first moment is detected; based on the resource usage data sequence, the number of first threads at a second moment in the future is predicted, and based on the first number of threads, the number of tasks of the node after the second moment is adjusted. Based on resource usage data sequences at multiple historical moments, the time series characteristics of the load data can be obtained, which improves the accuracy of predicting the number of threads occupied by nodes in the distributed system at future moments. The number of tasks can be adjusted in advance according to the resource occupancy at future moments, and the node can be controlled to execute tasks at the second moment based on the adjusted number of tasks. This can avoid excessive resource occupancy or idleness of the node, balance the load of the distributed system, and improve the work efficiency and stability of the distributed system. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a schematic diagram of an application mode of the task processing method of the distributed system provided in an embodiment of the present application;

[0023] Figure 2 is a structural diagram of an electronic device provided in an embodiment of the present application;

[0024] Figure 3A This is a first flow chart of the task processing method of the distributed system provided in an embodiment of the present application;

[0025] Figure 3B This is a second flow chart of the task processing method of the distributed system provided in an embodiment of the present application;

[0026] Figure 3C This is a second flow chart of the task processing method of the distributed system provided in an embodiment of the present application;

[0027] Figure 4A This is a first principle diagram of task processing provided by an embodiment of the present application;

[0028] Figure 4B This is a second principle diagram of task processing provided by an embodiment of the present application;

[0029] Figure 4C This is a flowchart of the task processing provided by the embodiment of the present application;

[0030] Figure 5 This is a fourth flow chart of the task processing method of the distributed system provided in an embodiment of the present application;

[0031] Figure 6 This is a log diagram provided in an embodiment of the present application.

[0032] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of distinction between the advantages and disadvantages of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0034] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0035] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0036] It should be pointed out that the collection and processing of relevant data in this application (for example, data collected by business indicator detection tasks and node resource usage data) should be strictly in accordance with the requirements of relevant laws and regulations when applied in instances, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0037] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0039] Before further explaining the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0040] 1) Dimension Table: A statistical table designed around the business process environment. Dimension tables store information describing various aspects of measurement data, such as time, location, product, and customer. Dimension tables primarily consist of a primary key and various dimension fields, known as dimension attributes.

[0041] 2) Fact table: A statistical table designed around a business process that expresses the business process by obtaining metrics that describe the business process. It includes referenced dimensions and metrics related to the business process.

[0042] 3) Routine Load: This is a method for continuously importing data from a data source (e.g., Kafka).

[0043] 4) Data access: The process of connecting the data table to the indicator system, building data warehouses for these data, and creating data indicators.

[0044] 5) Data indicators: These are quantitative metrics generated by meticulously breaking down business units. They make abstract business goals (such as user growth and product health) describable, measurable, and decomposable. For example, metrics like daily active users (DAU) and conversion rate can directly reflect the operational status of a product.

[0045] 6) Recurrent Neural Network (RNN): This is a special type of artificial neural network specifically designed for processing sequential data. It has extensive applications in fields such as natural language processing, speech recognition, and time series analysis. In many practical problems, data has sequential characteristics, such as a piece of text, a speech signal, or a time series of stock prices. RNNs are capable of processing this type of sequential data, leveraging previous information in the sequence to influence the current output. RNNs are able to process data with sequential structure, taking into account the order and dependencies between elements in the sequence.

[0046] Computing clusters or distributed systems can be used to perform tasks related to business indicator statistics. Task scheduling plays a vital role in the operation process of the entire system, and it has an extremely close and serious dependency on resources. Once resources are insufficient, the operation of the entire system will be seriously affected. For example: when the CPU processing power is insufficient, the execution speed of the task will be significantly slowed down, and the task may even be stuck or unresponsive for a long time; when the memory space is insufficient, the program may crash due to the inability to allocate enough memory. The task processing of distributed systems is dependent on resources. When some nodes have insufficient resources, other nodes may be idle, and there is a resource imbalance in the task processing process of distributed systems. In the business indicator detection task, the task scheduling of the distributed system is seriously dependent on resources. If there is a shortage of resources, the current indicator cannot be counted.

[0047] Embodiments of the present application provide a task processing method for a distributed system, a task processing device for a distributed system, an electronic device, a computer-readable storage medium, and a computer program product, which can balance the task load of nodes in a distributed system.

[0048] The following describes exemplary applications of electronic devices provided in embodiments of the present application. The electronic devices provided in embodiments of the present application can be implemented as terminal devices, such as laptop computers, tablet computers, desktop computers, set-top boxes, smart TVs, vehicle-mounted terminals, virtual reality (VR) devices, augmented reality (AR) devices, and other types of terminals, and can also be implemented as servers. The following describes exemplary applications of electronic devices implemented as terminal devices or servers.

[0049] refer to Figure 1 , Figure 1 This is a schematic diagram of an application mode of the task processing method of the distributed system provided in an embodiment of the present application; for example, Figure 1 The invention involves a distributed system 100, a server 200, a network 300, a database 500 and a terminal device 400. The terminal device 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. The server 200 and the distributed system 100 communicate via a wired network or a wireless network.

[0050] In some embodiments, the server 200 is a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0051] For example, the server 200 may be a server of a business system, a server cluster composed of multiple physical servers, or the server 200 may be a server node responsible for task scheduling in the distributed system 100 .

[0052] In some embodiments, taking an advertising scenario as an example, a user browses an advertisement provided by an advertiser through a terminal device 400 and performs relevant interactive behaviors on the advertisement. The terminal device 400 sends the relevant data of the user's interactive behavior for the advertisement (for example, likes, purchases, and forwarding) to the database 500 through the network 300 for storage. The server 200 extracts the relevant data from the database 500 and sends the tasks to be performed and the relevant data to the distributed system 100. The distributed system 100 generates advertising-related indicator data based on the relevant data and the tasks to be performed, such as click-through rate, conversion rate, etc. The server 200 calls the task processing method of the distributed system provided in the embodiment of the present application, generates a scheduling instruction, adjusts the number of tasks of the nodes in the distributed system 100, and balances the resource usage of the distributed system. The distributed system 100 can also send the indicator data to the advertiser's terminal device through the network 300 so that the advertiser can understand the effect of the advertisement.

[0053] The embodiments of the present application can be implemented using database technology. A database, in short, can be considered an electronic filing cabinet that stores electronic files, allowing users to add, query, update, and delete data in these files. A "database" is a collection of data that is stored together in a specific manner, can be shared by multiple users, has minimal redundancy, and is independent of applications.

[0054] A database management system (DBMS) is a computer software system designed for managing databases, typically providing basic functions such as storage, retrieval, security, and backup. DBMSs can be categorized by the database model they support, such as relational or XML (Extensible Markup Language); by the type of computer they support, such as server clusters or mobile phones; by the query language they use, such as SQL or XQuery; by performance priorities, such as maximum scale or maximum speed; or by other classification methods. Regardless of the classification method used, some DBMSs are cross-category, for example, supporting multiple query languages ​​simultaneously.

[0055] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may be Figure 1 Server 200, Figure 2 The server 200 shown includes: at least one processor 410, a memory 450, and at least one network interface 420. The various components in the server 200 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0056] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0057] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0058] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0059] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0060] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0061] A network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include Bluetooth, WiFi, and USB.

[0062] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2 The distributed system task processing device 455 stored in the memory 450 is shown. It can be software in the form of programs and plug-ins, including the following software modules: resource detection module 4551, resource prediction module 4552, and task processing module 4553. These modules are logical and can be arbitrarily combined or further split according to the functions implemented. Figure 2 For the sake of convenience, all the above modules are shown at once, but it should not be considered that the task processing device 455 in the distributed system excludes the implementation that can only include the task processing module 4553. The functions of each module will be explained below.

[0063] In some embodiments, a terminal device or a server can implement the task processing method of the distributed system provided in the embodiment of the present application by running a computer program. For example, a computer executable instruction can be a command, a machine instruction or a software instruction at the microprogram level. A computer program can be a native program or a software module in an operating system; it can be a local (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as an advertising platform APP or an instant messaging APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0064] The task processing method of the distributed system provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the terminal device provided in the embodiment of the present application.

[0065] Below, the task processing method of the distributed system provided by the embodiment of the present application is described. As mentioned above, the electronic device that implements the task processing method of the distributed system provided by the embodiment of the present application can be a terminal device or a server, or a combination of the two. Therefore, the execution entity of each step will not be repeated below.

[0066] It should be noted that in the examples of task processing below, the task is described as a statistical data indicator. Based on the understanding of the following, those skilled in the art can apply the task processing method of the distributed system provided in the embodiment of the present application to other types of task processing. In the embodiment of the present application, any node in the distributed system is used as an example for description. The task processing method of the distributed system provided in the embodiment of the present application is applicable to each node in the distributed system for executing tasks.

[0067] See also Figure 3A , Figure 3A This is a flow chart of the task processing method of the distributed system provided by the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained. Figure 3A The execution entity of the step is Figure 1 Server 200.

[0068] In step 301, for any node in a distributed system, a resource usage data sequence of the node during task execution within a first preset time period before a first moment is detected.

[0069] For example, a distributed system can be a cluster of multiple servers, and the node can be a terminal device such as a computer, or the node can be a network server or a cloud server; assuming that the node is a server, the resource usage data includes but is not limited to the following: the number of threads currently occupied by the server, and the computing resources, memory and disk input / output (I / O) of the server's central processing unit (CPU), etc.

[0070] In some embodiments, detection processing can be implemented through the routine import (Routine Load) function of the distributed online analytical processing (OLAP) database (StarRocks). The resource usage data obtained through detection and the execution results generated during task execution can be stored in the distributed online analytical processing database. The routine import function uses an efficient data transmission and processing mechanism to ensure data consistency and real-time performance. During the synchronization process, it will automatically detect data changes and load the latest metadata into the corresponding detailed table in a timely and accurate manner.

[0071] For example, the first moment refers to the moment at which the detection process is executed, or a moment between moments in which the detection process is executed. For example, if it is currently 2 p.m., then the moment before the first moment is before 2 p.m. Resource usage data is represented as a time series. The time series includes multiple elements, each element corresponding to a different moment, separated by a preset time interval, and each element represents information about the node's resource usage at that moment. For example, the CPU utilization of a node is 40%, the number of threads occupied is 5, and the upper limit of the number of threads that can be occupied is 20. In a distributed system, tasks on a node can be implemented using threads. A task can be assigned to one or more threads for execution. For example, a task to calculate a data metric (e.g., the click-through rate or conversion rate of an advertisement over a certain period of time) can be completed by a single process or executed concurrently by multiple threads to improve processing speed and efficiency. Continuing with the above example, assume that of the five occupied threads, three threads are executing Tasks A, B, and C, respectively, and two threads are executing Task D. The number of tasks executed concurrently by the node is four. In the embodiments of the present application, a task refers to an activity completed by an application in a node. Tasks are divided according to the data processing process involved. For example, reading and storing the data on the number of clicks on a certain advertisement at each moment in a certain period of time can be considered Task A. Calculating a certain indicator based on the stored data can be considered Task B. Task A and Task B are both subtasks of the parent task of the statistical data indicator.

[0072] In step 302, resource usage data at different moments in the resource usage data sequence are encoded to obtain a first feature vector sequence consisting of feature vectors at different moments.

[0073] In some embodiments, step 302 can be implemented in the following manner: perform the following processing on the resource usage data corresponding to each moment in the resource usage data sequence: vectorize the resource usage data at the moment to obtain a first resource feature vector; pool the first resource feature vector to obtain a second resource feature vector, wherein the dimension of the second resource feature vector is lower than the dimension of the first resource feature vector; and combine the second resource feature vectors of each moment into a first feature vector sequence according to the time sequence.

[0074] For example, vectorization is used to encode a variety of different resource usage data (for example, the number of threads occupied, CPU occupancy, etc.) at a certain moment in the resource usage data into sub-feature vectors, and the sub-features are combined to obtain a first resource feature vector; pooling processing is an important operation of convolutional neural networks (CNNs) in deep learning, which is mainly used to reduce the spatial dimension of feature maps while retaining the main features, reducing the amount of calculation and preventing overfitting. In the embodiment of the present application, the first resource feature vector can be reduced in dimension by average pooling processing to obtain a second resource feature vector.

[0075] In step 303, recursive prediction is performed based on the first feature vector sequence and the second preset duration to obtain the first number of threads.

[0076] Here, the first number of threads is the number of threads occupied by the node when executing the computing task after the second moment, the second moment is separated from the first moment by a second preset time length, and the second moment is later than the first moment.

[0077] For example, the number of first threads predicted after the second moment is the number of threads that the node may occupy while executing tasks after the first moment. Continuing with the above example, suppose that based on the resource usage data sequence before 2:00 PM (the first moment), the number of threads in the resource usage data of the node at 2:15 PM (the second moment) is predicted. In this case, the second preset duration is 15 minutes.

[0078] For example, the prediction processing can be implemented in the following manner: collecting a preset number of third moments from a second preset time length; predicting the number of second threads at the first third moment based on the first sequence characteristics; iteratively performing the following processing: predicting the number of second threads at each third moment based on the first sequence characteristics and the obtained number of second threads; combining each number of second threads into a prediction data sequence according to the time sequence; encoding the prediction data sequence to obtain the second sequence characteristics; performing a linear transformation based on the first sequence characteristics and the second sequence characteristics to obtain the number of first threads at the second moment.

[0079] For example, the preset number can be set according to the needs of the actual application scenario. The preset number is positively correlated with the length of the second preset duration. The time intervals between the collected third moments can be random or equal. For example: if the second preset duration is 15 minutes, 5 third moments are randomly selected, and the third moments are 1 minute, 3 minutes, 6 minutes, 10 minutes, and 14 minutes respectively. Or, 15 minutes can be evenly divided into multiple time periods with an interval of 2.5 minutes, and the third moments are: 2 minutes and 30 seconds, 5 minutes, 7 minutes and 30 seconds, 10 minutes, and 12 minutes and 30 seconds.

[0080] Predicting the number of second threads at the first moment based on the first sequence features can be achieved through a sequence-to-sequence (Seq2Seq) model, which iteratively generates predicted values ​​for multiple time steps in the future. For example, first predict the number of second threads at the first third moment, then use the number of second threads at the first third moment and the first sequence features as input to predict the number of second threads at the second third moment, and predict the number of second threads at the i+1 third moment based on the number of second threads at the 1st to i third moments and the first sequence features, and so on. The sequence-to-sequence model is essentially a method of mapping an input sequence (such as a historical time series) to an output sequence (such as a predicted value for multiple time steps in the future) through an encoder-decoder architecture. In an embodiment of the present application, a sequence of predicted numbers of threads at multiple moments between the second moment and the first moment is generated based on an existing time feature sequence to predict the number of threads at the second moment. Linear transformation processing is achieved through activation functions, such as Sigmoid function, Tanh function, and Rectified Linear Unit (ReLU).

[0081] For example, prediction can be achieved through a recurrent neural network. Before executing step 303, a historical resource usage data sequence is obtained, and the historical resource usage data is cleaned. Specifically, empty data and abnormal data in the historical resource usage data are deleted to obtain the cleaned data. The cleaned data is used as sample data to train the recurrent neural network to predict the number of threads in the resource usage data at the next moment.

[0082] In an embodiment of the present application, the number of threads that a node may occupy in the future is predicted based on the existing resource usage data, and the task load of the node can be adjusted in advance based on the number of threads, which can ensure that the thread usage of the node is within a reasonable range, maintain the stable operation of the distributed system, and ensure that each task can be processed normally. Reasonable task load distribution can make the resources of the node more efficiently utilized. By collecting multiple third moments and generating time series other than the time series of existing resource usage data, it is possible to capture long sequence dependencies, output multi-step predictions, avoid error accumulation, and accurately predict the number of threads that may appear at the second moment. When it is predicted that the number of threads is about to increase, some tasks are transferred to other nodes with lighter loads in advance, which can avoid the low processing efficiency of a single node due to too many tasks, thereby improving the task processing speed of the entire system.

[0083] In step 304 , based on the first number of threads, the first number of tasks of the node at the first moment is adjusted to obtain the second number of tasks of the node at the second moment.

[0084] Here, the first number of tasks is the actual number of tasks the node is executing in parallel at the first moment, and the second number of tasks is the planned number of tasks the node is executing in parallel at the second moment. That is, at the first moment, the node is executing the first number of tasks simultaneously, and these tasks are executed in parallel and independently.

[0085] For example, the number of first threads is the number of threads that the node may occupy in the future. The number of first threads can represent the trend of the node occupying resources. Adjusting the number of tasks at the second moment according to the number of first threads can adjust the resources required by the node in advance, thereby avoiding excessive resource occupation or excessive idle resources, thereby balancing the resource load of each node in the distributed system. The adjustment can be achieved in the following way: when the number of first threads is in the first interval, the number of tasks is increased by a preset number; when the number of first threads is in the second interval, the number of tasks is maintained as it is; when the number of first threads is in the third interval, the number of tasks is reduced by a preset number. In actual application, the preset number is set according to the needs of the application scenario, the first interval is smaller than the second interval, the second interval is smaller than the third interval, and the three intervals are adjacent.

[0086] In some embodiments, reference Figure 3B , Figure 3B This is a second flow chart of the task processing method of the distributed system provided by the embodiment of the present application. Step 304 can be performed by Figure 3B Steps 3041 to 3042 in the embodiment are implemented as described below.

[0087] In step 3041, when the number of first threads is less than the first preset number of threads, the number of first tasks at the first moment is increased according to the first preset percentage to obtain the second number of tasks of the node after the second moment.

[0088] For example, to balance the number of threads, the first and second preset numbers of threads can be generated based on a preset parameter. Assuming the preset parameter is L0, the first preset number of threads is obtained by multiplying the preset parameter by a first ratio, and the second preset number of threads is obtained by multiplying the preset parameter by a second ratio. The first ratio is smaller than the second ratio. Assuming the first ratio is 0.8 and the second ratio is 1.2, the preset parameter L0*0.8 equals the first preset number of threads L1, and the preset parameter L0*1.2 equals the second preset number of threads L2.

[0089] Increasing the number of first tasks at a first moment by a first preset percentage means increasing the number of tasks by the first preset percentage multiplied by the first number of tasks. If the product of the first preset percentage and the first number of tasks is not a positive number, the number of tasks is rounded down. The first preset percentage is set based on actual application scenarios and can be obtained based on experimental data. The node can withstand an increase in the number of tasks by the first preset percentage * the first number of tasks for a certain period of time.

[0090] For example, if the first thread number L is less than the first preset thread number L1, then the number of tasks is increased by 0.2N based on the first task number N to obtain the second task number 1.2N.

[0091] For example, each node has multiple tasks to be executed. When the number of tasks to be executed is newly increased, the node automatically reads the increased number of tasks from the multiple tasks to be executed and executes the read tasks.

[0092] In step 3042, when the first thread number is greater than the second preset thread number, the first task number at the first moment is reduced according to the second preset percentage to obtain the second task number of the node after the second moment.

[0093] Here, the first preset number of threads is smaller than the second preset number of threads.

[0094] For example, the relationship between the first preset percentage and the second preset percentage includes, but is not limited to: being equal, the first preset percentage being greater than the second preset percentage, and the first preset percentage being less than the second preset percentage. Both the first preset percentage and the second preset percentage are positive numbers.

[0095] Reducing the number of first tasks at a first moment by a second preset percentage means reducing the number of tasks by the second preset percentage multiplied by the first number of tasks. If the product of the second preset percentage and the first number of tasks is not a positive number, the number of tasks is rounded up. The second preset percentage is set based on actual application scenarios and can be obtained based on experimental data. The node can withstand a reduction in the number of tasks by the second preset percentage * the first number of tasks for a certain period of time.

[0096] For example, if the first thread number L is greater than the second preset thread number L2, then the second task number 0.8N is obtained by subtracting 0.2N tasks from the first task number N.

[0097] For example, when reducing the number of executing tasks, the node automatically selects the task with the least execution progress from the multiple tasks being executed, and suspends the processing of the selected task until the other tasks that are not suspended are completed. Then, the suspended task is started first.

[0098] In some embodiments, when the first number of threads is greater than or equal to the first preset number of threads and less than or equal to the second preset number of threads, the first number of tasks is maintained, and the second number of tasks is the first number of tasks.

[0099] For example, if the first thread number L is smaller than the second preset thread number L2 and larger than the first preset thread number L1, the first task number is equal to the second task number, and the first task number remains unchanged.

[0100] In the embodiments of the present application, the number of tasks is increased or decreased based on the thread, ensuring the task load on each execution node, avoiding overloading of some nodes while leaving others idle. When the task load on each execution node is balanced, each node can operate in a relatively stable and efficient state, improving the efficiency of the distributed system in executing tasks.

[0101] Continue to refer Figure 3A In step 305, the node is controlled to execute tasks based on the second task quantity at the second moment.

[0102] For example, if the number of tasks of the node before the second moment is greater than the second number of tasks, then some tasks are suspended based on the degree of completion of the tasks being executed by the node, so that the number of tasks at the second moment is the second number of tasks. Conversely, if the number of tasks of the node before the second moment is less than the second number of tasks, then the number of tasks currently being executed by the node is increased to the second number of tasks.

[0103] In an embodiment of the present application, the number of tasks required for the second moment to ensure load balancing of the distributed system is determined in advance before the second moment, and the nodes are controlled to execute tasks with the corresponding number of tasks at the second moment, thereby balancing the load of the nodes, avoiding node overload or idleness, and improving the stability and efficiency of the distributed system in processing tasks.

[0104] In some embodiments, when executing step 301, at least one of the following processes is performed:

[0105] Processing 1: When the detection duration is greater than the first preset duration, re-execute step 301.

[0106] Here, the detection duration is the time taken to detect resource usage data.

[0107] For example, during the process of detecting resource usage data of a node executing a task, server 200 automatically records the start time and duration of the detection process. If the detection process duration exceeds a first preset duration, it indicates that there is a problem with the node or the detection process has failed, and step 301 is re-executed. In other words, a retry strategy is implemented to improve detection efficiency.

[0108] For example, the time interval for re-execution of step 301 can be set according to the actual application scenario to avoid retrying a large number of tasks at the same time and reduce system pressure. The number of retries does not exceed three. If all three attempts fail, the intelligent alarm notification interface is invoked to not only send an alarm message but also provide a detailed task execution log and possible failure cause analysis, facilitating rapid human intervention.

[0109] Processing 2: When the execution time of the node is greater than the third preset time, perform an alarm operation.

[0110] Here, execution time is the time it takes for a node to execute any task.

[0111] For example, the execution duration can be the duration recorded when the node is executing a task, or the total duration recorded when the node completes a task. The total duration can be stored in a database as a time reference for the node's task execution, and the average of multiple total durations can be calculated as the second preset duration. If the node's execution duration is longer than the second preset duration, it indicates that the node has a fault when executing the task, for example: a poor network leading to slow execution speed, or a physical fault in the node.

[0112] The alarm operation can be performed in the following manner: the server 200 sends the alarm information to the terminal device of the technician via the network 300. The alarm information includes the fault solution, the cause of the node abnormality, the execution time of the node, and the node identification.

[0113] For example, process 1 and process 2 may be executed in parallel or sequentially. Process 1 and process 2 are independent of each other and do not interfere with each other.

[0114] In some embodiments, reference Figure 3C , Figure 3C This is a third flow chart of the task processing method of the distributed system provided in the embodiment of the present application. Before step 301, execute Figure 3C Steps 3011 to 3013 in the process are described in detail below.

[0115] In step 3011, the number of tasks to be sent in each batch is determined based on the number of tasks to be executed.

[0116] Here, the number of tasks sent per batch is the number of tasks sent to the distributed system per cycle.

[0117] For example, the cycle length can be preset based on the number of tasks to be executed, or based on the time granularity of sending tasks to and receiving tasks from the distributed system. For example, divide 60 seconds by the number of tasks to be executed per minute to get the number of tasks to be executed per second. To save the number of sends, multiply 1 second by 10, and then multiply the number of tasks to be executed per second by 10 to get the number of sends per batch, and the cycle length is 10 seconds.

[0118] In step 3012, the priority of each task is determined, and the order of task issuance is determined according to the order of priority of each task from high to low.

[0119] For example, the priorities of the tasks are formed by sorting the attribute information of the tasks in descending order.

[0120] In some embodiments, “determining the priority of each task” in step 3012 may be implemented by performing a linear transformation based on the attribute information of each task to obtain the priority of each task;

[0121] The attribute information includes at least one of the following: the execution cycle of the task, the execution time of the task, the type tag value of the task, the total number of tasks of the same type, and the timeliness tag value of the task.

[0122] For example, the priority can be expressed as the following formula:

[0123] priority = team * 20% + timeliness tag value * 20% + task count * 30% + periodicity tag value * 30% + task execution time + urgency tag value + business change tag value

[0124] The task type tag value, team, can be determined by looking up a table that lists each type and its corresponding tag value. The tag value ranges from 1 to 100. For the timeliness tag value, the value is 200 for real-time tasks and 100 for non-real-time tasks. The task count is the total number of tasks of the same type. For the periodicity tag value, the periodic tag value of a single-execution task is greater than the periodic tag value of a periodic task, and greater than the periodic tag value of a scheduled task. Business variability is categorized into promotional business and event business, with each type corresponding to a different tag value.

[0125] In some embodiments, the priority is dynamically updated according to changes in the attribute information of the task. For example, assuming that the business corresponding to the task has changed, the attribute information of the task is updated when the business changes, and the priority of each task is adjusted based on the attribute information of the task.

[0126] In step 3013, tasks are sent to nodes in the distributed system according to the task sending order and the number of tasks to be sent in each batch.

[0127] For example, a task list is formed based on the order in which tasks are dispatched. Starting from the head of the task list, each batch of tasks is dispatched to all nodes in the distributed system. During the task dispatch process, server 200 adjusts the dispatch order of tasks in real time based on priority to adjust the specific tasks that are dispatched to the downstream system for execution in each batch. Server 200 only sends a fixed number of tasks to the downstream distributed system in each batch.

[0128] In some embodiments, step 3013 may be implemented as follows:

[0129] From each batch of tasks, multiple tasks belonging to the same parent task are determined; and the multiple tasks belonging to the same parent task are distributed to multiple nodes in the distributed system.

[0130] Here, one node is assigned one task.

[0131] For example, a parent task is divided according to its objectives. A parent task consists of multiple subtasks. The distributed system as a whole is used to execute the parent task, and each node in the distributed system executes the corresponding subtask. For the same parent task, each node is responsible for executing one of the subtasks. Each node performs different tasks and has different functions. As a result, each subtask is as independent as possible, reducing inter-task dependencies.

[0132] In some embodiments, when a task is completed, the execution result of the task is stored in the message queue; when all tasks in a batch are completed, the detection result of the parent task is generated based on the execution result stored in the message queue.

[0133] In this example, each task is assigned to a different node for parallel execution. To avoid synchronization issues, a message queue is used to coordinate communication between tasks. After each task completes, the result is sent to the message queue, and other tasks that need the result retrieve the data from the message queue. This enables asynchronous communication between tasks and ensures efficient execution.

[0134] In some embodiments, after the first batch of tasks is executed, the distributed system waits for a second interval, and then immediately executes the second batch of tasks after the second interval, and so on, until the last batch is executed.

[0135] In some embodiments, when all tasks in a batch are completed, after the detection result of the parent task is generated based on the execution result stored in the message queue, the following processing is performed:

[0136] The detection results are stored in a database; a first time interval is determined at which the storage time of the detection results begins; a target storage time of the detection results is determined based on a mapping relationship between the first time interval and the preset storage time, wherein different first time intervals correspond to different preset storage times, and the preset storage time corresponding to each first time interval is positively correlated with the number of tasks to be executed in the first time interval; when the storage time of the detection results reaches the target storage time, the detection results in the database are deleted.

[0137] For example, the detection results are stored in different storage spaces according to the time granularity of the detection results, wherein the time granularity includes: second level, minute level, hour level and day level; partitioning and storing data of different time granularity can improve query efficiency and classification management. The number of tasks to be executed in the first time interval can measure the busyness of the business corresponding to the task. The preset storage time corresponding to each first time interval is positively correlated with the number of tasks to be executed in the first time interval, so that the preset storage time is positively correlated with the busyness of the business. The busier the business, the longer the storage time. The time range for data storage is automatically adjusted according to the seasonality, periodicity and other characteristics of the business. Appropriately extend the number of days for data storage in the peak season and appropriately reduce the number of days for data storage in the off-season.

[0138] In the embodiment of the present application, different target storage durations are used to determine whether to delete data stored in different time periods, which effectively saves storage space and avoids redundant data.

[0139] In an embodiment of the present application, for any node in a distributed system, a resource usage data sequence of the node during the task execution process in a time interval before the first moment is detected; based on the resource usage data sequence, the number of first threads at the second moment in the future is predicted, and based on the number of first threads, the number of tasks of the node after the second moment is adjusted. Based on the resource usage data sequence of multiple historical moments, the time series characteristics of the load data can be obtained, and the accuracy of predicting the number of threads occupied by the nodes of the distributed system at future moments can be improved. The number of tasks can be adjusted in advance according to the resource occupancy at the future moment, and the node can be controlled to execute tasks at the second moment based on the adjusted number of tasks. This can avoid excessive resource occupation of the node or idle node, balance the load of the distributed system, and improve the work efficiency and stability of the distributed system.

[0140] The following describes an exemplary application of the task processing method of the distributed system according to an embodiment of the present application in a practical application scenario.

[0141] In related technologies, distributed systems can perform indicator calculation tasks for business systems. For example, if a company wants to track the effectiveness of online advertising in real time, it can use distributed systems to calculate indicators such as advertising volume, advertising click-through rate, advertising conversion, and advertising revenue. To understand the principle of task execution, refer to Figure 4A , Figure 4A This is a first principle diagram of task processing provided by an embodiment of the present application. Before the task is executed, relevant data is extracted from the database 500, and step 401A is executed, and the indicator configuration and display application are performed. In step 402A, the indicator is constructed and scheduled. The scheduling application generates business-related data, and based on the business-related data, step 403A is executed to obtain the indicator data. Based on the obtained indicator data, step 404A can be executed to display the indicator. However, task scheduling is heavily dependent on resources. If there is a shortage of resources, the current indicator cannot be counted.

[0142] The embodiments of the present application address the problems existing in related technologies and provide a task processing method for a distributed system, which can dynamically adjust the execution strategy of tasks, thereby ensuring uniform load of tasks and efficient use of system resources. Figure 6 , Figure 6 This is a fourth flow chart of the task processing method of a distributed system provided in an embodiment of the present application.

[0143] In step 501, a task to be executed is obtained.

[0144] For ease of understanding, the principle of task execution processing in the embodiment of the present application is explained by taking obtaining business indicators as an example. Figure 4B , Figure 4BThis is a schematic diagram of the second principle of task processing provided by an embodiment of the present application; business-related routines are imported from database 401B into hourly detail table 402B and daily detail table 403B. The data in hourly detail table 402B and daily detail table 403B are input into indicator system 406B. The required business indicator types are preset through data source 404B, and fact table and dimension table 405B are generated. Indicator system 406B determines the tasks to be executed based on fact table and dimension table 405B and business data, and executes the corresponding tasks to generate business indicators 407B.

[0145] For example, the routine import function synchronizes metadata to hourly and daily detail tables in real time. This synchronization mechanism provides a reliable data foundation for subsequent indicator calculations, allowing the system to promptly reflect the latest business dynamics.

[0146] refer to Figure 4C , Figure 4C It is a flowchart of the task processing provided by the embodiment of the present application; on the data source 410C side, the following processing is performed: in step 411C, a new data source is created; in step 412C, it is determined whether the real-time indicator is connected, and if so, the next step is executed; in step 413C, a data source configuration test is performed, and if the execution is completed, the next step is executed; in step 414C, the data source configuration is saved.

[0147] On the data access system 420C side, in step 421C, a fact table and dimension table are created. After step 421C, steps 422C and 426C are executed synchronously. In step 422C, table configuration information is generated. In step 423C, a daily detail table is created based on the table configuration information. In step 424C, an hourly detail table is created based on the table configuration information. In steps 425C and 426C, data is written to the detail table (including daily and hourly levels). Once both tables are created, the indicator creation phase ends.

[0148] For example, before executing a task in a distributed system, pre-process the data required for the task. Collect the raw data required for the task in advance, including the indicators to be calculated and the current limiting parameters, and correct the missing parameters of the raw data, remove duplicate records, and outliers. Resolve data redundancy and inconsistency, and transform the data structure into formatted data suitable for calculation. Build a task execution queue and store the tasks that need to be executed within the current minute in the queue. The queue stores all the data required to execute the current task, including: userId, email, fact table, dimension table, calculation formula, calculation time, data source configuration, etc., and integrate it into an executable SQL of the insert into select type. UserId, email, and other information are used for authentication operations, and insert into select is used to perform indicator data statistics and storage.

[0149] For example, when creating real-time metrics, hourly breakdown tables are created based on the distributed online analytical processing (OLAP) database (StarRocks). A windowing mechanism is introduced to automatically adjust the data retention time range based on business seasonality and cyclicality. The number of days data is retained is appropriately extended during peak seasons and reduced during off-seasons. Historical records are used to predict the hourly and daily data that may be queried within a specific time period in the future, and the table's starting values ​​are set in advance. For example, selecting time_unit = HOUR and the time format yyyyMMddHH allows data to be accurate to the hour, providing an accurate time dimension for real-time metric calculations. The starting value of retained dynamic partitions can be -1, and the number of pre-created partitions is 1. This strategy enables dynamic data management, reduces unnecessary historical data usage, and improves query efficiency. Retaining only the last two hours of data effectively controls storage costs and avoids redundant data.

[0150] For daily detail tables, set time_unit to DAY, using the yyyyMMdd format. Dynamic partitioning can retain a starting value of -7. Create a partition with 1 in advance, saving up to seven days of data. This provides an appropriate time window for calculating daily metrics and helps analyze long-term data trends. High-frequency queries and time-consuming queries are managed separately, ensuring they do not affect each other. In other words, daily and hourly data are managed separately. Hourly data is primarily used to calculate minute-level metrics, requiring fast calculation speed. Daily data is used to calculate daily metrics or even longer-term metrics. High speed is not required, but sufficient data volume is required.

[0151] In step 502, the task execution order is determined.

[0152] For example, the order of task execution can be determined based on the task priority. Priority can be expressed as the following formula:

[0153] priority = team * 20% + timeliness tag value * 20% + task count * 30% + periodicity tag value * 30% + task execution time + urgency tag value + business change tag value

[0154] The task type tag value, team, can be determined by looking up a table that lists each type and its corresponding tag value. The tag value ranges from 1 to 100. For the timeliness tag value, the value is 200 for real-time tasks and 100 for non-real-time tasks. The task count is the total number of tasks of the same type. For the periodicity tag value, the periodic tag value of a single-execution task is greater than the periodic tag value of a periodic task, and greater than the periodic tag value of a scheduled task. Business variability is categorized into promotional business and event business, with each type corresponding to a different tag value.

[0155] For example, in the embodiment of the present application, the task priority calculation does not rely on fixed weights, but adopts a dynamic jump mechanism to optimize the weight distribution to adapt to the ever-changing task environment and needs. For example: the initial priority is: business change degree (10%) + timeliness (20%) + taskCount (30%) + periodicity (30%) + others; server load: 60%. When it is detected that the active order corresponding to the task increases by 300%, a weight increase request is immediately initiated; when the resource utilization feedback load reaches 80%, the new server load of 80% is added to the priority calculation formula to obtain the updated priority.

[0156] For example, the priority calculation parameters and the enumeration form of synchronous encryption are stored in the security configuration file, which not only ensures the confidentiality of the data, but also allows calculation on the encrypted data to prevent illegal tampering while supporting flexible management of the data.

[0157] In step 503, the tasks are sent to the distributed system according to the task execution order.

[0158] For example, before executing the task of statistical indicators, obtain the number of indicators that need to be counted in the current minute (the total number of tasks per minute), divide it by 60, and get the number of tasks that need to be executed per second. After obtaining the task list, only a fixed number of tasks are sent to the downstream distributed system in each batch. Due to system limitations, we can obtain a theoretical interval time for sending tasks in sequence based on 60 seconds / total number of tasks. Considering the minimum time unit in the system scheduling task process, we multiply the theoretical interval time by 10 to obtain the first interval time for sending each batch of tasks. The tasks sent to the downstream at one time are regarded as a batch, which also means that the tasks of each batch are the total number of tasks / 60*10. To ensure that tasks are evenly distributed within 1 minute, a start execution time can be set for each batch of tasks received.

[0159] After the distributed system completes the first batch, it waits for the second interval. The second batch is then executed immediately after the second interval. This continues in this order until the last batch is executed. A batch of tasks is dispatched every first interval. The distributed system completes the currently executing batch and begins executing the next batch after the second interval.

[0160] For example, when executing tasks in a distributed system, a task decomposition strategy can be used for certain large tasks that are particularly slow to compute and require a large number of executions. For example, a large task can be divided into multiple small tasks with relatively independent functions based on the tasks' functions and data dependencies. For example, a complex data analysis task can be broken down into multiple small tasks based on different data dimensions or analysis steps. To ensure that each small task is as independent as possible and reduce dependencies between tasks, the input and output interfaces of each small task are clearly defined during the task design phase, so that it relies only on external data and clear parameters, avoiding direct data sharing between tasks.

[0161] During the execution process, small tasks are assigned to different processors or nodes in the distributed system for parallel execution. To avoid synchronization barriers, message queues are used to coordinate communication between tasks. After each small task is completed, the result is sent to the message queue, and other small tasks that need the result obtain data from the message queue, thereby realizing asynchronous communication between tasks and ensuring efficient execution of tasks. For example: Currently, indicators are implemented as a group of indicators, which is equivalent to putting the data of the same report into an indicator group, which can provide fast query and filtering functions. In the calculation, it is set to the required dimension data, indicator data, base table name data, and the association information between the dimension table and the indicator table. By splitting the tasks, the required data can be prepared in advance and stored in the cache. When calculating, the data is directly assembled from the cache for calculation, saving the time of interacting with the database.

[0162] In some embodiments, the order of task distribution is adjusted in real time according to the priority to adjust the specific tasks distributed to the downstream system for execution in each batch. For example, tasks with long execution times are evenly distributed in each scheduling to avoid having time-consuming tasks in a certain scheduling, resulting in no available resources in the next scheduling.

[0163] In step 504, task detection is performed, and the number of parallel tasks of the distributed nodes is adjusted according to the detection results.

[0164] Exemplarily, the task detection is at the node level. For example, the detection task is executed every 10 s to detect the number of occupied threads and key metric data such as the CPU occupancy rate, memory, and disk I / O of the server. The number of parallel tasks of the distributed nodes is adjusted based on the number of threads occupied by the node. The calculation method is to determine a benchmark task number N, and then make a judgment according to the relationship between the current thread number L and the preset thread number threshold L0. If L > L0 * 1.2, the task number is reduced to N * 0.8; if L < L0 * 0.8, the task number is increased to N * 1.2; otherwise, the task number is N. Ensure the task load on each execution node to avoid overloading of some nodes while other nodes are idle. The thread number threshold L0 is usually a fixed parameter set according to system performance testing, historical data analysis, or experience, stored in a configuration file (such as a.properties file), and the value is obtained by reading the configuration file when the program starts.

[0165] For example, for a node, a detection is performed at 4 o'clock to obtain the detection data of the node at 4 o'clock. The detection data at 4 o'clock is combined with historical detection data to predict the predicted thread number after 4 o'clock. The number of currently parallel tasks of the node is adjusted according to the predicted thread number, which can adjust the current task number in advance according to the predicted value and achieve dynamic load balancing.

[0166] In some embodiments, if the execution of the detection task fails or times out, a retry operation is performed according to the retry strategy. There is a retry interval between the retry operations of each layer to avoid frequent retries. The retry interval is calculated based on the historical retry records of the task, the task type and the current load of the system. For example, for complex and important tasks, the retry interval increases relatively slowly to ensure that there is enough time to solve the problem. For general tasks, the retry interval increases relatively quickly. The first retry interval is initially designed to be 4 seconds. The subsequent retry interval is based on exponential growth and a certain dynamic fluctuation is added in combination with a random number generator to avoid a large number of tasks being retried at the same time, reducing system pressure. The number of retries does not exceed three times. If all three attempts are unsuccessful, the intelligent alarm notification interface is called to not only send an alarm message, but also provide a detailed task execution log and possible failure cause analysis to facilitate rapid manual intervention.

[0167] In step 505, business indicators generated by task execution are obtained, stored, and displayed.

[0168] For example, the business indicators generated by task execution are collected in JSON format to represent the execution log of the task. Figure 6 , Figure 6 This is a log diagram provided by the embodiment of the present application. The log in JSON format can be expressed as Figure 6 The form of , is convenient for troubleshooting and performance analysis.

[0169] In this application, the dynamic adjustment of task execution strategies ensures uniform task load and efficient utilization of system resources. By splitting the calculation table and detail table, task scheduling time is effectively reduced. Real-time monitoring of system load and computing power can improve the stability of distributed system task execution.

[0170] The following continues to describe the exemplary structure of the distributed system task processing device 455 provided by the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2As shown, the software modules in the task processing device 455 of the distributed system stored in the memory 450 may include: a resource detection module 4551, which is used to detect, for any node in the distributed system, a resource usage data sequence of the node during the task execution process within a first preset time period before a first moment; a resource prediction module 4552, which is used to encode the resource usage data at different moments in the resource usage data sequence to obtain a first feature vector sequence composed of feature vectors at different moments; recursively predict based on the first feature vector sequence and a second preset time period to obtain a first number of threads, wherein the first number of threads is the number of threads occupied by the node to perform computing tasks after the second moment, the second moment is separated from the first moment by the second preset time period, and the second moment is later than the first moment; a task processing module 4553, which is used to adjust the first number of tasks of the node at the first moment based on the first number of threads to obtain the second number of tasks of the node at the second moment; and control the node to perform tasks based on the second number of tasks at the second moment.

[0171] In some embodiments, the task processing module 4553 is used to increase the first number of tasks at the first moment according to a first preset percentage when the first number of threads is less than the first preset number of threads, and obtain the second number of tasks of the node after the second moment; and to reduce the first number of tasks at the first moment according to a second preset percentage when the first number of threads is greater than the second preset number of threads, and obtain the second number of tasks of the node after the second moment, wherein the first preset number of threads is less than the second preset number of threads.

[0172] In some embodiments, the task processing module 4553 is used to perform at least one of the following processing when detecting the resource usage data of the node during the task execution process before the first moment: when the detection duration is greater than the first preset duration, re-execute the step of detecting the resource usage data of the node during the task execution process before the first moment, wherein the detection duration is the time taken to detect the resource usage data; when the execution duration of the node is greater than the third preset duration, perform an alarm operation, wherein the execution duration is the time taken for the node to execute any one of the tasks.

[0173] In some embodiments, the task processing module 4553 is used to determine the number of tasks to be sent in each batch based on the number of tasks to be executed before detecting the resource usage data of the task execution process of the node before the first moment, wherein the number of tasks sent in each batch is the number of tasks sent to the distributed system in each cycle; determine the priority of each task, and determine the order of task issuance according to the order of the priority of each task from high to low; and issue tasks to the nodes in the distributed system according to the task issuance order and the number of tasks sent in each batch.

[0174] In some embodiments, the task processing module 4553 is used to perform a linear transformation based on the attribute information of each task to obtain the priority of each task; wherein the attribute information includes at least one of the following: the execution cycle of the task, the execution duration of the task, the type label value of the task, the total number of tasks of the same type, and the timeliness label value of the task.

[0175] In some embodiments, the task processing module 4553 is used to determine multiple tasks belonging to the same parent task from each batch of tasks; and assign the multiple tasks belonging to the same parent task to multiple nodes in the distributed system, wherein one node is assigned one task; in some embodiments, the task processing module 4553 is used to store the execution result of the task in a message queue when the execution of one task is completed; and generate the detection result of the parent task based on the execution result stored in the message queue when all tasks in the batch are completed.

[0176] In some embodiments, the task processing module 4553 is used to, when all tasks of the batch are completed, generate the detection results of the parent task based on the execution results stored in the message queue, and then store the detection results in the database; determine the first time interval at which the storage moment of the detection results begins; determine the target storage duration of the detection results based on the mapping relationship between the first time interval and the preset storage duration, wherein the preset storage duration corresponding to each first time interval is positively correlated with the number of tasks to be executed in the first time interval; and delete the detection results in the database when the storage duration of the detection results reaches the target storage duration.

[0177] An embodiment of the present application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program or computer-executable instructions from the computer-readable storage medium and executes the computer program or computer-executable instructions, causing the electronic device to perform the task processing method of the distributed system described in the embodiment of the present application.

[0178] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the task processing method of the distributed system provided by the embodiment of the present application, for example, Figure 3A The task processing method of the distributed system is shown.

[0179] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0180] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0181] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0182] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0183] In summary, through the embodiments of the present application, for any node in a distributed system, a resource usage data sequence of the node in the task execution process in a time interval before a first moment is detected; based on the resource usage data sequence, the number of first threads at a second moment in the future is predicted, and based on the first number of threads, the number of tasks of the node after the second moment is adjusted. Based on the resource usage data sequence of multiple historical moments, the time series characteristics of the load data can be obtained, and the accuracy of predicting the number of threads occupied by the nodes of the distributed system at future moments can be improved. The number of tasks can be adjusted in advance according to the resource occupancy at future moments, and the node can be controlled to execute tasks at the second moment based on the adjusted number of tasks. This can avoid excessive resource occupancy of the node or idle nodes, balance the load of the distributed system, and improve the work efficiency and stability of the distributed system.

[0184] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A task processing method for a distributed system, characterized in that: The method comprises: For any node in the distributed system, detecting a resource usage data sequence of the node during a task execution process within a first preset time period before a first moment; Encoding the resource usage data at different moments in the resource usage data sequence to obtain a first feature vector sequence composed of feature vectors at different moments; Recursively predicting based on the first feature vector sequence and a second preset duration to obtain a first number of threads, wherein the first number of threads is the number of threads occupied by the node when executing computing tasks after a second moment, and the second moment is separated from the first moment by the second preset duration and is later than the first moment; Adjusting the number of first tasks of the node at the first moment based on the first number of threads to obtain the number of second tasks of the node at the second moment; At the second moment, the node is controlled to execute tasks based on the second task quantity.

2. The method according to claim 1, characterized in that The recursive prediction based on the first feature vector sequence and the second preset duration to obtain the first number of threads includes: Collecting a preset number of third moments from the second preset time period; Predicting the number of second threads at the first third moment based on the first sequence feature; Iteratively performing the following processing: predicting the number of second threads at each third moment based on the first sequence feature and the obtained number of second threads; combining each of the second thread quantities into a prediction data sequence according to a time sequence; Encoding the predicted data sequence to obtain a second sequence feature; A linear transformation is performed based on the first sequence feature and the second sequence feature to obtain the first thread quantity at the second moment.

3. The method according to claim 1, characterized in that The encoding of the resource usage data at different moments in the resource usage data sequence to obtain a first feature vector sequence composed of feature vectors at different moments includes: Perform the following processing on the resource usage data corresponding to each moment in the resource usage data sequence: Vectorizing the resource usage data at the moment to obtain a first resource feature vector; Pooling the first resource feature vector to obtain a second resource feature vector, wherein the dimension of the second resource feature vector is lower than the dimension of the first resource feature vector; According to a time sequence, the second resource feature vector at each moment is combined into a first feature vector sequence.

4. The method according to claim 1, wherein The adjusting, based on the first number of threads, the number of first tasks of the node at the first moment to obtain the number of second tasks of the node at the second moment includes: When the first number of threads is less than a first preset number of threads, increasing the first number of tasks at the first moment according to a first preset percentage to obtain a second number of tasks for the node after the second moment; When the first number of threads is greater than the second preset number of threads, the first number of tasks at the first moment is reduced according to a second preset percentage to obtain the second number of tasks of the node after the second moment, wherein the first preset number of threads is less than the second preset number of threads.

5. The method according to claim 1, wherein When detecting a resource usage data sequence of the node during task execution within a first preset time period before the first moment, the method further includes: Perform at least one of the following actions: If the detection duration is greater than the first preset duration, re-performing the step of detecting a sequence of resource usage data during the task execution of the node within the first preset duration before the first moment, wherein the detection duration is the time taken to detect the resource usage data; When the execution time of the node is greater than a third preset time, an alarm operation is performed, wherein the execution time is the time taken by the node to execute any one of the tasks.

6. The method according to claim 1, characterized in that Before detecting a sequence of resource usage data during task execution of the node within a first preset time period before the first moment, the method further includes: Determine the number of tasks to be sent in each batch according to the number of tasks to be executed, wherein the number of tasks to be sent in each batch is the number of tasks sent to the distributed system in each cycle; Determine the priority of each task, and determine the order in which the tasks are issued according to the priority of each task from high to low; The tasks are sent to the nodes in the distributed system according to the task sending order and the sending quantity in each batch.

7. The method according to claim 6, characterized in that Determining the priority of each task includes: Performing a linear transformation based on the attribute information of each task to obtain the priority of each task; The attribute information includes at least one of the following: the execution cycle of the task, the execution duration of the task, the type tag value of the task, the total number of tasks of the same type, and the timeliness tag value of the task.

8. The method according to claim 6, characterized in that The sending of tasks to nodes in the distributed system according to the task sending order and the sending quantity in each batch includes: From each of the batches of tasks, determining a plurality of tasks belonging to the same parent task; Allocating a plurality of tasks belonging to the same parent task to a plurality of nodes in the distributed system, wherein one of the nodes is allocated the task; The method further comprises: When one of the tasks is completed, the execution result of the task is stored in the message queue; When all tasks in the batch are completed, a detection result of the parent task is generated based on the execution result stored in the message queue.

9. The method according to claim 8, characterized in that When all tasks in the batch are completed, after generating a detection result of the parent task based on the execution result stored in the message queue, the method further includes: Storing the test results in a database; Determining a first time interval for starting to store the detection result; Determining a target storage duration for the detection results based on a mapping relationship between the first time intervals and preset storage durations, wherein the preset storage duration corresponding to each first time interval is positively correlated with the number of tasks to be executed within the first time interval; When the storage time of the detection result reaches the target storage time, the detection result in the database is deleted.

10. A task processing device for a distributed system, characterized in that: The device comprises: A resource detection module is used to detect, for any node in the distributed system, a sequence of resource usage data during task execution of the node within a first preset time period before a first moment; a resource prediction module, configured to encode resource usage data at different moments in the resource usage data sequence to obtain a first feature vector sequence composed of feature vectors at different moments; and perform recursive prediction based on the first feature vector sequence and a second preset duration to obtain a first number of threads, wherein the first number of threads is the number of threads occupied by the node when executing computing tasks after a second moment, the second moment being separated from the first moment by the second preset duration and being later than the first moment; A task processing module is used to adjust the first task quantity of the node at the first moment based on the first thread quantity to obtain the second task quantity of the node at the second moment; and control the node to execute tasks based on the second task quantity at the second moment.

11. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions or computer programs; The processor is configured to implement the task processing method of the distributed system according to any one of claims 1 to 9 when executing the computer executable instructions or computer program stored in the memory.

12. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the task processing method of the distributed system according to any one of claims 1 to 9 is implemented.

13. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the task processing method of the distributed system according to any one of claims 1 to 9 is implemented.

Images