Micro-task scheduling method, system and electronic device

Abstract

The application relates to the technical field of data processing, and provides a micro task scheduling method and system and electronic equipment. The application obtains a plurality of micro task nodes encapsulating data processing logic, constructs a directed acyclic graph according to the dependency relationship between the nodes, generates addressing information stored independently by each micro task node according to the directed acyclic graph, and when any micro task node receives input data and processes the input data to obtain output data, queries a downstream node according to the addressing information stored in the micro task node, and directly sends the output data to the downstream node, so that the downstream node takes the output data as input data. According to the addressing information stored independently by each micro task node generated according to the directed acyclic graph, each micro task node is given autonomous addressing capability, the micro task node pushes data to a downstream node according to the addressing information stored in the micro task node, and the scheduling efficiency of the micro task is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a microtask scheduling method, system, and electronic device. Background Technology

[0002] In modern industrial manufacturing, especially in continuous process industries such as steel and chemicals, tens of thousands of sensors and devices are deployed on production lines, continuously generating real-time data. To support critical business applications such as process optimization, energy efficiency monitoring, and fault prediction, the entire computing logic needs to be broken down into multiple micro-tasks with execution logic. A stream processing framework is then used to aggregate, analyze, and process the real-time data in real time. A centralized scheduler or coordinating node is used to maintain the topology of the entire micro-task. After the upstream computing node completes data processing, the central node usually needs to coordinate the data flow, or the node needs to query the address of the downstream task from the central scheduler before the data is transferred and the task is dispatched.

[0003] However, since every data transfer and task triggering between microtasks highly depends on the coordination or query of the globally unique central scheduling node, the central scheduling node is subjected to huge concurrent request pressure. The centralized scheduling is inefficient and has extremely high latency. This not only consumes a lot of network costs and communication overhead, but also makes the central scheduler quickly become the performance bottleneck of the entire computing cluster. The scheduling efficiency of microtasks is low and cannot meet the stringent requirements of industrial sites for extremely low latency and efficient scheduling of massive microtasks. Summary of the Invention

[0004] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0005] In view of the shortcomings of the prior art described above, this application provides a microtask scheduling method, system and electronic device to improve the scheduling efficiency of microtasks.

[0006] In a first aspect, this application provides a microtask scheduling method, comprising: acquiring multiple microtask nodes, wherein each microtask node encapsulates data processing logic; constructing a directed acyclic graph based on the dependencies between the microtask nodes, and generating addressing information stored by each microtask node based on the directed acyclic graph; executing the upstream node in response to the upstream node receiving corresponding input data, wherein the execution process of the upstream node includes: processing the input data according to the data processing logic encapsulated by the upstream node to obtain output data, and querying a downstream node from each of the microtask nodes based on the addressing information stored by the upstream node, wherein the upstream node is any microtask node; sending the output data of the upstream node to the downstream node, triggering the downstream node to use the output data as input data for the downstream node.

[0007] In one embodiment of this application, obtaining multiple microtask nodes includes: obtaining microtask configuration, wherein the microtask configuration includes data processing logic; encapsulating an initial unit according to the microtask configuration; allocating a computing state space and a memory allocation area to the initial unit, and optimizing the data structure of the initial unit so that the memory usage of the initial unit is within a preset usage range, thereby obtaining the microtask node.

[0008] In one embodiment of this application, generating addressing information stored by each of the micro-task nodes according to the directed acyclic graph includes: determining micro-task nodes with dependencies as upstream nodes and downstream nodes corresponding to the upstream nodes according to the topology of the directed acyclic graph; storing the addresses of the downstream nodes in the internal memory of the upstream nodes in the form of a routing table to obtain the addressing information of the upstream nodes.

[0009] In one embodiment of this application, the method further includes: in response to a new microtask node creation event, adding the new microtask node to the directed acyclic graph (DAG), and updating the addressing information stored by each microtask node according to the DAG; in response to an update event of any microtask node, if the update event includes updating the dependency relationship corresponding to the microtask node, updating the DAG according to the updated dependency relationship, and updating the addressing information stored by each microtask node according to the DAG; in response to a deletion event of any microtask node, deleting the microtask node from the DAG, and updating the addressing information stored by each microtask node according to the DAG.

[0010] In one embodiment of this application, the method further includes: acquiring a real-time data stream containing a data identifier; determining a target node from each of the micro-task nodes according to a preset correspondence between the data identifier and the micro-task nodes; using the real-time data stream as input data for the target node; and routing and distributing the real-time data stream to the target node.

[0011] In one embodiment of this application, executing the upstream node includes: acquiring multiple worker threads and microtask node queues corresponding to each worker thread; storing upstream nodes that receive input data through each microtask node queue, wherein any microtask node queue is designated as a first queue; if the first queue is an empty set, determining a second queue containing microtask nodes from the microtask node queues corresponding to each worker thread, and moving at least a portion of the microtask nodes contained in the second queue to the first queue; if the first queue contains microtask nodes, executing the microtask nodes contained in the first queue through the worker thread corresponding to the first queue.

[0012] In one embodiment of this application, sending the output data of the upstream node to the downstream node includes: obtaining a shared memory region between the upstream node and the downstream node; storing the output data in the shared memory region to obtain a memory reference pointing to the output data; and sending the memory reference to the downstream node to trigger the downstream node to read the output data according to the memory reference.

[0013] In one embodiment of this application, after generating the addressing information stored by each of the micro-task nodes according to the directed acyclic graph, the method further includes: evaluating the upstream node and the downstream node according to preset multi-dimensional indicators to obtain a current evaluation result, wherein the multi-dimensional indicators include data filtering rate, data overlap rate, and node congestion rate; if the data processing logic of the downstream node includes data filtering, and the current evaluation result includes the data filtering rate of the downstream node being greater than or equal to a preset filtering rate threshold, then the dependency relationship between the upstream node and the downstream node is adjusted, and the directed acyclic graph is updated. The addressing information; if there are multiple downstream nodes, and the current evaluation result includes a data overlap rate between each downstream node that is greater than or equal to a preset overlap rate threshold, then the same data processing logic is extracted from the data processing logic corresponding to each downstream node, and a new micro-task node is generated based on the extracted data processing logic; if the current evaluation result includes a node congestion rate of the downstream node that is greater than or equal to a preset congestion rate threshold, then multiple new downstream nodes are generated according to the data processing logic encapsulated by the downstream node, and the addressing information of the upstream node is configured to point to each downstream node.

[0014] Secondly, this application also provides a microtask scheduling system, comprising: an acquisition module for acquiring multiple microtask nodes, wherein each microtask node encapsulates data processing logic; a generation module for constructing a directed acyclic graph based on the dependencies between the microtask nodes, and generating addressing information stored by each microtask node based on the directed acyclic graph; an execution module for executing the upstream node in response to the upstream node receiving corresponding input data, wherein the execution process of the upstream node includes processing the input data according to the data processing logic encapsulated by the upstream node to obtain output data, and querying a downstream node from each microtask node based on the addressing information stored by the upstream node, wherein the upstream node is any microtask node; and a sending module for sending the output data of the upstream node to the downstream node, triggering the downstream node to use the output data as input data.

[0015] Thirdly, this application also provides an electronic device, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the method described above.

[0016] The beneficial effects of this application are: By acquiring multiple microtask nodes encapsulating data processing logic and constructing a directed acyclic graph (DAG) based on the dependencies between nodes, and generating addressing information independently stored by each microtask node based on this DAG, when any microtask node receives input data and processes it to obtain output data, it queries downstream nodes based on its internally stored addressing information and directly sends the output data to the downstream nodes, thereby triggering the downstream nodes to use the output data as input data. In this way, compared to centralized scheduling of microtask nodes, generating addressing information stored separately by each microtask node based on the DAG, and giving each microtask node autonomous addressing capabilities, eliminates the need for microtask nodes to send status reports or initiate routing query requests to any global central node. Instead, it relies on its internally stored addressing information to push data to downstream nodes, breaking the bottleneck of centralized scheduling and eliminating the concurrent processing bottlenecks and single-point failure risks brought by the central scheduling node. In scenarios with a large number of microtasks, it achieves extremely low latency data-driven operations and extremely high scheduling throughput, thereby improving the scheduling efficiency of microtasks. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0018] In the attached diagram: Figure 1 This is a flowchart illustrating a microtask scheduling method in an embodiment of this application; Figure 2 This is a flowchart illustrating another microtask scheduling method in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a microtask scheduling system in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a computer system of an electronic device in an embodiment of this application. Detailed Implementation

[0019] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0020] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0021] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0022] The terms "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0023] Unless otherwise stated, the term "multiple" means two or more.

[0024] In this application, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0025] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0026] To facilitate a clearer and more accurate understanding of the technical solutions disclosed in this application by those skilled in the art, the following will first explain and describe the specific meanings of some key technical terms involved in the specification and claims within the specific technical context of this application. It should be particularly emphasized that the following explanations aim to clarify the internal logic and data relationships constructed by this invention to solve specific technical problems, and do not imply that these terms themselves are prior art, nor do they constitute any limitation on the scope of protection of this application.

[0027] A microtask (Actor) is a basic execution entity in a concurrent computing model based on a message passing mechanism. Each Actor encapsulates a private running state and behavior, and has an extremely lightweight context and a very low memory resource consumption. It is designed to serve as the smallest schedulable unit, carrying a single computation step to support the high-density concurrent computing needs of a large amount of data stream.

[0028] Data processing logic is the core instruction sequence and rule set in a computer program that performs substantive parsing, transformation, calculation, or evaluation of input data. From the perspective of underlying algorithm implementation, its specific manifestations include, but are not limited to, mathematical operation formulas, digital signal processing algorithms, operator aggregation operations, and Boolean logic branch determination based on preset thresholds.

[0029] A routing table is a basic data structure used to store the mapping relationship between data flow paths and addresses. In node-level data interaction scenarios, the routing table typically records the logical identifier of the data receiver and its corresponding real physical address, network communication endpoint, or memory instance handle in the form of a hash table, key-value pairs, or pointer array. By retrieving this data structure, the data sending node can determine the next-hop target address of the data packet or memory message, thereby achieving precise addressing and targeted delivery of data.

[0030] Combination Figure 1 As shown, this application provides a microtask scheduling method, including: Step S101: Obtain multiple microtask nodes; Each microtask node encapsulates data processing logic.

[0031] In some embodiments, a microtask node is an independent computing unit based on a lightweight encapsulation of the Actor model, i.e., an Actor instance.

[0032] In some embodiments, the data processing logic comprises various transformation, calculation, or evaluation operations performed on the industrial data stream. The data processing logic includes signal preprocessing logic, statistical aggregation logic, and business decision logic. For example, the signal preprocessing logic includes fast Fourier transform, low-pass filtering, and denoising algorithms performed on vibration sensor data. The statistical aggregation logic includes calculation of the average temperature of the heating furnace and extraction of the maximum / minimum values ​​performed within a sliding time window. The business decision logic includes equipment health assessment algorithms, real-time energy efficiency analysis algorithms, or threshold over-limit alarm logic derived from equipment operating characteristics.

[0033] Step S102: Construct a directed acyclic graph based on the dependencies between each microtask node, and generate addressing information stored by each microtask node based on the directed acyclic graph.

[0034] In some embodiments, the task manager parses the input-output dependencies of each microtask node. For example, the input of the "calculation of average furnace temperature" node depends on the output of multiple "temperature sensor acquisition" nodes. Logically, it constructs a DAG (Directed Acyclic Graph) topology representing the unidirectional flow of data and generates the addressing information stored by each microtask node based on the topology of the DAG. Step S103: In response to the upstream node receiving the corresponding input data, execute the upstream node; The execution process of the upstream node includes processing the input data according to the data processing logic encapsulated by the upstream node to obtain the output data, and querying the downstream node from each micro-task node according to the addressing information stored in the upstream node. The upstream node can be any micro-task node.

[0035] In some embodiments, the input data is timestamped sampled data or output data from an upstream node. The sampled data includes raw physical indicators collected by the sensor, device status codes, or intermediate calculation results from the previous microtask node.

[0036] Step S104: Send the output data of the upstream node to the downstream node, triggering the downstream node to use the output data as the input data of the downstream node.

[0037] In some embodiments, after completing data processing, the upstream node queries its internally stored addressing information to trigger the downstream node to use the upstream node's output data as input data for the next calculation in a point-to-point manner, thereby forming a data flow driven process.

[0038] The microtask scheduling method provided in this application acquires multiple microtask nodes encapsulated with data processing logic and constructs a directed acyclic graph (DAG) based on the dependencies between nodes. Addressing information is generated independently for each microtask node based on this DAG. When any microtask node receives input data and processes it to obtain output data, it queries downstream nodes based on its internally stored addressing information and directly sends the output data to the downstream nodes, thereby triggering the downstream nodes to use the output data as input data. Compared to centralized scheduling of microtask nodes, generating addressing information for each microtask node based on the DAG grants each microtask node autonomous addressing capabilities. Microtask nodes no longer need to send status reports or initiate routing query requests to any global central node; instead, they rely on their internally stored addressing information to push data to downstream nodes. This breaks the bottleneck of centralized scheduling, eliminates the concurrent processing bottlenecks and single-point failure risks associated with central scheduling nodes, and achieves extremely low-latency data-driven scheduling and extremely high scheduling throughput in scenarios with a large number of microtasks, thereby improving the scheduling efficiency of microtasks.

[0039] Optionally, multiple microtask nodes can be obtained, including: obtaining microtask configuration, wherein the microtask configuration includes data processing logic; encapsulating an initial unit according to the microtask configuration; allocating a computational state space and memory allocation area to the initial unit, and optimizing the data structure of the initial unit so that the memory usage of the initial unit is within a preset usage range, thereby obtaining the microtask node.

[0040] In some embodiments, all defined microtask configurations are loaded from persistent storage such as a database or configuration center. These microtask configurations include microtask identifiers, input source identifiers, output targets, and required algorithm libraries. In some embodiments, the computation state space is used to store business context data that microtasks need to retain across events during continuous stream computation. By allocating an isolated computation state space to each unit, it is ensured that each microtask node can only read and write its own state, thereby eliminating shared state lock contention during multi-threaded concurrent execution.

[0041] In some embodiments, to prevent a large number of microtask nodes from frequently requesting or releasing memory, which could lead to severe memory fragmentation, a dedicated memory allocation area is allocated for the initial unit, and all temporary variables during the operation of the microtask node are divided within this dedicated area.

[0042] In some embodiments, a compact memory layout design is adopted, and specific compiler instructions are used to eliminate unnecessary memory byte alignment gaps in the data structure, so that the memory usage of the initial unit is within a preset usage range, and the memory usage of the micro-task node is limited to a preset usage range, for example, the usage range is set to between 200 bytes and 500 bytes, so that a single server hardware can support the concurrent operation of millions of micro-tasks.

[0043] Optionally, addressing information stored by each microtask node is generated based on the directed acyclic graph, including: determining the microtask nodes with dependencies as upstream nodes and the downstream nodes corresponding to the upstream nodes according to the topology of the directed acyclic graph; storing the addresses of the downstream nodes in the internal memory of the upstream nodes in the form of a routing table to obtain the addressing information of the upstream nodes.

[0044] In some embodiments, the task manager traverses the directed acyclic graph, identifies the data provider as the upstream node and the data receiver as the downstream node for each directed edge in the graph; extracts the instance address or handle identifier of the downstream node in physical memory, and stores the set of addresses in the internal storage space of the upstream node in the form of a routing table, thereby enabling the upstream node to have routing distribution capabilities and realizing a completely decentralized communication topology.

[0045] Optionally, the method further includes: in response to a creation event of a new microtask node, adding the new microtask node to the directed acyclic graph (DAG) and updating the addressing information stored by each microtask node according to the DAG; in response to an update event of any microtask node, if the update event includes updating the dependency relationship corresponding to the microtask node, updating the DAG according to the updated dependency relationship and updating the addressing information stored by each microtask node according to the DAG; in response to a deletion event of any microtask node, deleting the microtask node from the DAG and updating the addressing information stored by each microtask node according to the DAG.

[0046] In some embodiments, if an "energy efficiency prediction" node is received, the "energy efficiency prediction" node is initialized in memory and added to the DAG as a new node. The upstream node that provides the data is located, and the memory address of the new node is appended to the routing table inside the upstream node. There is no need to restart the system.

[0047] In some embodiments, a fixed-size contiguous physical memory is allocated for the newly added microtask node in a dedicated memory allocation area, and its internal data processing logic is initialized, generating an instance handle for the new node. Based on the data dependencies of the newly added microtask node, the upstream node providing data is determined. To avoid blocking frequently running upstream nodes, a copy of the upstream node's current routing table array is cloned, and the instance handle of the new node is appended to the end of the cloned routing table array. A processor-level compare-and-swap atomic instruction is invoked to replace the pointers within the upstream node that point to the old routing table with pointers that point to the new routing table, thus completing the hot update of the addressing information. The old routing table is automatically reclaimed after its reference count reaches zero.

[0048] In some embodiments, if the algorithm formula of a microtask node is upgraded, the new logic is hot-loaded onto the node. At the same time, if the dependency source of the microtask node changes, the routing tables of the relevant upstream and downstream nodes are updated synchronously.

[0049] In some embodiments, if the update event represents an upgrade in the data processing logic within the microtask node, a new logic function block is loaded into memory and a new function pointer is obtained; an update message with a control identifier is sent to the mailbox of the microtask node; when the microtask node processes the update message, its internal logic function pointer is replaced with the new function pointer through atomic operations, so that the updated data processing logic is automatically adopted when processing subsequent input data; if the update event represents a new data dependency source for the microtask node, the new upstream node is determined according to the directed acyclic graph; based on the aforementioned copy-on-write and atomic replacement mechanism, the instance handle of the current microtask node is appended to the internal routing table of the new upstream node.

[0050] In some embodiments, if a microtask node is deleted, a safety stop signal is sent to the microtask node to release its memory and notify all its upstream nodes to remove the address of the deleted node from their internal routing tables, thereby cutting off the data flow.

[0051] In some embodiments, all upstream nodes corresponding to the microtask node to be deleted are determined; based on the copy-on-write mechanism, a new routing table is generated that removes the address of the microtask node to be deleted, and atomic instructions are used to replace the routing pointers of each upstream node to cut off the new data flow to the microtask node to be deleted; for the historical input data that may be backed up in the lockless double-ended queue corresponding to the microtask node to be deleted, a termination flag message is delivered to the tail of the lockless double-ended queue; the microtask node to be deleted continues to execute and processes the historical input data backed up in its queue in turn; when the termination flag message is read and parsed, the destructor logic is triggered to return the computing state space and memory allocation area occupied by the microtask node to the memory pool, completing the safe release of node resources without data loss.

[0052] Optionally, the method further includes: acquiring a real-time data stream containing data identifiers; determining a target node from each microtask node based on a preset correspondence between the data identifiers and microtask nodes; using the real-time data stream as input data for the target node; and routing and distributing the real-time data stream to the target node.

[0053] In some embodiments, the front end of the microtask scheduling system is deployed with a data routing module, which consumes real-time industrial data streams containing data identifiers from message queue layers such as Kafka and RabbitMQ. The data identifiers include the physical network address of the device, the measurement point identifier, etc., for example, the data identifier is "Vibration_Sensor_001". After obtaining the data stream, the target node is determined by the mapping hash table between the data identifier and the microtask node, and the data stream is delivered to one or more target nodes in a parallel distribution form as the initial input data to trigger their execution.

[0054] Optionally, executing upstream nodes includes: obtaining multiple worker threads and microtask node queues corresponding to each worker thread; storing upstream nodes that receive input data through each microtask node queue, wherein any microtask node queue is used as a first queue; if the first queue is an empty set, then determining a second queue containing microtask nodes from the microtask node queues corresponding to each worker thread, and moving at least a portion of the microtask nodes contained in the second queue to the first queue; if the first queue contains microtask nodes, then executing the microtask nodes contained in the first queue through the worker thread corresponding to the first queue.

[0055] In some embodiments, multiple worker threads are acquired. The number of worker threads is typically created based on the number of physical CPU cores on the server. Each worker thread is bound to a specific CPU core via an underlying operating system API (Application Programming Interface), such as sched_setaffinity in Linux, to minimize cache invalidation overhead caused by thread context switching. Each worker thread corresponds to a microtask node queue, which is implemented using a lock-free double-ended queue. An upstream node that is activated upon receiving input data is pushed to the head of the lock-free double-ended queue of its bound worker thread. The task backlog of each queue is continuously monitored based on a dynamic load balancing strategy. If the first queue is executed too quickly, resulting in an empty queue, to prevent the CPU core from being idle, the worker thread corresponding to the first queue selects the worker thread with the highest load, uses its microtask node queue as the second queue, and moves at least a portion of the microtask nodes from the second queue to its own first queue for execution. This not only balances the load of the multi-core CPU but also avoids resource contention for memory addressing with the original worker thread.

[0056] Optionally, sending the output data of the upstream node to the downstream node includes: obtaining the shared memory region between the upstream node and the downstream node; storing the output data in the shared memory region, obtaining a memory reference pointing to the output data, and sending the memory reference to the downstream node to trigger the downstream node to read the output data according to the memory reference.

[0057] In some embodiments, after the upstream node completes its computation, its output data is not serialized. Instead, a shared memory region is established between the upstream and downstream nodes using memory mapping technology. After the upstream node writes its output data into the shared memory region, it only obtains a memory reference pointing to the output data and sends the memory reference to the downstream node. Upon receiving the memory reference, the downstream node reads the output data in the shared memory region according to the memory reference. The lifetime of the shared memory is managed through a reference counting mechanism. When the reference count reaches zero, it indicates that all downstream nodes have finished reading, and the data in the memory is released, thereby avoiding the CPU memory bandwidth consumption and communication latency caused by deep copying.

[0058] Optionally, after generating the addressing information stored by each microtask node based on the directed acyclic graph, the method further includes: evaluating upstream and downstream nodes based on preset multi-dimensional indicators to obtain the current evaluation result, wherein the multi-dimensional indicators include data filtering rate, data overlap rate, and node congestion rate; if the data processing logic of the downstream node includes data filtering, and the current evaluation result includes the data filtering rate of the downstream node being greater than or equal to a preset filtering rate threshold, then the dependency relationship between the upstream and downstream nodes is adjusted, and the directed acyclic graph and addressing information are updated; if there are multiple downstream nodes, and the current evaluation result includes the data overlap rate between each downstream node being greater than or equal to a preset overlap rate threshold, then the same data processing logic is extracted from the data processing logic corresponding to each downstream node, and new microtask nodes are generated based on the extracted data processing logic; if the current evaluation result includes the node congestion rate of the downstream node being greater than or equal to a preset congestion rate threshold, then multiple new downstream nodes are generated according to the data processing logic encapsulated by the downstream nodes, and the addressing information of the upstream node is configured to point to each downstream node.

[0059] In some embodiments, the formula for calculating the data filtering rate includes: Formula (1) In formula (1), For data filtering rate, This represents the number of data entries filtered by the downstream node based on conditional decisions. This represents the number of data entries in the input data for the downstream node.

[0060] In some embodiments, if the current evaluation result includes a data filtering rate of the downstream node that is greater than or equal to the filtering rate threshold, it indicates that there is a large amount of invalid data in the output data of the upstream node, which generates unnecessary network transmission overhead. This triggers a logic pre-operation to adjust the dependency relationship between the upstream and downstream nodes. For example, the data filtering conditions in the downstream node are extracted and merged into the data processing logic of the upstream node, so that the upstream node performs filtering judgment before sending data and synchronously updates the directed acyclic graph and addressing information, thereby avoiding the flow of invalid data.

[0061] In some embodiments, the data overlap rate characterizes the proportion of similar or identical computational logic executed by downstream nodes of multiple parallel branches when processing data from the same source. For example, the data overlap rate can be quantified by comparing the abstract syntax tree or instruction signature of the data processing logic of each downstream node.

[0062] In some embodiments, if downstream node B and downstream node C both depend on upstream node A, and downstream node B and downstream node C contain the same "smoothing filter" logic, and the data overlap rate is greater than or equal to the overlap rate threshold, then the "smoothing filter" logic is extracted from the data processing logic corresponding to node B and node C respectively, and a new intermediate micro-task node X is generated according to the extracted data processing logic. Then, the data of upstream node A is output to node X, and node X distributes the processed data to nodes B and C, which have had their duplicate logic removed.

[0063] In some embodiments, node congestion rate characterizes the extent to which the processing capacity of a microtask node lags behind the data arrival rate. For example, node congestion rate is determined by quantifying the amount of unprocessed input data backed up in the lockless double-ended queue of the downstream node, or by quantifying the average dwell time of input data in the queue.

[0064] In some embodiments, if the input data of the downstream node is too large, resulting in a high node congestion rate, multiple identical new downstream node copies are generated in memory according to the data processing logic encapsulated by the downstream node; the addressing information of the corresponding upstream node is configured to point to these multiple new downstream nodes. For example, when the upstream node generates output data, it uses a round-robin allocation algorithm to transmit the output data to any downstream node, and when generating new output data, it transmits it to another downstream node, thereby avoiding node bottlenecks.

[0065] Combination Figure 2 As shown, this application provides a microtask scheduling method, including: Step S201: Obtain microtask configuration; The microtask configuration includes data processing logic; Specifically, the initial unit is encapsulated according to the microtask configuration, a computational state space and memory allocation area are allocated to the initial unit, and the data structure of the initial unit is optimized so that the memory usage of the initial unit is within a preset range, thus obtaining the microtask node. Step S202: Construct a directed acyclic graph based on the dependencies between each microtask node; Step S203: Generate the addressing information stored by each microtask node based on the directed acyclic graph; Based on the topology of the directed acyclic graph, the micro-task nodes with dependencies are identified as upstream nodes and their corresponding downstream nodes. The addresses of the downstream nodes are stored in the internal memory of the upstream nodes in the form of routing tables, thereby generating the addressing information stored by each micro-task node. Step S204: Monitor the events of the micro-task nodes in real time, and update the addressing information stored by each micro-task node according to the monitoring results. In response to the creation event of a new microtask node, the new microtask node is added to the directed acyclic graph, and the addressing information stored by each microtask node is updated according to the directed acyclic graph. In response to an update event of any microtask node, if the update event includes updating the dependency relationship corresponding to the microtask node, the directed acyclic graph is updated according to the updated dependency relationship, and the addressing information stored by each microtask node is updated according to the directed acyclic graph. In response to the deletion event of any microtask node, the microtask node is deleted from the directed acyclic graph, and the addressing information stored by each microtask node is updated according to the directed acyclic graph. Step S205: Obtain the real-time data stream containing data identifiers, and determine the target node from each microtask node according to the preset correspondence between the data identifiers and microtask nodes. Step S206: Use the real-time data stream as input data for the target node, and route and distribute the real-time data stream to the target node; Step S207: In response to the upstream node receiving the corresponding input data, execute the upstream node; The execution process of the upstream node includes processing the input data according to the data processing logic encapsulated by the upstream node to obtain the output data, and querying the downstream node from each micro-task node according to the addressing information stored in the upstream node. The upstream node is any microtask node; Step S208: Send the output data of the upstream node to the downstream node, triggering the downstream node to use the output data as the input data of the downstream node, and then proceed to step S207.

[0066] Combination Figure 3 As shown, this application provides a microtask scheduling system, including an acquisition module 301, a generation module 302, an execution module 303, and a sending module 304.

[0067] The acquisition module 301 is used to acquire multiple micro-task nodes, each of which encapsulates data processing logic.

[0068] The generation module 302 is used to construct a directed acyclic graph based on the dependency relationships between each microtask node, and to generate addressing information stored by each microtask node based on the directed acyclic graph.

[0069] The execution module 303 is used to execute the upstream node in response to the upstream node receiving the corresponding input data. The execution process of the upstream node includes processing the input data according to the data processing logic encapsulated by the upstream node to obtain output data, and querying the downstream node from each micro-task node according to the addressing information stored by the upstream node. The upstream node is any micro-task node.

[0070] The sending module 304 is used to send the output data of the upstream node to the downstream node, triggering the downstream node to use the output data as the input data of the downstream node.

[0071] The microtask scheduling system provided in this application acquires multiple microtask nodes encapsulated with data processing logic and constructs a directed acyclic graph (DAG) based on the dependencies between nodes. Addressing information is generated independently for each microtask node based on this DAG. When any microtask node receives input data and processes it to obtain output data, it queries downstream nodes based on its internally stored addressing information and directly sends the output data to the downstream nodes, thereby triggering the downstream nodes to use the output data as input data. Compared to centralized scheduling of microtask nodes, generating addressing information for each microtask node based on the DAG grants each microtask node autonomous addressing capabilities. Microtask nodes no longer need to send status reports or initiate routing query requests to any global central node; instead, they rely on their internally stored addressing information to push data to downstream nodes. This breaks the bottleneck of centralized scheduling, eliminates the concurrent processing bottlenecks and single-point failure risks associated with central scheduling nodes, and achieves extremely low-latency data-driven scheduling and extremely high scheduling throughput in scenarios with a large number of microtasks, thereby improving the scheduling efficiency of microtasks.

[0072] This application also provides an electronic device, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the above-described method.

[0073] Figure 4 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 4 The computer system 400 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0074] like Figure 4As shown, the computer system 400 includes a Central Processing Unit (CPU) 401, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 402 or programs loaded from storage portion 408 into Random Access Memory (RAM) 403. The RAM 403 also stores various programs and data required for system operation. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An Input / Output (I / O) interface 405 is also connected to the bus 404.

[0075] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to I / O interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 410 as needed so that computer programs read from it can be installed into storage section 408 as needed.

[0076] The electronic device disclosed in this embodiment includes a processor, a memory, a transceiver, and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and enable communication between them. The memory is used to store computer programs, the communication interface is used for communication, and the processor and transceiver are used to run the computer programs, causing the electronic device to perform the various steps of the above method. The above description and drawings fully illustrate the embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Unless explicitly required, individual components and functions are optional, and the order of operation may vary. Parts and subsamples of some embodiments may be included in or replace parts and subsamples of other embodiments. Moreover, the terminology used in this application is only for describing embodiments and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated subsamples, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other subsamples, wholes, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes the element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0077] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0078] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some sub-samples may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. Furthermore, the functional units in this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0079] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of the systems, methods, and computer program products according to this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than those disclosed in the description; sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A microtask scheduling method, characterized in that, include: Multiple micro-task nodes are obtained, wherein each micro-task node encapsulates data processing logic. A directed acyclic graph is constructed based on the dependencies between the micro-task nodes, and addressing information stored by each micro-task node is generated based on the directed acyclic graph. In response to the upstream node receiving the corresponding input data, the upstream node is executed. The execution process of the upstream node includes processing the input data according to the data processing logic encapsulated by the upstream node to obtain output data, and querying the downstream node from each of the micro-task nodes according to the addressing information stored by the upstream node. The upstream node is any micro-task node. The upstream node sends its output data to the downstream node, triggering the downstream node to use the output data as its input data.

2. The method according to claim 1, characterized in that, Obtain multiple microtask nodes, including: Obtain microtask configuration, wherein the microtask configuration includes data processing logic; The initial unit is encapsulated according to the microtask configuration; The initial unit is allocated a computational state space and a memory allocation area, and the data structure of the initial unit is optimized so that the memory usage of the initial unit is within a preset range, thus obtaining a microtask node.

3. The method according to claim 1, characterized in that, Based on the directed acyclic graph, addressing information stored by each of the microtask nodes is generated, including: Based on the topology of the directed acyclic graph, the micro-task nodes with dependencies are respectively identified as upstream nodes and downstream nodes corresponding to the upstream nodes; The addresses of the downstream nodes are stored in the internal memory of the upstream nodes in the form of a routing table to obtain the addressing information of the upstream nodes.

4. The method according to claim 3, characterized in that, The method further includes: In response to the creation event of a new microtask node, the new microtask node is added to the directed acyclic graph, and the addressing information stored by each microtask node is updated according to the directed acyclic graph. In response to an update event of any microtask node, if the update event includes updating the dependency relationship corresponding to the microtask node, the directed acyclic graph is updated according to the updated dependency relationship, and the addressing information stored by each of the microtask nodes is updated according to the directed acyclic graph. In response to the deletion event of any microtask node, the microtask node is deleted from the directed acyclic graph, and the addressing information stored by each microtask node is updated according to the directed acyclic graph.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Obtain a real-time data stream containing data identifiers; Based on the preset correspondence between the data identifier and the micro-task node, the target node is determined from each of the micro-task nodes; The real-time data stream is used as input data for the target node, and the real-time data stream is routed and distributed to the target node.

6. The method according to any one of claims 1 to 4, characterized in that, Executing the upstream node includes: Obtain multiple worker threads and the microtask node queues corresponding to each worker thread; The upstream nodes that receive input data are stored in each of the micro-task node queues, wherein any one of the micro-task node queues is designated as the first queue. If the first queue is an empty set, then a second queue containing micro-task nodes is determined from the micro-task node queues corresponding to each worker thread, and at least a portion of the micro-task nodes contained in the second queue are moved to the first queue. If the first queue contains microtask nodes, then the microtask nodes contained in the first queue are executed through the worker thread corresponding to the first queue.

7. The method according to any one of claims 1 to 4, characterized in that, Sending the output data of the upstream node to the downstream node includes: Obtain the shared memory region between the upstream node and the downstream node; The output data is stored in the shared memory area to obtain a memory reference pointing to the output data, and the memory reference is sent to the downstream node to trigger the downstream node to read the output data according to the memory reference.

8. The method according to any one of claims 1 to 4, characterized in that, After generating the addressing information stored by each of the microtask nodes based on the directed acyclic graph, the method further includes: The upstream and downstream nodes are evaluated based on preset multi-dimensional indicators to obtain the current evaluation result. The multi-dimensional indicators include data filtering rate, data overlap rate and node congestion rate. If the data processing logic of the downstream node includes data filtering, and the current evaluation result includes that the data filtering rate of the downstream node is greater than or equal to a preset filtering rate threshold, then the dependency relationship between the upstream node and the downstream node is adjusted, and the directed acyclic graph and the addressing information are updated. If there are multiple downstream nodes, and the current evaluation result includes a data overlap rate between each downstream node that is greater than or equal to a preset overlap rate threshold, then the same data processing logic is extracted from the data processing logic corresponding to each downstream node, and a new micro-task node is generated based on the extracted data processing logic. If the current evaluation result includes a downstream node congestion rate greater than or equal to a preset congestion rate threshold, then multiple new downstream nodes are generated according to the data processing logic encapsulated by the downstream node, and the addressing information of the upstream node is configured to point to each downstream node.

9. A microtask scheduling system, characterized in that, include: The acquisition module is used to acquire multiple micro-task nodes, wherein each micro-task node encapsulates data processing logic. The generation module is used to construct a directed acyclic graph based on the dependency relationships between the micro-task nodes, and to generate addressing information stored by each of the micro-task nodes based on the directed acyclic graph. An execution module is used to execute the upstream node in response to receiving corresponding input data. The execution process of the upstream node includes processing the input data according to the data processing logic encapsulated by the upstream node to obtain output data, and querying the downstream node from each of the micro-task nodes according to the addressing information stored by the upstream node. The upstream node is any micro-task node. The sending module is used to send the output data of the upstream node to the downstream node, triggering the downstream node to use the output data as its input data.

10. An electronic device, characterized in that, include: Processor and memory; The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to cause the electronic device to perform the microtask scheduling method according to any one of claims 1 to 8.

Images