An extension method for pipeline parallel scheduling execution
By employing flexible scheduling strategies and dynamic expansion methods in the database system, the problem of limited parallelism in the pipeline was solved, enabling flexible control of parallelism and ordered data output, thereby improving processing efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING JUYUN WEIZHI INFORMATION TECH CO LTD
- Filing Date
- 2022-11-27
- Publication Date
- 2026-07-03
AI Technical Summary
In traditional database systems, each transformer in the pipeline can only be driven by one worker thread, which limits the degree of parallelism. Especially when the data scale is large and the number of queries is limited, the number of threads becomes the processing bottleneck. Furthermore, the amount of input data that the transformer needs to process is unpredictable, leading to insufficient memory and data accumulation on disk.
A flexible scheduling strategy is adopted, a thread pool is created through the PipelineExecutor unit, the internal structure of the pipeline is dynamically expanded, and the transformer state is managed by the worker unit and TaskQueue, so as to achieve flexible control of parallelism and adaptive scheduling.
This system enables pipelines generated by different queries to use the same degree of parallelism, dynamically expands the pipeline structure to adapt to concurrency requirements under different states, ensures the overall orderliness of the output data, and solves the problem of limited parallelism.
Smart Images
Figure CN115718659B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of database system technology, specifically to an extended method for parallel scheduling and execution of pipelines. Background Technology
[0002] Once created, a pipeline cannot be modified, and each transformer in the pipeline can only be driven by one worker thread at a time. In traditional database systems, [reference needed]. Figure 1 In general, a query processing flow is: Parse—Plan—Execute—Storage; in the Plan stage, a Pipeline assembly is often added (one transformer represents one data processing).
[0003] All transformers are orchestrated into a pipeline and then handed over to executors for execution. Each time a transformer is executed, the dataset is processed and output, continuing down to the downstream sinker. In some database scenarios, the total number of queries is limited, but the data scale is large, making it crucial to maximize the parallelism of each query-related pipeline. Each transformer in the pipeline can only be driven and executed by one thread at a time.
[0004] The amount of input data that a transformer needs to process is difficult to know when building a pipeline. Some transformers require all input data before they can actually process and output it (such as sort). However, memory is limited, so the input data needs to be written to disk after reaching a certain amount, which may result in a lot of data accumulating on the disk. After all the input data is obtained, the transformer actually has a lot of work to do, but a transformer can only be handled by one thread, and the number of threads will become the bottleneck of the transformer's processing capacity. Summary of the Invention
[0005] (a) Technical problems to be solved
[0006] To address the shortcomings of existing technologies, this invention provides an extension method for parallel scheduling and execution of pipelines. In this proposal, after a pipeline is handed over to an executor, the executor requests a thread pool with a fixed number of threads. The executor employs a flexible scheduling strategy, selecting an appropriate transformer and assigning it to worker thread groups for processing. This ensures that pipelines generated from different queries can utilize the same degree of parallelism, and the degree of parallelism is controllable. During the pipeline's scheduling and execution by the executor, its internal structure can be dynamically expanded to adapt to the concurrent processing requirements of units in different states, thus solving the problems mentioned in the background technology.
[0007] (II) Technical Solution
[0008] To achieve the above objectives, the present invention provides the following technical solution:
[0009] An extension method for parallel scheduling and execution of a pipeline includes: Step 1, creating a PipelineExecutor unit. The pipeline unit is generated externally, with the number of worker units passed in from the external source. The PipelineExecutor unit creates a corresponding number of worker units and threads, with each worker unit associated with one thread; Step 2, finding transformer units with a "Ready" state from the leaf nodes of the pipeline unit and placing them into multiple queues in the TaskQueue unit in a robin manner; the transformer unit contains multiple states, and the worker units determine the processing of the transformer unit based on the state of the transformer unit;
[0010] Step 3: Retrieve transformer units from TaskQueue units via worker units. When a worker unit retrieves a transformer unit from a TaskQueue unit, it prioritizes retrieving it from its own associated Queue unit. If the Queue unit does not have a transformer unit, it retrieves it from another Queue unit. Alternatively, it can drive the transformer unit to process the data by calling the worker unit within the transformer unit. Step 4: Collect the changed ports units within the processed transformer units. A transformer unit itself may be associated with multiple input port units and multiple output port units. Clear the changed state of the port units to facilitate processing by the transformer unit's next worker unit function.
[0011] Furthermore, the transformer unit includes the following states: NeedData state: The transformer unit needs some input data to continue working; PortFull state: The transformer unit cannot continue working because the output is full; Finished state: All work has been completed and no further work is needed; Ready state: The transformer unit will work synchronously.
[0012] Furthermore, the transformer unit, the actual data processing unit, is broadly categorized into source units (without input), sink units (without output), and transformer units (with input and output). During processing, the transformer unit interacts with its own port unit. When the state of the PortState unit associated with the port unit changes, the port unit will mark itself as having changed.
[0013] Furthermore, the port unit is the corresponding entity for the input and output of the transformer unit, associated with upstream and downstream transformer units, and manages the input and output data; the InputPort unit stores records of the OutputPort units associated with it; the OutputPort unit records of the InputPort units associated with it; the associated InputPort units and OutputPort units are all associated with the same PortState unit object.
[0014] Furthermore, the PortState unit object consists of its own state and its internal data; wherein, the IS_NEEDED state indicates that the internal data of the PortState is empty and data needs to be provided from the outside; the HAS_DATA state indicates that the internal data of the PortState exists and the outside can retrieve the data for processing; the IS_FINISHED state indicates that the external processing has been completed and the PortState does not need to store or retrieve data; the data in the PortState is put into it by its associated OutputPort and taken out by its associated InputPort; the externally related transformer unit determines its own state through the state of the PortState object in its associated Port.
[0015] Furthermore, the status of the processed transformer unit is checked to determine whether to put this transformer unit into the TaskQueue unit. After the worker unit finishes processing a transformer unit, its status is checked again through the prepare state of this transformer unit. When the transformer unit is in the Ready state, it is put into the TaskQueue unit and its associated Queue unit.
[0016] Furthermore, the transformer unit adds other transformer units to the pipeline unit; the prepare interface in the transformer unit is used to return the current state of the transformer unit; the work unit interface in the transformer unit is used to process the input data internally when the transformer unit is in the Ready state, and put the processed data into the relevant OutputPort unit.
[0017] Furthermore, within the pipeline unit, the processing of a piece of data needs to be completed in multiple steps, with each step corresponding to a transformer unit, and the transformer units are interdependent. There is also a workergroup unit, which consists of multiple threads, with the specific number of threads specified externally. Each worker unit is associated with a thread, which selects the transformer unit to process the data from the pipeline unit based on the current state of the transformer units, and drives the transformer unit to process the data.
[0018] Furthermore, transformer units in the Ready state form TaskQueue units, which are shared by all worker units. Each TaskQueue unit contains multiple Queue units, with the number of Queue units matching the number of worker unit data. Each worker unit is associated with one Queue unit. When a worker unit finds a transformer in the Ready state, it only places it into its own associated Queue unit. The PipelineExecutor unit handles the scheduling and execution of pipeline units, and workergroup units are created internally by the PipelineExecutor unit.
[0019] (III) Beneficial Effects
[0020] This invention provides an extended method for parallel scheduling and execution of pipelines, which has the following beneficial effects:
[0021] By employing a flexible scheduling strategy through the executor, a suitable transformer can be selected and handed over to the worker thread group for processing. This allows different query-generated pipelines to use the same degree of parallelism, and the degree of parallelism is controllable. During the execution of the pipeline by the executor, the internal structure of the pipeline can be dynamically expanded to adapt to the concurrent processing requirements of operators in different states.
[0022] The system uses a merge sort to globally sort multiple inputs, ensuring the overall order of the output data. Data is received from the last input port and written to the first output port. Data is taken from all inputs (multiple lines related to them in the pipeline), and the merge sort is used to globally sort multiple inputs, ensuring the overall order of the output data. Finally, the overall ordered data is output to the downstream.
[0023] The pipeline can be scheduled and executed according to a specified degree of parallelism, regardless of its simplicity or complexity. If some transformers in the pipeline require the full input data to process and the total amount of input data is uncertain, the transformer can be dynamically expanded to change its parallel processing capability. Attached Figure Description
[0024] Figure 1 This is a schematic diagram of the existing Query processing flow in this invention;
[0025] Figure 2 This is a schematic diagram of a pipeline scheduling and execution scheme that already exists in this invention;
[0026] Figure 3 This is a schematic diagram of another existing pipeline scheduling and execution scheme according to the present invention;
[0027] Figure 4 This is a schematic diagram of an execution flow for the pipeline scheduling with controllable concurrency of the present invention;
[0028] Figure 5 This is a schematic diagram of another execution structure for the pipeline scheduling with controllable concurrency of the present invention;
[0029] Figure 6 This is a schematic diagram of the dynamic expansion process in the pipeline scheduling execution of the present invention. Detailed Implementation
[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0031] Among the existing technologies, the following technical solutions exist:
[0032] The design concept of Option 1 is shown in the figure. Figure 2 As shown: Step 1, abstract each transformer unit in the pipeline unit into an iterator; Step 2, each iterator has a next unit method; Step 3, each call to this method will return the data generated by this pipeline unit; Step 4, a thread continuously calls the next unit method at the root node of the tree to obtain all the results of the entire query.
[0033] The design concept of Option 2 is shown in the figure. Figure 3 As shown, the design idea is as follows: Step 1: Based on Scheme 1, in order to improve the parallelism of pipeline unit execution, each transformer unit is driven by a separate thread; Step 2: The output of the transformer unit is put into a queue, and the next transformer unit takes data from the queue as its own input.
[0034] From the perspective of existing technical solutions, the above two technical solutions have the following problems:
[0035] Option 1: The abstraction of the transformer unit is better, but the entire pipeline unit can only be driven by one worker thread to execute, which cannot be used on a single machine with multiple cores;
[0036] Option 2: Based on Option 1, it improves concurrency; however, the number of transformer units that make up the pipeline unit varies depending on the complexity of the query; the occupation of thread resources on a single machine by a single pipeline unit is completely uncontrollable.
[0037] Please see Figure 4 and Figure 5 This invention provides an extended method for parallel scheduling and execution of pipeline units. This technical solution involves the following modules or units:
[0038] Transformer unit: The actual data processing unit, which can be broadly classified into source units (without input), sink units (without output), and transformer units (with input and output).
[0039] The transformer unit contains several states, and the worker unit determines how to process the transformer unit based on these states: NeedData state: The transformer needs some input data to continue working; PortFull state: The transformer cannot continue working because the output is full; Finished state: All work has been completed (all data has been processed or all outputs have been closed), and no further work is needed; Ready state: The transformer will perform some synchronous work.
[0040] ExpandPipeline Unit: The transformer unit wants to add other transformer units to the pipeline unit; the prepare unit interface in the transformer unit is used to return the current state of the transformer unit; the work unit interface in the transformer unit is used to process the input data internally when the transformer unit is in the Ready state, and put the processed data into the relevant OutputPort unit.
[0041] Port unit: The corresponding entity for the input and output of the transformer unit, associated with upstream and downstream transformer units, and manages input and output data; the InputPort unit records the OutputPort unit associated with it; the OutputPort unit records the InputPort unit associated with it; the associated InputPort unit and OutputPort unit are all associated with the same PortState unit object.
[0042] A PortState object consists of its own state and its internal data;
[0043] IS_NEEDED status: Indicates that the data inside PortState is empty and data needs to be provided from outside.
[0044] HAS_DATA status: indicates that the data inside the PortState exists and can be retrieved and processed externally;
[0045] IS_FINISHED state: indicates that the external processing has been completed and PortState no longer needs to access the data;
[0046] Data in PortState is entered by its associated OutputPort and retrieved by its associated InputPort.
[0047] Externally related transformers determine their own state based on the state of the PortState object in their associated Port.
[0048] Pipeline unit: The processing of a piece of data needs to be completed in multiple steps, each step corresponds to a transformer unit, and there are dependencies between transformer units; the pipeline unit is the overall representation of how to process this data.
[0049] A workergroup unit consists of multiple threads, the specific number of which is specified externally. Each worker group is associated with one thread. The main responsibility of a worker group is to select a suitable transformer unit from the pipeline units based on the current state of the transformer units, and drive the transformer unit to process the data.
[0050] TaskQueue unit: Stores transformer units in the Ready state. All worker units share the same TaskQueue unit.
[0051] The TaskQueue unit contains multiple Queue units, with the number of Queue units being the same as the number of worker units. Each worker unit is associated with one Queue unit. When a worker unit finds a transformer in the Ready state, it only puts it into its own associated Queue unit. The PipelineExecutor unit handles the scheduling and execution of pipeline units. The workergroup unit is created internally by the PipelineExecutor unit.
[0052] Based on the above, the overall processing flow of the extended methods in pipeline parallel scheduling execution is as follows:
[0053] Creating and initializing PipelineExecutor units: PipelineExecutor units are created, while pipeline units are generated externally;
[0054] PipelineExecutor unit initialization: The number of worker units is passed in from the outside. The PipelineExecutor unit creates the corresponding number of worker units and threads, with each worker unit associated with one thread. It searches for transformer units with a Ready status from the leaf nodes in the pipeline unit and puts them into multiple queues in the TaskQueue unit in a robin manner.
[0055] Furthermore, the worker unit retrieves the transformer unit from the TaskQueue unit for processing.
[0056] Furthermore, when a worker unit retrieves a transformer unit from a TaskQueue unit, it prioritizes retrieving it from its own associated Queue unit. If there is no transformer unit in this Queue unit, it retrieves it from other Queue units. Alternatively, it can drive the transformer unit to process the data by calling the worker unit within the transformer unit.
[0057] During the processing, the transformer unit interacts with its own port unit. When the state of the PortState unit associated with the port unit changes, the port unit will mark itself as changed for subsequent processing.
[0058] At the same time, the status of the processed transformer unit is checked to determine whether to put this transformer unit into the TaskQueue unit. After the worker unit finishes processing a transformer unit, its status is checked again through the prepare state of this transformer unit. When the transformer unit is in the Ready state, it is put into the TaskQueue unit and its associated Queue unit.
[0059] Furthermore, collect the ports that have changed within the processed transformer unit; where the transformer unit itself may be associated with multiple input port units and multiple output port units, identify and record the port units that changed during processing from these port units of the transformer unit.
[0060] Furthermore, clearing the change status of the port unit facilitates processing by the transformer unit in the next work unit function.
[0061] When in use, the associated transformer unit is obtained through these changed port units. The prepare unit method of the transformer unit is used to determine whether these transformer units need to be added to the TaskQueue unit, and to determine whether the pipeline scheduling execution is complete.
[0062] When all Queue units in the TaskQueue unit are empty and all worker units are in an idle state (no transformer units are being processed and they are waiting for new transformer units to be added to the TaskQueue unit), it means that the pipeline unit scheduling and execution is complete. When the cancel unit in the PipelineExecutor unit is actively called from outside, it is necessary to wait for all the tasks currently being processed by all worker units to finish before the pipeline unit scheduling and execution are completely completed.
[0063] In an extended method for parallel pipeline scheduling, the main logic is as follows: The sort unit of the orderby unit is used as an example to illustrate the processing mechanism of the expand unit. The overall logic of this pipeline unit is to output the data in an orderly manner to the downstream unit. The process first ensures the overall order within a single line of the pipeline unit, and then merges the data from all lines of the pipeline unit to ensure overall order. The main reason for adding the expand unit is that the amount of data within a single line of the pipeline unit may be very large. Data exceeding a certain amount needs to be written to disk, and many files may accumulate on the disk. After all the data on a single line is received, these written files need to be processed concurrently to ensure that the data within a single line of the pipeline unit is output to the downstream unit in an orderly manner.
[0064] It should be noted that:
[0065] PartialSortingTransform unit: Each piece of data input to this unit is not ordered. This unit needs to sort the data internally to ensure that the output to downstream units is an ordered data block, in which the data block is ordered.
[0066] MergeSortingTransform unit: Each data block is ordered within the input data block. This unit needs to process all input data blocks, sort all data blocks as a whole, and ensure that the output is ordered data after all data blocks are sorted.
[0067] The BufferingToFileTransform unit: Input data is provided by the MergeSortingTransform unit. The BufferingToFileTransform unit is responsible for writing the data to disk and then loading the data.
[0068] MergeSortingTransform unit: There is a certain threshold limit to the amount of data that can be stored in memory. When this threshold is exceeded, the data in memory needs to be sorted as a whole and then written to disk using the BufferingToFileTransform unit. Whenever the threshold is exceeded and disk write is required, the BufferingToFileTransform unit will add an outputport unit, which will be associated with a newly created BufferingToFileTransform unit.
[0069] The ordered data provided by the MergeSortingTransform unit will be divided into multiple blocks by the BufferingToFileTransform unit and written to the same file.
[0070] The MergeSortedTransform unit has multiple inputs, each of which is ordered as a whole. This unit needs to sort the multiple ordered inputs as a whole to ensure that the output is ordered data as a whole.
[0071] The PartialSortingTransform unit ensures that each input data block is ordered; while the MergeSortingTransform unit receives multiple ordered data blocks and stores them in its internal buffer.
[0072] The specific workflow is as follows:
[0073] During the creation of the expand unit, new Transform units are added, and the relationships between the new Transform units are properly handled; furthermore, a new BufferingToFileTransform unit is created, and an inputport unit and an outputport unit are created for it; a new outputport unit is created and associated with the inputport unit of the newly created BufferingToFileTransform unit;
[0074] If the internal MergeSortedTransform unit is not created, a new MergeSortedTransform unit is created, and an outputport unit is added to it; a new inputport unit is added to the MergeSortingTransform, and the outputport unit of the internal MergeSortedTransform unit is associated with the newly added inputport unit of the MergeSortingTransform unit.
[0075] Furthermore, a new inputport unit is created for the internal MergeSortedTransform unit, and the outputport of the newly created BufferingToFileTransform unit is associated with the newly created inputport unit of the MergeSortedTransform unit; when the worker unit in the PipelineExecutor unit calls the prepare unit interface of the MergeSortingTransform unit, the status of the ExpandPipeline unit is returned.
[0076] When a worker unit within the PipelineExecutor unit discovers that a Transform unit is in the ExpandPipeline state, it calls the expandPipeline method of the Transform unit. The expandPipeline method of the MergeSortingTransform unit returns the newly created Transform unit. The worker unit within the PipelineExecutor unit receives these newly added Transform units and checks their state to see if they are Ready through the prepare method of these Transform units, deciding whether to add them to the TaskQueue unit.
[0077] The MergeSortedTransform unit takes data from all inputs and performs a global sorting of multiple inputs through a merging method to ensure that the output data is generally ordered.
[0078] The MergeSortingTransform unit receives data from the last input port unit and writes it to the first output port unit. The MergeSortingTransform unit takes data from all inputs (multiple lines in the pipeline) and performs global sorting on the multiple inputs through merging to ensure that the output data is ordered as a whole. Finally, the ordered data is output to the downstream unit.
[0079] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.
[0080] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0081] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0082] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division of an extended method for pipeline parallel scheduling execution, such as a waterway underwater terrain change analysis system and method. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.
[0083] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0084] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0085] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0086] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0087] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. An extended method for parallel scheduling and execution of pipelines, characterized in that: include: Step 1: PipelineExecutor unit creation. The pipeline unit is generated externally, with the number of worker units passed in from the external source. The corresponding number of worker units and threads are created within the PipelineExecutor unit, with each worker unit associated with one thread. Step 2: Find the transformer units with the state "Ready" from the leaf nodes of the pipeline unit, and put them into multiple queues in the TaskQueue unit in a robin manner; The transformer unit contains multiple states, and the worker unit determines how to process the transformer unit based on the transformer unit's state. Step 3: The worker unit retrieves the transformer unit from the TaskQueue unit for processing. When the worker unit retrieves the transformer unit from the TaskQueue unit, it first retrieves it from its own associated Queue unit. If there is no transformer unit in this Queue unit, it retrieves it from other Queue units. Alternatively, it can drive the transformer unit to process the data by calling the worker unit in the transformer unit. Step 4: Collect the ports that have been changed within the processed transformer unit; the transformer unit itself may be associated with multiple input port units and multiple output port units; clear the changed status of the port units to facilitate processing by the transformer unit's next work unit function.
2. The extended method for parallel scheduling and execution of a pipeline according to claim 1, characterized in that: The transformer unit has the following states: NeedData state: The transformer unit needs some input data to continue working; PortFull state: The transformer unit cannot continue working because the output is full; Finished state: All work has been completed and no further work is needed; Ready state: The transformer unit will work synchronously.
3. The extended method for parallel scheduling and execution of a pipeline according to claim 2, characterized in that: The transformer unit is the actual data processing unit, which is broadly classified into source units that do not contain input, sink units that do not contain output, and transformer units that contain input and output. During the processing, the transformer unit will interact with its own port unit. When the state of the PortState unit associated with the port unit changes, the port unit will mark itself as having changed.
4. The extended method for parallel scheduling and execution of a pipeline according to claim 3, characterized in that: The port unit is the corresponding entity for the input and output of the transformer unit, and it is associated with upstream and downstream transformer units for the management of input and output data. The InputPort unit stores records of the OutputPort units associated with it. The OutputPort unit records of the InputPort units associated with it. The associated InputPort units and OutputPort units are all associated with the same PortState unit object.
5. An extended method for parallel scheduling and execution of a pipeline according to claim 4, characterized in that: The PortState unit object consists of its own state and its internal data; wherein... IS_NEEDED status: Indicates that the data inside PortState is empty and data needs to be provided from outside. HAS_DATA status: indicates that the data inside the PortState exists and can be retrieved and processed externally; IS_FINISHED state: indicates that the external processing has been completed and PortState does not need to access the data; Data in PortState is put into it by its associated OutputPort and taken out by its associated InputPort; the external transformer unit determines its own state by the state of the PortState object in its associated Port.
6. The extended method for parallel scheduling and execution of a pipeline according to claim 3, characterized in that: The status of the processed transformer unit is checked to determine whether to put the transformer unit into the TaskQueue unit. After the process of a transformer unit is completed by the worker unit, its status is checked again by the prepare state of the transformer unit. When the transformer unit is in the Ready state, it is put into the TaskQueue unit and its associated Queue unit.
7. An extended method for parallel scheduling and execution of a pipeline according to claim 6, characterized in that: The transformer unit adds other transformer units to the pipeline unit; the prepare interface in the transformer unit is used to return the current state of the transformer unit; the work unit interface in the transformer unit is used to process the input data internally when the transformer unit is in the Ready state, and put the processed data into the relevant OutputPort unit.
8. The extended method for parallel scheduling and execution of a pipeline according to claim 1, characterized in that: In a pipeline unit, the processing of a piece of data needs to be completed in multiple steps, each step corresponding to a transformer unit, and the transformer units depend on each other. There is also a worker group unit: it consists of multiple threads, the specific number of which is specified externally. Each worker unit is associated with a thread, which selects the transformer unit to process the data from the pipeline unit based on the current state of the transformer units.
9. An extended method for parallel scheduling and execution of a pipeline according to claim 8, characterized in that: Transformer units in the Ready state form TaskQueue units, which are shared by all worker units. Each TaskQueue unit contains multiple Queue units, with the number of Queue units matching the number of worker units. Each worker unit is associated with one Queue unit. When a worker unit finds a Transformer in the Ready state, it only places it into its own associated Queue unit. The PipelineExecutor unit handles the scheduling and execution of pipeline units. Workergroup units are created internally by the PipelineExecutor unit.