Platform for constructing graphical multi-mode intelligent agent

Abstract

The invention discloses a graphical multi-modal agent-oriented construction platform, relates to the technical field of artificial intelligence and graphical programming, and is used for solving the problem of result timeliness caused by dependence on management during existing multi-task parallel execution. According to the platform, graphical programming interfaces of a programming area and a stage area are adopted, and the capabilities of large models such as vision, voice and text are packaged into large model building blocks capable of being dragged for communication; a large model communication unit, a communication state detection unit, a communication result data storage unit and a communication result data processing unit form a data closed loop, meanwhile, a heterogeneous task asynchronous collaborative scheduling module discovers a task parallel calling model with the in-degree being zero in real time, and result consistency is kept in combination with a logic clock version stamp and a failure event; the problems of timeliness and resource waste caused by multi-task dependence are solved, and code-free construction, elastic stretching and fault self-healing are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and graphical programming, and more particularly, to a graphical multi-modal agent construction platform. BACKGROUND

[0002] With the rapid iteration of large models (LLM) and generative AI, the traditional "block drag" graphical programming platform widely used for primary and secondary school maker education has begun to carry the development task of integrating AI capabilities such as vision, voice, and text. However, in the process of introducing AI capabilities into graphical teaching content, the traditional platform is difficult to build a complete process containing multi-modal agents, and the main reasons include:

[0003] Data transmission and retention lack a unified protocol, and model input and output are prone to format and version inconsistencies during cross-node transmission; data classification granularity is coarse, and cannot be finely annotated and permission isolated for multi-modal features; heterogeneous task parallel asynchronous execution lacks scheduling support, long and short tasks cannot be fully concurrent, and system resource utilization is low; fine scheduling and result synchronization mechanism are missing, and intermediate results completed first are prone to failure during waiting, affecting overall timeliness.

[0004] To solve the above problems, the present application provides a solution. SUMMARY

[0005] The purpose of the present application is to provide a graphical multi-modal agent construction platform for solving the problem of result timeliness caused by dependency management during multi-task parallel execution.

[0006] The purpose of the present application can be achieved by the following technical solutions:

[0007] The graphical multi-modal agent construction platform comprises a graphical programming interface including a programming area for editing graphical code and a stage area for displaying the execution effect of the graphical code, the programming area including a code area and a programming instruction block box, the programming instruction block box providing a plurality of programming instruction blocks including a communication large model communication instruction, and a user forming a graphical code by dragging the programming instruction blocks to the code area;

[0008] The communication large model block is presented in the form of a block in the programming area and is dragged to the code area to generate the graphical code, and can trigger communication with a large model server;

[0009] The large model communication unit is configured to receive a user-configured communication task through the graphical programming interface, establish a communication connection with a large model server, and execute the communication task;

[0010] The communication state detection unit is configured to detect whether the communication state of the large model communication unit after performing the communication task includes completion or non-completion, and generate data type identification;

[0011] The communication result data storage unit is configured to store the communication result data when the communication state detection unit detects that the communication task is completed.

[0012] The communication result data processing unit is configured to process the communication result data stored in the communication result data storage unit to obtain target communication result data, and the target communication result data is transmitted to the large model communication unit as input data of another communication task.

[0013] As a further scheme of the present application, the large model communication unit specifically comprises: receiving an arbitrary large model capability type identification corresponding to a communication task configured by a user through the graphical programming interface, and establishing a large model communication unit type library; defining a message format, a serialization mode and a communication state code for each large model communication unit type; and when receiving a current session context and a set of tasks to be processed, generating a corresponding number of large model communication unit instances by calling the communication unit type library.

[0014] As a further scheme of the present application, the communication state detection unit specifically comprises: the communication state detection unit continuously receives a heartbeat timestamp, real-time load and average processing delay reported by each large model communication unit instance through a bidirectional streaming channel, and analyzes the communication state to determine whether the communication state is completed; when the communication state is completed, a data type identification including a node identification, a logical version timestamp and a communication result data size is generated and written into an event bus to prompt the communication result data storage unit to save the communication result data and trigger the communication result data processing unit; when the communication state is not completed and the duration exceeds a preset timeout threshold, a retry event is generated and an exception type is recorded.

[0015] As a further scheme of the present application, the communication result data storage unit comprises: writing the communication result data smaller than a preset result standard value into a row-level transaction library, and writing the communication result data larger than the preset result standard value into an object storage and a logical version mapping table.

[0016] As a further scheme of the present application, it further comprises a heterogeneous task asynchronous cooperative scheduling module configured to obtain a task directed acyclic graph describing task nodes and their dependency relationships according to the graphical programming interface; and implement dynamic topology scanning to identify task nodes with an in-degree of zero in real time and write them into a ready queue.

[0017] The dynamic topology scanning includes two modes of automatic identification and manual annotation, and the manual annotation can modify the task dependency relationship through the graphical interface.

[0018] The tasks in the ready queue are executed in parallel by asynchronously calling the large model communication unit instance bound to the task node through the coroutine pool. Each task is accompanied by an assigned logical clock version stamp upon completion. After receiving a task completion event, the in-degree count of the relevant downstream task nodes in the directed acyclic graph of the task is incrementally updated, and the downstream task nodes with an in-degree updated to zero are written into the ready queue. When it is detected that the logical clock version stamp returned by a task is newer than the existing logical clock version, a failure event is issued to the downstream task node with the existing logical clock version stamp, its state is reset, and it is rewritten into the ready queue.

[0019] As a further aspect of the present invention: the dynamic topology scan includes: traversing the task directed acyclic graph and calculating the in-degree value of all task nodes; if the in-degree value of a task node is 0, it is determined that it does not depend on the output of any upstream task node and its state is recorded as ready and written into the runnable queue; otherwise, it is recorded as waiting and written into the suspended queue, while saving the unsatisfied dependency set.

[0020] The algorithm locates completed task nodes using an incremental approach; then iterates through all outgoing edges of the large model communication unit instance corresponding to the task node, decrementing the in-degree count of downstream task nodes by 1; if the in-degree of a downstream task node becomes 0, it means that all its dependencies have been satisfied, and it is immediately moved into the runnable queue and its state is changed to ready.

[0021] As a further aspect of the present invention: the asynchronous call of the coroutine pool includes: polling the runnable queue to determine the target large model communication unit instance for each popped task node based on the dependency relationship of the large model communication unit; creating an active coroutine and calling the corresponding large model communication unit instance through a bidirectional streaming channel; after the active coroutine is created, it enters a suspended state until the target large model communication unit instance generates result data and a logical clock version stamp.

[0022] As a further aspect of the present invention: after the heterogeneous task asynchronous collaborative scheduling module, a waiting configuration module is also included to record the start waiting time of the waiting task nodes, and calculate the waiting time every preset analysis period. By using the fusion of the moving average and ARIMA dual models, the maximum value of the two is taken as the remaining time of the upstream task node; the scheduling priority is adjusted or recalculation is triggered according to the preset multi-level lag threshold.

[0023] As a further aspect of the present invention: the step of triggering the adjustment of scheduling priority or triggering recalculation based on the preset multi-level lag threshold specifically includes: comparing the waiting time corresponding to the task node in the waiting state and the predicted remaining time of the upstream task node with the preset three-level lag threshold and the preset maximum waiting time, respectively; the three-level lag threshold is set as the first-level lag threshold, the second-level lag threshold, and the third-level lag threshold proportionally based on the preset maximum waiting time.

[0024] If the waiting time is greater than or equal to the first-level lag threshold and the predicted remaining time of the upstream task node is greater than a certain multiple of the maximum waiting time, then the corresponding waiting task node is marked as slightly lagging; and only the priority of the target task node in the runnable queue is increased, without triggering recalculation; where is a preset proportional parameter, which splits the maximum waiting time into two decision lines;

[0025] If the waiting time is greater than or equal to the second-level lag threshold and the predicted remaining time of the upstream task node is greater than the maximum waiting time, then the corresponding waiting task node is marked as moderately lagging; and the slowest upstream task node is triggered to send part of the recalculation.

[0026] If the waiting time is greater than or equal to the level 3 lag threshold, the corresponding waiting task node will be marked as severely lagging and all ancestor task nodes that are less than or equal to the target task node will be fully recalculated.

[0027] As a further aspect of the present invention: the triggering recalculation further includes: when the task node triggering the recalculation outputs a new logical clock version stamp, if the downstream task node is in a failed state, its in-degree is reset and it is re-added to the runnable queue after all dependencies are updated.

[0028] If the downstream task node is still waiting and the lag level has fallen below the first-level lag threshold, then the lag mark is revoked and the original priority is restored.

[0029] If the same upstream task node fails to recalculate multiple times in a row, the active recalculation trigger will execute the fault degradation compensation mechanism, which will inject default placeholder results into all downstream task nodes waiting for the upstream task node; compensation calculation will be performed after the node recovers.

[0030] The beneficial effects of this invention are:

[0031] (1) This invention uses a heterogeneous task asynchronous collaborative scheduling module that includes a dynamic topology scanning algorithm and a ready queue mechanism to scan the directed acyclic graph of tasks in real time, identify task nodes with an in-degree of zero, and add them to the ready queue; the coroutine pool then asynchronously and in parallel calls the tasks in the ready queue, allowing tasks without dependencies, such as image recognition and speech synthesis, to be executed concurrently; after a task is completed, the in-degree of downstream tasks is incrementally updated, and new tasks with an in-degree of zero immediately enter the ready queue; the timing of task startup and actual dependency satisfaction is decoupled, avoiding resource idleness and inefficiency caused by linear serial execution; and the problem of unnecessary waiting caused by strict global order of multiple parallel tasks is solved.

[0032] (2) This invention solves the problem of early-completed results becoming invalid due to excessive waiting time caused by significant differences in task completion times, by combining a version consistency submodule based on logical clock version stamps and a failure event mechanism with a dynamic rescheduling submodule. Each task is accompanied by a logical clock version stamp upon completion; when a task returns a new version stamp, the version management module immediately broadcasts a failure event to downstream tasks that have read the old results. Downstream tasks are interrupted, their status is reset (including in-degree rollback), and they are re-enqueued for execution after all their dependencies are updated and ready; the dynamic rescheduling module prioritizes scheduling these failed tasks to ensure they are quickly recalculated using the latest input; this mechanism effectively prevents downstream tasks from operating on outdated data due to updates in upstream task results, ensuring the correctness and timeliness of the final result. Attached Figure Description

[0033] The invention will now be further described with reference to the accompanying drawings.

[0034] Figure 1 This is a schematic diagram of the framework structure of the graphical multimodal intelligent agent construction platform of the present invention;

[0035] Figure 2 This is a schematic diagram of the platform architecture for building graphical multimodal intelligent agents according to the present invention;

[0036] Figure 3 This is a schematic diagram illustrating the implementation process of the graphical multimodal intelligent agent construction platform of the present invention;

[0037] Figure 4 This is an example diagram showing the mapping of multiple dependency edges in the directed acyclic graph of tasks in the heterogeneous task asynchronous collaborative scheduling module of this invention.

[0038] Figure 5 This is a schematic diagram of the recalculation logic triggered in the configuration module of this invention;

[0039] Figure 6 This is a schematic diagram illustrating the core functions of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] This invention provides a graphical multimodal intelligent agent construction platform. Its main purpose is to encapsulate heterogeneous large models such as vision, speech, and text into draggable communication unit blocks through a visual programming interface. This enables teenagers, educators, and industry developers without programming experience to quickly build, debug, and deploy intelligent agents with complex reasoning and generation capabilities without writing code by hand. This overcomes the technical problems of existing graphical programming systems, such as the lack of large model connection methods, high development threshold, and low resource utilization efficiency. Furthermore, it promotes the popularization of artificial intelligence education and the agile iteration of enterprise-level multimodal applications.

[0042] Example 1

[0043] Please see Figures 1-2 As shown, the present invention is a graphical multimodal intelligent agent construction platform, including: a graphical programming interface, communication large model building blocks, large model communication unit, communication status detection unit, communication result data storage unit, and communication result data processing unit;

[0044] The graphical programming interface mainly includes a programming area for editing graphical code and a stage area for displaying the execution effect of the graphical code. The programming area includes a code area for displaying the graphical code and a programming instruction block box for providing programming instruction blocks for the code area.

[0045] The graphical programming interface is used to communicate with the large model server. Combined with code logic processing, the development of AI agent is finally realized. The communication large model block can be dragged to the code area to generate the graphical code.

[0046] The graphical programming interface provides a module registration interface that encapsulates multimodal large-scale model capabilities such as visual recognition, speech understanding, and text generation into draggable, communicative large-scale model building blocks with unified input and output specifications, and writes them into the module metadata repository. Developers can place and connect these building blocks in the code area by dragging and dropping them. Users dragging in building blocks instantiates task nodes, and connecting them generates dependency edges, building tasks in real time and assigning a unique identifier and capability tag to each task node. At the same time, the platform derives the required data flow, control flow, and resource quotas based on node dependencies and pre-sets an initial mapping table for heterogeneous task asynchronous collaborative scheduling modules. Users can open the parameter panel for the selected module in the same code area to visually configure parameters such as model version, inference accuracy, caching strategy, and timeout threshold. All parameter modifications are recorded as a retrospective version snapshot.

[0047] Based on the above principles, users can complete the entire closed loop of task parameter setting, task drag-and-drop arrangement, operation monitoring, and result reuse in the graphical programming interface. It should be emphasized that the graphical programming interface is only responsible for the graphical configuration and persistence of task dependencies and parameters, and does not directly encapsulate or execute any large model inference capabilities.

[0048] The communication large model building block is a number of large model communication units formed by dividing the capabilities of each model. Each large model communication unit independently encapsulates the corresponding large model capabilities and allows for visual drag-and-drop calls.

[0049] The capabilities of the large model include, but are not limited to, image recognition and generation, speech synthesis, and dialogue language modeling.

[0050] In this embodiment, the large communication model building block is designed as the platform's smallest running entity, encapsulating the generation capabilities of heterogeneous models such as vision, speech, and text, and writing it into the task metadata table during instantiation to complete the unified mapping from model capabilities to atomic task handles; as the platform's smallest running entity, the large communication model building block only accepts scheduling and executes model inference during runtime.

[0051] Please refer to Figure 3 S100-S400, the large model communication unit, is used to receive communication tasks configured by the user through the graphical programming interface, establish a communication connection with the large model server, and execute the communication tasks.

[0052] The information input from the large model building blocks is collected and input into the large model communication unit. The message is sent to the cloud server, and then the cloud server transmits the information to the large model communication unit deployed in the cloud computing server.

[0053] The information input to the large communication model building block includes: parameter values ​​set by various communication units when triggered, communication result data stored when the communication result data storage unit is embedded in the large model communication unit, and data that has been changed between the large model communication units;

[0054] Furthermore, the parameter values ​​set when various large model communication units are triggered include: AI painting model related to the painting model, image size, and image content; speech synthesis model related to AI speech synthesis, synthesis speed, synthesis tone, synthesis volume, synthesis timbre, and synthesis content; and dialogue model related to the large language model, whether context is enabled, whether web search is enabled, and dialogue content.

[0055] Preferably, the communication unit type definition platform receives the capability type identifier of any large model communication unit defined by the user through the graphical interface; and establishes a large model communication unit type library CU = {CU in CU sched CU mergeCU out , ...}; where CU in For input preprocessing, the type identifier of the large model communication unit, CU sched For the type identification of communication units in the large inference scheduling model, CU merge The results aggregate the type identifier of the large model communication unit, CU out This serves as the type identifier for the large model communication unit in the output post-processing; and for each type of large model communication unit, a message format Fmsg, a serialization method Fser, and a communication status code Fstat are defined to ensure interoperability and consistency of cross-node transmission.

[0056] When this platform receives the current session context ω and the set of tasks to be processed Τ, it calls the communication unit type library to generate a corresponding number of large model communication unit instances {CU}. i |i∈[1,n]};where i is the index of the large model communication unit instance, n is the number of large model communication unit instances, and during instantiation, for each communication unit instance CU i Configure runtime parameters such as model version, batch processing dimension, resource tags, and timing constraints to meet the requirements of heterogeneous hardware and multi-tenant isolation.

[0057] It should be noted that, in this embodiment, the session context ω refers to the set of runtime context information formed by the user during a continuous operation in the graphical programming interface. It includes the currently built communication unit topology, the configuration parameters of each communication unit, and the intermediate results cached upstream, in order to accurately describe the platform's input and dependency relationship to subsequent model calls at that moment.

[0058] The set of task nodes to be processed, T, refers to the list of model calls to be executed obtained by the platform based on the session context ω. Each list element corresponds to a reasoning or generation request for a predetermined large model and carries the target model identifier, request body and service quality constraints to guide the scheduler to create parallel large model communication unit instances.

[0059] Furthermore, the platform maps the various model communication unit instances to independent computing nodes via the cluster resource manager and establishes a point-to-point high-speed interconnection channel; the mapping relationship can be represented as f:CU i →〈Node k GPU m >; where k and m are the task node and accelerator card numbers, ensuring that there are no port conflicts within the same task node and that the network bandwidth is greater than or equal to the preset minimum bandwidth; the accelerator card number identifies the specific GPU / NPU bound within the computing node;

[0060] This ensures that the processing progress and status can be tracked in real time during the information processing of large models; by continuously querying the cloud server, the processing results are obtained in a timely manner, ensuring the continuity and accuracy of the process; and by storing the results in the communication result data storage unit, a data caching and sharing mechanism is provided for the subsequent steps that may require the use of these results, enabling different large models to further interact and collaborate based on previous processing results.

[0061] The communication status detection unit continuously receives heartbeat timestamps, real-time load, and average processing latency reported by each large model communication unit instance via a bidirectional streaming channel, and parses the communication status code to determine whether the communication status is complete. When the communication status is complete, it generates a data type identifier containing a node identifier, logical version stamp, and the size of the communication result data, and writes it to the event bus to prompt the communication result data storage unit to save the communication result data and trigger the communication result data processing unit. When the communication status is incomplete and the duration exceeds a preset timeout threshold, it generates a retry event and records the exception type. Specifically, it registers heartbeat information (heartbeat timestamp, real-time load, average processing latency) with the global scheduler in all large model communication unit instances; and generates capacity distributions based on the real-time load and average processing latency. The scheduler first calculates the load and latency scores. Then, it combines these scores with a pre-defined load coefficient and latency coefficient, linearly weighting them to generate a comprehensive weight that represents the instance's current attractiveness to new tasks. For scenarios prioritizing low latency, the latency coefficient can be set higher; for those requiring higher throughput, the load coefficient can be increased. After weight calculation, the scheduler selects the target instance and routes requests immediately, either using the highest weight priority or a random selection based on weight ratio. If the target instance's status code suddenly changes to abnormal, a second selection is performed to ensure requests do not reach faulty task nodes. Each time routing is completed, the target instance's load count increases instantly. After processing and returning results, the scheduler updates the instance's average latency, reflecting the latest operational status in real-time during the next weight calculation. Through this continuous cycle, the scheduler automatically decides to scale up or down based on throughput trends, triggering self-healing operations when necessary to smoothly migrate traffic to backup instances and restart failed task nodes, maximizing service continuity and overall performance.

[0062] Based on the preset monitoring period ΔT, heartbeat detection, compute node liveness checks, and log anomaly retrieval are performed on the communication unit instances of each model; when the detection results meet the FaultFlag... i When = 1, the scheduler immediately triggers failover, including switching current traffic to the backup large model communication unit instance (CU). r Where r is the identifier of the standby large model communication unit instance; restart or replace the faulty large model communication unit instance CU. iUpdate the large model communication unit instance mapping table and weight matrix to ensure business continuity.

[0063] This embodiment also includes the continuous reporting of throughput, average latency, and memory utilization as communication result data by each large model communication unit instance during operation; the monitoring center identifies bottleneck units based on a sliding window strategy and dynamically adjusts the batch size or increases the number of large model communication unit instances to achieve adaptive scaling. Furthermore, a unique tenant label is assigned to each large model communication unit instance, and data channels are restricted through policy-based access control lists (ACLs), allowing only calls between the same tenant or authorized different tenants.

[0064] The communication result data storage unit in the extension of different large model communication unit instances will store the corresponding communication result data; the communication result data storage units between different large model communication units do not interfere with each other; the communication result data storage unit that has stored the communication result once will overwrite the previously stored result when the corresponding large model communication unit is called again;

[0065] In summary, this embodiment achieves high parallelism, elastic scaling, and fault self-healing capabilities on the platform through a series of steps, including defining the type of communication units of multiple models, instantiating them, deploying them in parallel, load balancing, health checks, and performance optimization. This provides stable, scalable, and efficient underlying support for the construction of multimodal intelligent agents.

[0066] The communication result data storage unit is used to store and record the communication result data after the large model communication unit performs the communication task with the large model server and the data type of the communication status detection unit is represented as communication completion;

[0067] Specifically: when the communication status detection unit confirms the target node's status as complete through continuous heartbeat monitoring, it immediately generates a completion event (node ​​identifier, version stamp, and communication result data size) and writes it to the event bus; the event consumer result writer is then awakened; the result writer first reads the communication result data size and selects a writing strategy accordingly: if the communication result data is less than the preset result standard value, an insert statement is constructed to write to the row-level transaction database, committing the fields "result text, version stamp, and generation time" at once; if the communication result data is greater than the preset result standard value, the MinIO software development kit is first called to upload the binary file to the object bucket, obtain the object key, and then write it to the logical version mapping table, establishing a unique mapping between the logical version and the object key; the entire database writing process is completed within a distributed transaction, ensuring the atomicity and consistency of the result file and metadata;

[0068] After successful writing, the result writer triggers a post-processing trigger within the same transaction's post-hook. The trigger reads the function subscription rules registered in the component registry and compares each one against the input pattern. For any function that declares it can consume the node's output format, a post-processing event (function identifier, task identifier, node identifier) ​​is generated. The event is dispatched to the Knative function pool where the communication result data processing unit resides. The processing unit first retrieves the results according to the event information: direct queries for results located in the row-level transaction database; and downloads results located in object storage to the sandbox directory by object key. Then, it executes calculations based on user-written aggregation, statistical, or visualization logic to produce new data. If the output again meets the upstream link's re-call conditions, the processing unit will also return a next call event to the scheduler, completing the result loop.

[0069] The communication result data processing unit is used to perform specific function processing on the data stored in the communication result data storage unit to obtain target communication result data for presentation or to use the obtained target communication result data as input to establish a communication connection between the large model communication unit and the large model server.

[0070] Once the function finishes execution and the result writer is called again to write the result to the database, the platform pushes the final visible object to the stage area in real time through the result channel. The stage area user interface locates the canvas node based on the node identifier, switches its status from "running" to "completed", and displays a summary report, line chart, or agent reply text in the visualization output area. If an anomaly occurs in the function chain, a red alarm pops up simultaneously, and an error log is printed on the terminal panel. Through the five tightly linked steps of "detection → writing → triggering → processing → presentation", the platform ensures that every inference result is traceable, versioned, and scalable, and is fed back to the user's view with minimal latency.

[0071] Furthermore, this embodiment also includes a heterogeneous task asynchronous collaborative scheduling module, which includes a task directed acyclic graph modeling submodule, a parallel execution control submodule, a dependency graph management submodule, a version consistency submodule, and a dynamic rescheduling submodule;

[0072] Please see Figure 4 The example shown is a mapping of multiple dependency edges in a directed acyclic graph (DAG). When the user drags blocks to form business logic in the graphical interface, the task graph modeling submodule maps all major model communication units to task nodes (TN) in real time. Simultaneously, while the user drags and connects lines in the graphical interface, it also maps all major model communication units to task nodes and parses the connection relationships between task nodes. Each vertex in the DAG is defined as a task node (TN), and each directed edge is defined as a dependency edge (DE). Furthermore, based on the task node type, the corresponding major model communication unit category is determined and written into the node-unit binding table.

[0073] Furthermore, the dynamic topology scan algorithm is used to calculate the in-degree remainder in real time, specifically as follows:

[0074] Traverse the directed acyclic graph of tasks and calculate the in-degree of all task nodes TN. If the in-degree of a task node TN is 0, it is determined that it does not depend on the output of any upstream task node and can be executed concurrently immediately after the agent starts. Therefore, its state is marked as ready and written into the runnable queue as Q. exec Otherwise, it is marked as waiting and written to the pending queue as Q. wait Meanwhile, it saves the set of unsatisfied dependencies; after the task graph modeling submodule obtains the directed acyclic graph of the task, it writes the task node information into the task node record and the edge information into the dependency edge record; both tables are persisted in a distributed key-value store.

[0075] As an optional example, the heterogeneous task asynchronous collaborative scheduling module of this invention supports flexible switching between fully automatic mode and manual intervention mode. Manual operation, as a supplement to automated scheduling, is used to handle special scenarios that dynamic topology scanning cannot cover (such as temporarily inserting high-priority tasks or correcting dependency errors). Both are synchronized through a unified task directed acyclic graph (DAG) data structure. That is, the dynamic topology scanning can also include both automatic identification and manual annotation modes. Manual annotation allows modification of task dependencies through a graphical interface.

[0076] By employing a dual-mode scheduling approach (manual / automatic), this invention maintains the high efficiency of automated scheduling while addressing the issue of insufficient scenario adaptability in traditional systems due to over-reliance on preset rules. For example, when a task cannot be automatically triggered due to external data delays, the user can manually force the release of downstream tasks, thus shortening the task execution cycle.

[0077] The graphical programming interface provides manual annotation functionality, allowing users to modify the dependencies of task nodes through drag-and-drop operations. The system updates the in-degree state of the DAG in real time and distinguishes manually adjusted nodes on the interface with different colors (e.g., manually modified dependency edges are highlighted in red).

[0078] The coroutine pool asynchronous call module supports manual interface triggering. Users can click the 'Execute Now' button on the task node details page to force the node to be added to the ready queue, and the in-degree value of downstream tasks will be automatically updated. This operation does not affect the consistency check of the logical clock version stamp.

[0079] Asynchronous calls to coroutine pools are performed using a coroutine pool approach, including polling the runnable queue Q. execFor each pop-up task node, determine the target large model communication unit instance based on the node-unit binding table; create an active coroutine and call the corresponding large model communication unit instance through a bidirectional streaming channel; immediately after the active coroutine is created, relinquish the CPU and enter suspension until the target large model communication unit instance generates the result data;

[0080] After the target large model communication unit instance generates the result, it returns the result data and logical clock version stamp through a bidirectional streaming channel; after the active coroutine receives the response, it generates a completion event and writes it to the event bus, and at the same time writes the result to the result storage area; the logical clock version stamp is a globally unique incrementing version identifier; the bidirectional streaming refers to a bi-directional streaming RPC connection based on gRPC.

[0081] The dependency graph maintenance submodule subscribes to task completion events in real time. For any task event, it first sets the state of the corresponding large model communication unit instance to "completed," then it traverses all outgoing edges of the large model communication unit instance, decrementing the in-degree count of downstream task nodes by 1. If the in-degree of a downstream task node becomes 0, it means that all its dependencies have been satisfied, and it is immediately moved into the runnable queue Q. exec And change the state to ready; the time complexity of this incremental update algorithm is proportional to the number of outgoing edges, avoiding a full scan of the directed acyclic graph of the complete task;

[0082] The version consistency submodule maintains a global version index table; whenever a new completion event is recorded, it compares the returned logical clock version stamp RV with the old version number in the index; if the new version number is greater, it updates the index and broadcasts the version update event.

[0083] For a downstream task node that is running and has a stored logical clock version stamp input, a failure event is issued, the control unit is instructed to interrupt the corresponding active coroutine in parallel execution, and the state of the task node is reset to waiting, with its in-degree rolled back as needed.

[0084] The dynamic rescheduling submodule simultaneously monitors failure events, recalculation events, and large model communication unit instance fault alarms.

[0085] If any event is received, the scheduling priority of the affected task node is raised to the highest level, and the global scheduler is invoked to update the weight parameters, so that it has priority in obtaining the large model communication unit instance in the next round of resource contention; if the original execution large model communication unit instance crashes and cannot migrate its context, the task node is directly regenerated and put into the runnable queue Q. exec Re-execute;

[0086] When the parallel execution control unit detects that the heartbeat of the large model communication unit instance has been lost or the global scheduler actively pushes a fault alarm, the dynamic rescheduling unit immediately marks all associated task nodes as failed and generates a recalculation event.

[0087] For example, suppose a task node and There are no prerequisites. rely , After startup , Simultaneously enter the runnable queue Each of them calls different large model communication unit instances. When Upon completion, the dependency graph maintenance unit will... The in-degree is reduced by 1, but it is still pending. Finish, Keep waiting. Soon... The update was triggered again, generating a new version. Greater than Version consistency management unit broadcasts version update events and enables... The existing coroutine is invalid. (To be continued) Completed and Once the new results are ready, Re-added to the runnable queue The output is recalculated using the latest input to ensure consistency.

[0088] Please see Figure 5 and Figure 6 ,in Figure 5 This diagram illustrates the logic for triggering recalculation in the configuration waiting module. Figure 6 The core technology modules 101-107 are shown, and the module also includes a waiting configuration module for detecting the waiting time of waiting task nodes and predicting the remaining time of upstream tasks, and dynamically adjusting the scheduling priority or triggering recalculation events based on multi-level thresholds.

[0089] The waiting configuration module continuously monitors task nodes that have entered a waiting state in the heterogeneous task asynchronous collaborative scheduling module, records their initial waiting time, calculates the elapsed waiting time every preset analysis time (e.g., 3 seconds), and predicts the remaining time of upstream task nodes; it also triggers adjustments to scheduling priorities or recalculation events based on preset three-level lag thresholds, including:

[0090] The waiting time of the task node in the waiting state and the predicted remaining time of the upstream task node are compared with the preset three-level lag threshold and the preset maximum waiting time, respectively. The three-level lag threshold is set proportionally to the preset maximum waiting time as the first-level lag threshold β1, the second-level lag threshold β2, and the third-level lag threshold β3.

[0091] If the waiting time is greater than or equal to the first-level lag threshold β1 and the predicted remaining time of the upstream task node is greater than θ times the maximum waiting time, then the task node in the corresponding waiting state is marked as slightly lagging; where θ is a preset proportional parameter, the specific value of which is set by relevant field personnel based on experience, and the maximum waiting time is divided into two judgment lines.

[0092] If the waiting time is greater than or equal to the second-order lag threshold β2 and the predicted remaining time of the upstream task node is greater than 1-θ times the maximum waiting time, then the task node in the corresponding waiting state is marked as moderately lagging.

[0093] If the waiting time is greater than or equal to the level 3 lag threshold β3, then the corresponding waiting task node is marked as severely lagging.

[0094] It should be noted that the prediction of the remaining time of the upstream task node adopts a fusion of short-term moving average and medium-term ARIMA dual models, and the maximum value of the two is taken as the prediction of the remaining time of the upstream task node; the above-mentioned short-term moving average and medium-term ARIMA dual models are existing mature technologies and will not be described in detail in this embodiment.

[0095] Furthermore, when a waiting task node is marked as slightly lagging, only the priority of the target task node in the runnable queue is increased, without triggering a recalculation;

[0096] When a waiting task node is marked as moderately lagging, a partial recalculation is sent to the slowest upstream task node, and the weight of that task node is increased to the highest value in the cluster to preempt GPU resources.

[0097] When a waiting task node is marked as severely delayed, a full recalculation is sent in parallel to all ancestor task nodes that are less than or equal to d from the target task node. In this embodiment, d = 2 by default. The task node in the current waiting state is marked as invalid to ensure that the coroutine becomes invalid after the old activity.

[0098] When the task node that issued the recalculation event outputs the new logical clock version stamp RV new If a downstream task node is in a failed state, its in-degree will be reset and it will be put back into the runnable queue after all dependencies have been updated.

[0099] If the downstream task node is still waiting and the lag level has fallen below the first-level lag threshold β1, then the lag mark is revoked and the original priority is restored.

[0100] If the same upstream task node fails to recalculate twice in a row, the active recalculation trigger will execute the fault degradation compensation mechanism, which will inject default placeholder results into all downstream task nodes waiting for the upstream task node; compensation calculation will be performed after the node recovers, thus avoiding long-term blockage of the entire link.

[0101] The invention also includes a graphical dependency monitoring module, which displays the running status and performance indicators of each module in real time, enabling visualized configuration and monitoring of the entire system process.

[0102] The graphical dependency monitoring module mainly includes: a status display submodule and a parameter hot-tuning submodule; the specific implementation of this module is as follows:

[0103] The status display submodule is used to display the running status of each task node in real time. Specifically, it includes whether the task node is in different states such as ready, running, waiting, recalculating, or completed, as well as information such as the current processing progress, the number of times it has been completed, and the type of the most recent error.

[0104] The status display submodule can also generate independent alarm prompts for each task node, such as timeout, excessive recalculation count, and outdated dependency results, and promptly notify users in the interface via pop-ups or bubbles. It also supports one-click jump to detailed logs and diagnostic pages to help users quickly locate anomalies.

[0105] The parameter hot-tuning submodule allows users to dynamically adjust key scheduling parameters of each node during system operation, including but not limited to scheduling priority, maximum concurrency, and resource tags. When parameters are modified, the parameter hot-tuning unit automatically assembles update instructions and pushes them to the backend scheduler via a message queue, achieving seamless hot updates of scheduling layer parameters. Upon receiving the update instructions, the backend takes effect immediately without interrupting the current task flow, thus greatly improving the system's flexibility and adaptability.

[0106] The above sub-modules are interconnected through an internal bus and event-driven mechanism, realizing a closed-loop management of configuration, display, hot-tuning, and feedback. Every time a user drags and drops a configuration, monitors the status, or adjusts a parameter, it is reflected synchronously in the front-end interface and the back-end scheduler, ensuring the consistency and efficiency of task process collaboration.

[0107] In specific implementation, the operation process of the graphical dependency monitoring module includes the following steps:

[0108] S1: The status display submodule periodically subscribes to the running events of each node and automatically refreshes the color and progress information of each node;

[0109] S2: When an operational anomaly or alarm condition is detected, the platform will pop up a notification to the user, who can then view the cause and suggestions with one click.

[0110] S3: The parameter hot-tuning submodule allows users to adjust parameters at any time. All adjustments take effect immediately on the backend and are visually reflected on the frontend.

[0111] This module not only provides users with highly visual and interactive task flow configuration tools, but also greatly improves the ease of use, flexibility, and reliability of the intelligent agent platform through real-time status and alarm monitoring. Users can orchestrate complex task dependencies and optimize performance without writing code, and can promptly detect and respond to various abnormal states during runtime.

[0112] This invention encapsulates large model capabilities such as vision, speech, and text into draggable atomic modules to construct a directed acyclic graph of tasks. The heterogeneous task asynchronous collaborative scheduling module discovers nodes with zero in-degree in real time through dynamic topology scanning and uses a coroutine pool to call the large model communication unit in parallel. After the task is completed, it uses a logical clock version stamp to incrementally update dependencies and broadcast failure events to avoid global blocking and result expiration, significantly improving collaborative efficiency and ensuring correct and timely results. It also provides drag-and-drop configuration, runtime monitoring, and parameter hot-tuning to achieve no-code construction, elastic scaling, and fault self-healing.

[0113] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0114] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0115] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and inventive constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0116] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0117] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0118] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A graphical multimodal intelligent agent construction platform, characterized in that: include: The graphical programming interface includes a programming area for editing graphical code and a stage area for displaying the execution effect of the graphical code. The programming area includes a code area and a programming instruction block box. The programming instruction block box provides a variety of programming instruction blocks, including communication instructions for a large communication model. Users can form graphical code by dragging and dropping the programming instruction blocks to the code area. The communication large model building blocks are presented in the form of building blocks in the programming area and can be dragged to the code area to generate the graphical code and trigger communication with the large model server. The large model communication unit is used to receive communication tasks configured by the user through the graphical programming interface, establish a communication connection with the large model server, and execute the communication tasks. The communication status detection unit is used to detect the communication status (complete or incomplete) after the large model communication unit performs the communication task, and to generate a data type identifier; The communication result data storage unit is used to store communication result data when the communication status detection unit detects that the communication task is completed; The communication result data processing unit is used to process the communication result data stored in the communication result data storage unit to obtain target communication result data, which is then passed into the large model communication unit as input data for another communication task.

2. The graphical multimodal intelligent agent construction platform according to claim 1, characterized in that, The large model communication unit specifically comprises: receiving the arbitrary large model capability type identifier corresponding to the communication task configured by the user through the graphical programming interface, and establishing a large model communication unit type library; defining the message format, serialization method and communication status code for each large model communication unit type; and when receiving the current session context and the set of tasks to be processed, calling the communication unit type library to generate a corresponding number of large model communication unit instances.

3. The graphical multimodal intelligent agent construction platform according to claim 1, characterized in that, The communication status detection unit specifically receives the heartbeat timestamp, real-time load, and average processing latency reported by each large model communication unit instance through a bidirectional streaming channel, and parses the communication status code to determine whether the communication status is complete. When the communication status is complete, it generates a data type identifier containing the node identifier, logical version stamp, and the size of the communication result data and writes it to the event bus to prompt the communication result data storage unit to save the communication result data and trigger the communication result data processing unit. When the communication status is incomplete and the duration exceeds the preset timeout threshold, a retry event is generated and the exception type is recorded.

4. The graphical multimodal intelligent agent construction platform according to claim 1, characterized in that, The communication result data storage unit includes: writing the communication result data into the row-level transaction database when it is less than the preset result standard value, and writing the communication result data into the object storage and the logical version mapping table when it is greater than the preset result standard value.

5. The graphical multimodal intelligent agent construction platform according to claim 1, characterized in that, It also includes a heterogeneous task asynchronous collaborative scheduling module that communicates with the large model communication unit, used to obtain a directed acyclic graph of tasks describing task nodes and their dependencies based on a graphical programming interface; and to perform dynamic topology scanning to identify task nodes with an in-degree of zero in real time and write them into the ready queue. The dynamic topology scan includes two modes: automatic identification and manual annotation. Manual annotation allows modification of task dependencies through a graphical interface. The tasks in the ready queue are executed in parallel by asynchronously calling the large model communication unit instance bound to the task node through the coroutine pool. Each task is accompanied by an assigned logical clock version stamp upon completion. After receiving a task completion event, the in-degree count of the relevant downstream task nodes in the directed acyclic graph of the task is incrementally updated, and the downstream task nodes with an in-degree updated to zero are written into the ready queue. When it is detected that the logical clock version stamp returned by a task is newer than the existing logical clock version stamp, a failure event is issued to the downstream task node with the existing logical clock version stamp, its state is reset, and it is rewritten into the ready queue.

6. The graphical multimodal intelligent agent construction platform according to claim 5, characterized in that, The dynamic topology scan used to update the in-degree of task nodes includes: traversing the directed acyclic graph of tasks and calculating the in-degree value of all task nodes; if the in-degree value of a task node is 0, it is determined that it does not depend on the output of any upstream task node and its state is recorded as ready and written into the runnable queue; otherwise, it is recorded as waiting and written into the suspended queue, while saving the set of unsatisfied dependencies. The algorithm locates completed task nodes using an incremental approach; then iterates through all outgoing edges of the large model communication unit instance corresponding to the task node, decrementing the in-degree count of downstream task nodes by 1; if the in-degree of a downstream task node becomes 0, it means that all its dependencies have been satisfied, and it is immediately moved into the runnable queue and its state is changed to ready.

7. The graphical multimodal intelligent agent construction platform according to claim 6, characterized in that, The asynchronous call to the coroutine pool includes: polling the runnable queue to determine the target large model communication unit instance for each popped task node based on the dependency relationship of the large model communication unit; creating an active coroutine and calling the corresponding large model communication unit instance through a bidirectional streaming channel; and suspending the active coroutine after its creation until the target large model communication unit instance generates the result data and logical clock version stamp.

8. The graphical multimodal intelligent agent construction platform according to claim 5, characterized in that, The heterogeneous task asynchronous collaborative scheduling module also includes a waiting configuration module to record the start waiting time of waiting task nodes and calculate the waiting time every preset analysis period. By using a sliding average and ARIMA dual model fusion, the maximum value of the two is taken as the remaining time of the upstream task node. The scheduling priority is adjusted or a recalculation is triggered according to the preset multi-level lag threshold.

9. The graphical multimodal intelligent agent construction platform according to claim 8, characterized in that, The step of triggering scheduling priority adjustment or triggering recalculation based on preset multi-level lag thresholds specifically includes: comparing the waiting time of the task node in the waiting state and the predicted remaining time of the upstream task node with preset three-level lag thresholds and preset maximum waiting time, respectively; the three-level lag thresholds are set proportionally based on the preset maximum waiting time as first-level lag thresholds, second-level lag thresholds, and third-level lag thresholds. If the waiting time is greater than or equal to the first-level lag threshold and the predicted remaining time of the upstream task node is greater than a certain multiple of the maximum waiting time, then the corresponding waiting task node is marked as slightly lagging; and only the priority of the target task node in the runnable queue is increased, without triggering recalculation; where is a preset proportional parameter, which splits the maximum waiting time into two decision lines; If the waiting time is greater than or equal to the second-level lag threshold and the predicted remaining time of the upstream task node is greater than the maximum waiting time, then the corresponding waiting task node is marked as moderately lagging; and the slowest upstream task node is triggered to send part of the recalculation. If the waiting time is greater than or equal to the level 3 lag threshold, the corresponding waiting task node will be marked as severely lagging and all ancestor task nodes that are less than or equal to the target task node will be fully recalculated.

10. The graphical multimodal intelligent agent construction platform according to claim 9, characterized in that, The triggering of recalculation also includes: when the task node that triggers recalculation outputs a new logical clock version stamp, if the downstream task node is in a failed state, its in-degree will be reset and it will be put back into the runnable queue after all dependencies are updated. If the downstream task node is still waiting and the lag level has fallen below the first-level lag threshold, then the lag mark is revoked and the original priority is restored. If the same upstream task node fails to recalculate multiple times in a row, the active recalculation trigger will execute the fault degradation compensation mechanism, which will inject default placeholder results into all downstream task nodes waiting for the upstream task node; compensation calculation will be performed after the node recovers.

Images