A multi-modal agent based process automation method and system

Abstract

The application discloses a process automation method and system based on a multi-modal intelligent agent, and belongs to the field of artificial intelligence and process automation. The method comprises collecting multi-modal process data and preprocessing; constructing a knowledge graph and a vector index, obtaining an evidence set through mixed retrieval; using a RAG-ReAct dual-engine collaborative reasoning to process the evidence set to generate an executable process plan, and calling an external tool to execute the plan; performing consistency detection and recursive reflection on the reasoning and execution process to generate reflection data and scheduling parameters; training the reasoning engine based on the reflection data, and performing parameter directional injection on the reasoning engine; dynamically scheduling and scaling the computing resources based on the scheduling parameters; constructing a reward signal, and continuously optimizing the process automation method through reinforcement learning. The application solves the problems that the traditional government and enterprise process automation method cannot effectively cope with cross-modal semantic fusion, evidence traceability, complex reasoning consistency and high concurrency stability.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and process automation technology, specifically relating to a process automation method and system based on multimodal intelligent agents. Background Technology

[0002] In the field of government and enterprise process automation, traditional technical solutions mainly rely on rule engines, robotic process automation (RPA), or business process management (BPM) systems. These systems primarily use structured form fields and fixed interfaces as inputs, and achieve automated execution through manually configured process orchestration and rule logic. However, with the diversification of material formats in government and enterprise processes (such as text policies, scanned documents, images, tables, audio and video, logs, etc.) and frequent policy updates, process inputs have gradually evolved into multimodal and heterogeneous data. Traditional methods struggle to effectively address challenges such as cross-modal semantic fusion, evidence traceability, consistency of complex reasoning, and high-concurrency stability.

[0003] While traditional technical solutions have achieved partial automation of processes in different scenarios, they still suffer from the following common technical shortcomings: First, their multimodal semantic fusion capabilities are insufficient, making it difficult to establish unified semantic alignment between policy clauses, material evidence, and process nodes, leading to unstable extraction, omissions, and misjudgments. Second, their knowledge update and retrieval strategies lack adaptability, making it difficult to dynamically adjust in the face of frequent policy changes and task distribution variations, affecting the accuracy and timeliness of recall. Third, their reasoning process lacks verifiable mechanisms, one-time generated results lack evidentiary constraints, and tool calls and observations do not form a traceable closed loop, making it difficult to support auditing and review. Fourth, their stability is insufficient in high-concurrency scenarios, with uneven node load, communication overhead, and hotspot sharding issues causing latency fluctuations, and a lack of parallel detection and scheduling mechanisms oriented towards conflict, dependency, and granularity. Fifth, their decision-making methods are difficult to continuously optimize, and execution quality and user feedback are difficult to distill into trainable signals, lacking a closed loop of reflection, distillation, and strategy updates, thus limiting the system's self-improvement capabilities. Summary of the Invention

[0004] The purpose of this invention is to provide a process automation method and system based on multimodal intelligent agents, so as to achieve unified representation and auditable sedimentation of multimodal materials, verifiable reasoning and replayable execution, and continuous optimization of decision-making methods, while ensuring stable operation in high-concurrency scenarios.

[0005] In a first aspect, the present invention provides a process automation method based on a multimodal intelligent agent, comprising the following steps:

[0006] Collect multimodal process data, preprocess and uniformly represent the multimodal process data, and generate a unified data object;

[0007] Knowledge graphs and vector indexes are built or updated based on unified data objects, and evidence sets are obtained through hybrid retrieval.

[0008] The system employs a dual-engine collaborative reasoning approach, combining a retrieval-enhanced generation engine and a tool-invoking inference engine, to align and constrain the evidence set, generate an executable process plan, and invoke external tools to execute the executable process plan.

[0009] Consistency checks and recursive reflections are performed on the reasoning and execution processes to generate reflection data, and scheduling parameters are generated based on the dependencies and granularity factors in the reasoning and execution processes.

[0010] Based on reflective data, the inference engine is trained or fine-tuned through cognitive distillation, and parameters are injected into the trained inference engine in a targeted manner.

[0011] Based on scheduling parameters and real-time performance metrics, computing resources are dynamically scheduled and scaled up or down.

[0012] Reward signals are constructed based on business metrics and language auditability metrics, and process automation methods are continuously optimized through reinforcement learning.

[0013] As an alternative implementation, the multimodal process data is preprocessed and uniformly represented to generate a unified data object, including:

[0014] Standardize the process data of different modalities to generate standardized loads;

[0015] Standardized loads are subjected to quality gating, and low-quality data is sent to the supplementary collection or verification queue.

[0016] Extract semantic representations of each modality, and select a fusion strategy based on task requirements, data quality, and historical contribution to generate a unified representation;

[0017] Generate a unified data object containing standardized payloads, process metadata, audit tags, and evidence citation identifiers.

[0018] As an alternative implementation, a dual-engine collaborative reasoning approach involving a retrieval-enhanced generation engine and a tool-invoking inference engine is employed, including:

[0019] The retrieval enhancement generation engine binds citation identifiers to evidence fragments in the evidence set and constrains its output to carry citation identifiers;

[0020] The tool invocation inference engine breaks down the task objective into a sequence of executable steps that includes the tool type, input parameters, and observation write-back strategy, and calls external tools in sequence.

[0021] When the results of the tool call conflict with the evidence cited or the evidence is insufficient, a secondary search or a correction prompt is triggered, and the next round of search and execution iteration begins.

[0022] As an alternative implementation method, consistency checks and recursive reflections are performed on the reasoning and execution processes to generate reflection data, including:

[0023] Perform citation coverage verification, evidence conflict detection, and version consistency verification on the inference results and execution observations, and output conflict points and correction suggestions;

[0024] The conflict points and corrective suggestions are reflected upon at the result level, process level, and strategy level, and the reflection results are precipitated into structured reflection data.

[0025] As an alternative implementation, based on reflective data, the inference engine is trained or fine-tuned through cognitive distillation, and parameters are injected into the trained inference engine in a targeted manner, including:

[0026] Reflective data, verifiable evidence, and correct trajectories are used to construct teacher signals to train students' reasoning models.

[0027] By biasing the structured knowledge extracted from the knowledge graph into gradient directions, the parameters of the inference engine are updated in a targeted manner.

[0028] As an alternative implementation method, dynamic scheduling and scaling of computing resources are performed based on scheduling parameters and real-time performance metrics, including:

[0029] It receives scheduling parameters and real-time operational metrics including node load, latency, and conflict risks, scores candidate nodes, and dynamically adjusts request routing, sharding concurrency, and the number of computing resource replicas based on the scoring results.

[0030] As an alternative implementation method, a reward signal is constructed based on business metrics and language auditability metrics, and a process automation method is continuously optimized through reinforcement learning, including:

[0031] The business rewards are constructed from the first-time completion rate, rework rate, timeliness, and human intervention rate of the process, while the language rewards are constructed from the completeness of citations and clarity of expression. Based on the weighted sum of the business rewards and language rewards, the weights of hybrid retrieval, the weights of feature fusion, and the method of tool invocation are updated through the strategy gradient algorithm.

[0032] Secondly, the present invention provides a process automation system based on a multimodal intelligent agent, comprising:

[0033] The data acquisition and preprocessing module is configured to: acquire multimodal process data, preprocess and uniformly represent the multimodal process data, and generate a unified data object;

[0034] The knowledge graph and index building module is configured to: build or update knowledge graphs and vector indexes based on unified data objects, and obtain evidence sets through hybrid retrieval;

[0035] The dual-engine collaborative reasoning module is configured to: use a retrieval-enhanced generation engine and a tool-invoking reasoning engine for dual-engine collaborative reasoning, perform evidence alignment and citation constraints on the evidence set, generate an executable process plan, and call external tools to execute the executable process plan;

[0036] The detection and reflection module is configured to: perform consistency detection and recursive reflection on the reasoning and execution process, generate reflection data, and generate scheduling parameters based on the dependencies and granularity factors in the reasoning and execution process;

[0037] The distillation and injection module is configured to: train or fine-tune the inference engine based on reflective data through cognitive distillation, and inject parameters into the trained inference engine in a targeted manner.

[0038] The resource scheduling module is configured to dynamically schedule and scale up / down computing resources based on scheduling parameters and real-time performance metrics.

[0039] The model optimization module is configured to construct reward signals based on business metrics and language auditability metrics, and continuously optimize process automation methods through reinforcement learning.

[0040] Thirdly, the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0041] Fourthly, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0043] This invention proposes a process automation method based on multimodal intelligent agents. By constructing a unified representation and auditable sedimentation mechanism for multimodal materials, combined with RAG-ReAct dual-engine collaborative reasoning, consistency detection, and recursive reflective distillation, it achieves an end-to-end closed loop of traceable evidence, verifiable reasoning, and replayable execution. Furthermore, it introduces a distributed parallel detection and resource scheduling mechanism to dynamically adjust routing and concurrency strategies to ensure throughput and stability in high-concurrency scenarios. Finally, it continuously optimizes retrieval weights, fusion strategies, and tool invocation paths by integrating language and business rewards through reinforcement learning. Compared to existing technologies, this proposal significantly improves multimodal material processing capabilities and the auditability of the reasoning process, enhances the system's stability and scalability in high-concurrency environments, and reduces human intervention and compliance risks through a continuous strategy evolution mechanism. It provides auditable, traceable, and scalable intelligent support for government and enterprise process automation, demonstrating significant technological advancement and commercial application value. Attached Figure Description

[0044] Figure 1 This is an overall architecture diagram of a process automation system based on a multimodal intelligent agent disclosed in an embodiment of the present invention;

[0045] Figure 2 This is a diagram of the distributed hybrid retrieval and scheduling architecture disclosed in the embodiments of the present invention;

[0046] Figure 3 This is a flowchart of a process automation method based on a multimodal intelligent agent disclosed in an embodiment of the present invention;

[0047] Figure 4 This is the RAG-ReAct collaborative inference timing diagram disclosed in the embodiments of the present invention;

[0048] Figure 5 This is an example diagram of the thought display structure disclosed in the embodiments of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0050] The technical solutions disclosed in the various embodiments of this application will be described in detail below.

[0051] Example 1

[0052] like Figures 1 to 5As shown, this embodiment provides a process automation method based on multimodal intelligent agents, including the following steps:

[0053] Step S1: Collect multimodal process data, preprocess and uniformly represent the multimodal process data, and generate a unified data object;

[0054] Step S2: Construct or update the knowledge graph and vector index based on the unified data object, and obtain the evidence set through hybrid retrieval;

[0055] Step S3: Employ a dual-engine collaborative reasoning approach, combining a retrieval-enhanced generation engine and a tool-invoking inference engine, to align and constrain the evidence set, generate an executable process plan, and invoke external tools to execute the executable process plan.

[0056] Step S4: Perform consistency checks and recursive reflections on the reasoning and execution processes, generate reflection data, and generate scheduling parameters based on the dependencies and granularity factors in the reasoning and execution processes;

[0057] Step S5: Based on the reflective data, train or fine-tune the inference engine through cognitive distillation (CD), and inject parameters into the trained inference engine in a targeted manner.

[0058] Step S6: Based on scheduling parameters and real-time operating metrics, dynamically schedule and scale computing resources.

[0059] Step S7: Construct reward signals based on business metrics and language auditability metrics, and continuously optimize process automation methods through reinforcement learning.

[0060] The present invention will now be described in further detail.

[0061] Step S1: Collect multimodal process data, preprocess and uniformly represent the multimodal process data, and generate a unified data object.

[0062] This step aims to transform multimodal process data from diverse sources and in various formats into standardized objects that can be uniformly processed and traced within the engine.

[0063] S1.1 Multimodal Data Acquisition: When the process engine initiates a task (such as "Business Establishment Application") or a material change event is triggered, the multimodal data acquisition module accesses data from multiple channels, including the government gateway, business system API, RPA acquisition terminal, document delivery channel, and log subscription. This data includes: text policy documents, form fields, scanned documents / images (such as scanned copies of business licenses), audio and video (such as remote inspection videos), system logs, etc. The acquisition module generates a multimodal payload and associated process metadata (such as item ID, process instance ID, node ID, subject information, policy version number, etc.) for each piece of data.

[0064] S1.2 Edge Preprocessing: The edge preprocessing module performs standardization processing on data of different modalities. For text, it performs unified encoding, sentence segmentation, and retains character offsets. For images, it performs skew correction, noise reduction, page splitting, and generates page indexes. For audio and video, it unifies sampling rates and encoding formats and generates timeline indexes. Simultaneously, based on predefined audit and compliance tags (AuditTag), it de-identifies field-level or fragment-level sensitive information, generates signatures, and forms standardized payloads.

[0065] S1.3 Quality Gating: Before materials are written to the business database or knowledge base, a quality assessment is performed at the edge preprocessing node on the access side. Assessment metrics may include modal characteristic variability, calculated using the following formula: ,in, n The number of samples within the window. x i For the first i 1 eigenvalue, The mean. When v When the value exceeds the preset threshold, it indicates that the modal input fluctuates greatly and the quality is unstable. The data will be sent to the supplementary collection or manual review queue to ensure the reliability of subsequent processing.

[0066] S1.4 Feature Extraction and Fusion: The feature extraction module generates semantic representations for data from various modalities. For example, visual vectors are extracted from images using convolutional neural networks, and text vectors are extracted from text using models such as BERT. The fusion strategy library dynamically selects a fusion strategy based on the current task requirements (e.g., "material compliance verification"), data quality scores, and historical contributions to generate a unified representation. One weighted fusion method is as follows: ,in, f For process context features, w and w i Weights that can be learned or output from a policy library. For the first Semantic representation of a modality.

[0067] The fusion strategy library can comprehensively score the contribution of each modality and generate fusion weights using Softmax:

[0068] ;

[0069] in, For tasks and modalities semantic relevance, For modal quality scoring, For the statistics of historical marginal contribution, For the final fusion weights, , , The adjustment coefficients are manually configurable weights used to balance the importance of semantics, quality, and historical experience. For the first The score for each modality, For all modalities currently participating in the fusion, For the calculated modes The score. When When the value is below the threshold, modal downweighting / removal is triggered and the process is transferred to supplementary sampling or manual review.

[0070] S1.5 Output a Unified Data Object: To ensure consistency and traceability in subsequent modules, this step outputs a standardized data object, including at least: multimodal payload (original material references + preprocessing results), process metadata (items, nodes, versions, subjects, timelines, etc.), audit and compliance tags (de-identification rules, permission domains, signature summaries, etc.), and evidence citation identifiers EvidenceId / CiteId (material source, page number, bounding box, offset, timeline, etc.). It also outputs a structured field set FieldKV, a semantic block set EvidenceBlock, a unified vector UnifiedEmbedding, and relation candidates TripleCandidate to support incremental updates of the graph construction and vector index. As an idempotent key, it is used to avoid data duplication. In the formula, CaseId is the process instance identifier, NodeId is the process node identifier, Source is the material source channel identifier, and PayloadHash is the load summary. When the same IdemKey appears repeatedly, the cached result can be reused directly or the duplicate entry into the database can be rejected.

[0071] The permissions and auditing module generates audit records for each access and change during the data collection, access, preprocessing, and storage stages (including operator / service, timestamp, data domain, operation type, summary signature, and association (IdemKey)). The compliance policy / de-identification rules module performs de-identification / watermarking / security classification on field-level and fragment-level data according to AuditTag, and associates the traceable de-identification mapping table with evidence citations in a controlled manner to ensure that subsequent retrieval and playback can reproduce the evidence chain within the scope of permissions.

[0072] Step S2: Construct or update the knowledge graph (KG) and vector index based on the unified data object, and obtain the evidence set through hybrid retrieval.

[0073] This step aims to build a versionable and searchable knowledge base and provide traceable evidence for subsequent reasoning.

[0074] S2.1 Knowledge Graph and Vector Index Construction: The knowledge graph construction / update module extracts triples from (relation candidate) TripleCandidate and (structured field set) FieldKV, and incrementally updates the semantic graph and policy clause graph. The vector index module performs partitioned storage and fragmented index construction after vectorizing the unified representation and text fragments.

[0075] The knowledge graph construction / update module organizes government and enterprise process knowledge using an ontology and versioning approach: entities can include at least items, process nodes, materials, fields, subjects, legal clauses, rules, risk points, tools, and evidence blocks; relationships can include at least the materials / fields required for an item, clause-constrained fields, evidence of field sources, rule conflicts, node prerequisites, and fields / materials verified by tools. The graph records version, valid_from, valid_to, confidence, and evidence_ref for each node and edge, updating incrementally as events when policies or processes change, and performing consistency checks to identify duplicate entities, mutually exclusive rules, and risks of cross-version misuse.

[0076] The vector index module stores UnifiedEmbedding and evidence fragment vectors in a partitioned and sharded manner, and supports incremental writes and shard rebalancing: hot items / nodes can be split into partitions or have replicas added, while low-frequency partitions can be merged and compressed; the index structure can adopt HNSW, IVF / PQ or a combination thereof, and the index sharding and routing strategies are uniformly managed by the coordinator to ensure stable throughput and P95 latency during high-concurrency retrieval.

[0077] S2.2 Hybrid Retrieval: The hybrid retrieval module generates query vectors and keyword queries based on the workflow tasks. It performs parallel recall on the vector index and full-text search engine to obtain a candidate evidence set. To improve recall quality, a re-ranking fusion model is used to sort the candidate set. A common linear fusion method is as follows: Where sim is the cosine similarity between the query vector and the evidence vector, and BM25 is the keyword relevance score. The fusion weights can be dynamically adjusted based on historical feedback.

[0078] S2.3 Output Evidence Package: The sorted evidence is organized into a unified evidence package, EvidencePack. Each piece of evidence includes EvidenceId, source system, original location information (page number / bbox / offset / time segment), fragment content summary, overall score, applicable version information, and anonymization status. This evidence package is associated with audit logs, providing the foundation for the subsequent "conclusion-citation-execution-replay" closed loop.

[0079] Step S3: Employ a dual-engine collaborative reasoning approach, combining the Retrieval-Augmented Generation (RAG) engine and the Reasoning and Acting (ReAct) engine, to align and constrain the evidence set, generate an executable process plan, and call external tools to execute the executable process plan.

[0080] This step enables deep collaboration between RAG constraint generation and ReAct planning execution.

[0081] S3.1 RAG Evidence Alignment: After receiving the EvidencePack, the RAG engine assigns a unique CiteId to each evidence fragment and inserts it into the input context of the generative model. For each decision field or conclusion to be generated, RAG forces the model to bind one or more CiteIds, forming a field-evidence reference mapping.

[0082] S3.2 ReAct Planning and Execution: The ReAct engine receives tasks (such as "reviewing business registration materials") and breaks them down into executable steps such as "Step 1: Call the OCR service to extract business license information," "Step 2: Query the industrial and commercial interface to verify information," and "Step 3: Compare information consistency." Each step specifies the tool type, input parameters, expected observations, and failure fallback strategy. Subsequently, the tool invocation module calls external tools (such as OCR services, databases, and APIs) sequentially based on the permissions and interface definitions in the tool registry.

[0083] The tool invocation module uses a tool registry to manage invocation capabilities: each tool must include at least a tool identifier, input / output schema, permission domain, rate limit, audit fields, and failure handling strategy. ReAct selects only tools allowed by the current permissions and scenario when generating a plan, and records audit information (caller, parameter summary, return summary, time taken, error code, association (IdemKey), and CiteId) before and after each invocation.

[0084] The execution and write-back module forms a unified observation object (including returned data, status code, exception stack summary, time consumed, number of retries, affected fields and evidence references) for the tool call results, and writes it back to the process system and audit link in an idempotent manner; when partial failure occurs or external system rate limiting occurs, the module performs retries, downgrades to read-only verification or transfers to manual review according to the rollback strategy, and incorporates the reasons for failure into the subsequent reflection sample.

[0085] S3.3 Collaborative Iteration: During tool execution, if the observation results (such as field values ​​returned by OCR conflicting with version constraints in the graph) or the evidence is insufficient, the ReAct engine will trigger a "secondary search" instruction, returning to step S2 with the current context to obtain more accurate evidence. The RAG engine then uses the new evidence to correct the context, guiding ReAct to the next round of planning. This process iterates until all steps are completed and consistency constraints are met.

[0086] Step S4: Perform consistency checks and recursive reflections on the reasoning and execution processes, generate reflection data, and generate scheduling parameters based on the dependencies and granularity factors in the reasoning and execution processes.

[0087] This step aims to conduct a self-assessment and in-depth analysis of the execution process, providing a data foundation for system optimization.

[0088] S4.1 Consistency / Conflict Detection: The consistency detection module performs multi-dimensional verification of inference results and execution observations. For example, it checks "citation coverage" (whether any conclusions are without cited evidence), "evidence conflict" (whether the same field references two contradictory clauses), and "version consistency" (whether the cited policy version is consistent with the version required by the current process node). The output of consistency detection is a structured report, ConflictReport, which includes the conflict type, severity level, related fields, and corrective recommendations (such as "re-retrieve").

[0089] S4.2 Recursive Reflection: The recursive reflection module performs hierarchical reflection on ConflictReport and execution trajectory.

[0090] Outcome-level reflection: Identifying specific instances where "the conclusions are inconsistent with the evidence".

[0091] Process-level reflection: Identify problems such as "redundant tool calls" (e.g., calling the same API multiple times to obtain the same information) and "inefficient paths".

[0092] Strategy-level reflection: Analyze the underlying causes of the above problems, such as "unreasonable search weight settings" or "incorrect selection of fusion strategy." The reflection output is a structured ReflectionRecord, which includes the problem type, root cause attribution, and corrective action templates.

[0093] S4.3 Data Parallelism Detection: The data parallelism detection module extracts the dependencies and computational granularity factors between nodes from successful inference paths or execution trajectories, and combines this with historical communication costs and conflict frequencies to generate scheduling parameters. For example, the parallelization gain factor G is calculated using the following formula: Where ComputeCost is the computation cost, CommCost is the communication cost, and DepDepth is the dependency depth. The larger the G value, the more suitable the subtask is for parallel execution.

[0094] like Figure 3 As shown, Figure 3 The flowchart illustrates the closed-loop operation logic of the entire system. If a consistency check (step S4.1) detects a problem, it triggers a backtracking to either step S2 (re-retrieval) or step S3 (re-inference), forming a short-cycle inference closed loop. After distillation (step S5), the reflected data (step S4.2) output is used to update the retrieval weights in step S2 and the inference strategy in step S3, forming a medium-cycle strategy optimization closed loop. Scheduling parameters (step S4.3) and system performance metrics are used for real-time scheduling in step S6 and also serve as part of the reward signal for reinforcement learning in step S7, ultimately providing feedback to optimize all strategies, including the scheduling strategy in step S6, forming a long-cycle evolutionary closed loop.

[0095] Step S5: Based on the reflective data, train or fine-tune the inference engine through cognitive distillation (CD), and inject parameters into the trained inference engine in a targeted manner.

[0096] This step transforms the reflective data into reusable knowledge, which is then used to optimize the inference engine.

[0097] S5.1 Cognitive Distillation: The cognitive distillation module constructs a high-quality teacher signal from the correct trajectory in the reflection output ReflectionRecord, the correction suggestions in the consistency detection output ConflictReport, and the corresponding evidence pack EvidencePack. A student model (i.e., the inference engine) is trained to mimic the behavior of the teacher model, enabling it to directly select a better inference path in similar situations.

[0098] S5.2 Distillation Training: The distillation loss function can be designed as follows: ,in, and The output distributions for the teacher and student models are respectively. For task-related losses (such as field classification). Control the distillation intensity, Control the strength of the entropy regularization term.

[0099] S5.3 Targeted Injection: This involves injecting distilled knowledge, such as gradient directions extracted from knowledge graphs. To inject intensity and step length Parameters of the inference engine Update: This approach allows structured knowledge biases to be injected into neural network models.

[0100] This step also includes visualizing thoughts.

[0101] S5.4 Mind Visualization: The Mind Visualization module maps the complete chain of "evidence-citation-reasoning steps-tool call-observation results" into an interactive graph structure, which is used by auditors and business experts for review, accountability and strategy analysis.

[0102] Step S6: Based on scheduling parameters and real-time operating metrics, dynamically schedule and scale up / down computing resources.

[0103] This step uses node load balancing data to schedule resources and scale up or down, achieving stable high-concurrency operation. Through intelligent scheduling, it ensures the stability of the system under high load.

[0104] S6.1 Scheduling Decision: The resource scheduling module receives scheduling parameters and real-time monitoring metrics (such as CPU utilization, queue length, network latency, and hotspot shard access frequency) from step S4. For candidate computing nodes (such as retrieval service nodes and inference service nodes), a scoring function is used to evaluate their merits. Where Load is the node load (such as a normalized combination of CPU / GPU / queue length), Latency is the end-to-end latency or network latency estimate, and Conflict is the risk of resource conflicts and rate limiting.

[0105] S6.2 Policy Update: The scheduler dynamically adjusts routing strategies (such as routing new requests to the node with the highest score), shard concurrency (such as adding replicas to hot vector shards), and scaling strategies (such as automatically starting new inference service instances during peak periods) based on the scoring results and scheduling parameters. When external tool systems (such as OCR services) experience congestion, the scheduler will queue or prioritize process instances that depend on that tool.

[0106] S6.3 Degradation Path: When the system detects that the P95 latency continuously exceeds the threshold or the collision rate abnormally increases, a degradation path is automatically triggered. This may include reducing the number of secondary searches, switching to a more conservative fusion strategy (such as using only BM25), restricting the use of high-cost tools, or transferring some requests to manual processing. The degradation strategy and its triggering conditions are also recorded in the audit log.

[0107] Step S7: Construct reward signals based on business metrics and language auditability metrics, and continuously optimize process automation methods through reinforcement learning.

[0108] This step utilizes reinforcement learning to enable the entire system to evolve itself based on long-term goals.

[0109] S7.1 Reward Function Design: The model optimization module defines a composite reward function designed to maximize long-term returns. This function is: .in, These are environmental / business incentives, which can include positive indicators such as "first-time completion rate of processes", "shorter processing time", and "reduction in the number of times human intervention is required". These are language / auditability awards, which may include "completeness of citations", "clarity of expression", "consistency of evidence", etc. It is a coefficient that balances the two types of rewards. This is the discount factor.

[0110] S7.2 Policy Update: The model optimization module takes the complete "trajectory-reward" pair as input and updates the policy of the entire process automation engine using a policy gradient algorithm (such as PPO). The optimized policies include: feature fusion weights in step S1, hybrid retrieval weights in step S2, the prompt template and tool selection policy of the ReAct engine in step S3, and the scheduling parameter generation policy in step S6. The updated policies are released in a versioned manner and can be A / B tested and quickly rolled back.

[0111] This invention automates and concurrently processes material collection, extraction, verification, and reporting, reducing manual review and data entry costs, shortening processing times, and improving the efficiency of government and enterprise processes. Through traceable evidence chains and consistency checks, it enhances the interpretability and auditability of reviews, reducing compliance risks caused by policy version differences or material anomalies. Distributed retrieval and scheduling ensure stable operation under high concurrency, supporting cross-departmental, cross-system, and multi-regional deployment. The reflective distillation and reinforcement learning loop of this invention can transform online feedback into trainable signals, continuously improving accuracy and automation coverage.

[0112] In summary, the application of this invention in the field of government and enterprise process automation can bring significant benefits, including improving processing efficiency, enhancing compliance and risk control, ensuring stable operation under high concurrency, and promoting continuous iterative optimization, thereby providing auditable, traceable, and scalable intelligent support for the digital governance and operation of government and enterprises.

[0113] Example 2

[0114] like Figure 1 As shown, this embodiment provides a process automation system based on a multimodal intelligent agent, including:

[0115] The data acquisition and preprocessing module is configured to: acquire multimodal process data, preprocess and uniformly represent the multimodal process data, and generate a unified data object;

[0116] The knowledge graph and index building module is configured to: build or update knowledge graphs and vector indexes based on unified data objects, and obtain evidence sets through hybrid retrieval;

[0117] The dual-engine collaborative reasoning module is configured to: use a retrieval-enhanced generation engine and a tool-invoking reasoning engine for dual-engine collaborative reasoning, perform evidence alignment and citation constraints on the evidence set, generate an executable process plan, and call external tools to execute the executable process plan;

[0118] The detection and reflection module is configured to: perform consistency detection and recursive reflection on the reasoning and execution process, generate reflection data, and generate scheduling parameters based on the dependencies and granularity factors in the reasoning and execution process;

[0119] The distillation and injection module is configured to: train or fine-tune the inference engine based on reflective data through cognitive distillation, and inject parameters into the trained inference engine in a targeted manner.

[0120] The resource scheduling module is configured to dynamically schedule and scale up / down computing resources based on scheduling parameters and real-time performance metrics.

[0121] The model optimization module is configured to construct reward signals based on business metrics and language auditability metrics, and continuously optimize process automation methods through reinforcement learning.

[0122] This system also includes an access control and auditing module and a compliance policy / masking rules module. The access control and auditing module generates audit records for each access and change during the data collection, access, preprocessing and storage stages (including operator / service, timestamp, data field, operation type, summary signature, and association (IdemKey)). The compliance policy / masking rules module performs masking / watermarking / security classification on field-level and fragment-level data according to AuditTag, and associates the traceable masking mapping table with evidence citations in a controlled manner to ensure that subsequent retrieval and playback can reproduce the evidence chain within the scope of permissions.

[0123] It should be noted that the above modules correspond to the steps in Embodiment 1, and the examples and application scenarios implemented by the above modules and their corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules can be executed in a computer system as part of the system.

[0124] In further embodiments, the following is also provided:

[0125] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0126] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0127] A computer-readable storage medium for storing computer instructions that, when executed by a processor, perform the method of Embodiment 1.

[0128] The method in Example 1 can be directly executed by a hardware processor, or it can be executed by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0129] A computer program product includes a computer program that, when executed by a processor, implements the method in Embodiment 1.

[0130] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0131] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0132] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0133] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0134] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A process automation method based on multimodal intelligent agents, characterized in that, Includes the following steps: Collect multimodal process data, preprocess and uniformly represent the multimodal process data, and generate a unified data object; Knowledge graphs and vector indexes are built or updated based on unified data objects, and evidence sets are obtained through hybrid retrieval. A dual-engine collaborative reasoning approach, employing a retrieval-enhanced generation engine and a tool-invoking inference engine, is used to align and constrain the evidence set, generate an executable process plan, and invoke external tools to execute the executable process plan. The dual-engine collaborative reasoning includes: The retrieval enhancement generation engine binds reference identifiers to evidence fragments in the evidence set and constrains its output to carry the reference identifiers; The tool invocation inference engine breaks down the task objective into a sequence of executable steps that includes the tool type, input parameters, and observation write-back strategy, and calls external tools in sequence. When the tool's query results conflict with the evidence cited or the evidence is insufficient, a secondary search or prompt for correction is triggered, and the next round of search and execution iteration begins. Consistency checks and recursive reflections are performed on the reasoning and execution processes to generate reflection data, and scheduling parameters are generated based on the dependencies and granularity factors in the reasoning and execution processes. Based on reflective data, the inference engine is trained or fine-tuned through cognitive distillation, and parameters are injected into the trained inference engine in a targeted manner. Based on scheduling parameters and real-time performance metrics, computing resources are dynamically scheduled and scaled up or down. Reward signals are constructed based on business metrics and language auditability metrics, and process automation methods are continuously optimized through reinforcement learning.

2. The process automation method based on multimodal intelligent agents as described in claim 1, characterized in that, Multimodal process data is preprocessed and uniformly represented to generate a unified data object, including: Standardize the process data of different modalities to generate standardized loads; Standardized loads are subjected to quality gating, and low-quality data is sent to the supplementary collection or verification queue. Extract semantic representations of each modality, and select a fusion strategy based on task requirements, data quality, and historical contribution to generate a unified representation; Generate a unified data object containing standardized payloads, process metadata, audit tags, and evidence citation identifiers.

3. The process automation method based on multimodal intelligent agents as described in claim 1, characterized in that, Consistency checks and recursive reflections are performed on the reasoning and execution processes to generate reflective data, including: Perform citation coverage verification, evidence conflict detection, and version consistency verification on the inference results and execution observations, and output conflict points and correction suggestions; The conflict points and corrective suggestions are reflected upon at the result level, process level, and strategy level, and the reflection results are precipitated into structured reflection data.

4. The process automation method based on multimodal intelligent agents as described in claim 1, characterized in that, Based on reflective data, the inference engine is trained or fine-tuned through cognitive distillation, and then parameters are injected into the trained inference engine in a targeted manner, including: Reflective data, verifiable evidence, and correct trajectories are used to construct teacher signals to train students' reasoning models. By biasing the structured knowledge extracted from the knowledge graph into gradient directions, the parameters of the inference engine are updated in a targeted manner.

5. The process automation method based on multimodal intelligent agents as described in claim 1, characterized in that, Based on scheduling parameters and real-time performance metrics, computing resources are dynamically scheduled and scaled up or down, including: It receives scheduling parameters and real-time operational metrics including node load, latency, and conflict risks, scores candidate nodes, and dynamically adjusts request routing, sharding concurrency, and the number of computing resource replicas based on the scoring results.

6. The process automation method based on multimodal intelligent agents as described in claim 1, characterized in that, Reward signals are constructed based on business metrics and language auditability metrics, and process automation methods are continuously optimized through reinforcement learning, including: The process completion rate, rework rate, timeliness, and human intervention rate are used to construct business rewards, while the integrity of citations and clarity of expression are used to construct language rewards. Based on the weighted sum of the business rewards and language rewards, the weights of hybrid retrieval, feature fusion, and tool invocation methods are updated through a strategy gradient algorithm.

7. A process automation system based on multimodal intelligent agents, characterized in that, include: The data acquisition and preprocessing module is configured to: acquire multimodal process data, preprocess and uniformly represent the multimodal process data, and generate a unified data object; The knowledge graph and index building module is configured to: build or update knowledge graphs and vector indexes based on unified data objects, and obtain evidence sets through hybrid retrieval; The dual-engine collaborative reasoning module is configured to: employ a retrieval-enhanced generation engine and a tool-invoking inference engine for dual-engine collaborative reasoning, perform evidence alignment and citation constraints on the evidence set, generate an executable process plan, and invoke external tools to execute the executable process plan; wherein, the dual-engine collaborative reasoning of the retrieval-enhanced generation engine and the tool-invoking inference engine includes: The retrieval enhancement generation engine binds reference identifiers to evidence fragments in the evidence set and constrains its output to carry the reference identifiers; The tool invocation inference engine breaks down the task objective into a sequence of executable steps that includes the tool type, input parameters, and observation write-back strategy, and calls external tools in sequence. When the tool's query results conflict with the evidence cited or the evidence is insufficient, a secondary search or prompt for correction is triggered, and the next round of search and execution iteration begins. The detection and reflection module is configured to: perform consistency detection and recursive reflection on the reasoning and execution process, generate reflection data, and generate scheduling parameters based on the dependencies and granularity factors in the reasoning and execution process; The distillation and injection module is configured to: train or fine-tune the inference engine based on reflective data through cognitive distillation, and inject parameters into the trained inference engine in a targeted manner. The resource scheduling module is configured to dynamically schedule and scale up / down computing resources based on scheduling parameters and real-time performance metrics. The model optimization module is configured to construct reward signals based on business metrics and language auditability metrics, and continuously optimize process automation methods through reinforcement learning.

8. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-6.

Images