Artificial intelligence-based assessment methods, devices, equipment, and media for digital employees

By decomposing and vectorizing the work content data of digital employees, a task rule vector library is generated, and the execution results of artificial intelligence are compared. This solves the problem of quantitatively evaluating the degree of artificial intelligence implementation of digital employees, and realizes refined operation optimization and cost management.

CN123117579APending Publication Date: 2026-11-17CHINA UNITED NETWORK COMM GRP CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611002585.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-07
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing digital employee operation solutions lack quantitative assessment and dynamic measurement methods for the degree of artificial intelligence implementation, making it impossible to accurately grasp the level of artificial intelligence in each work process. This makes it difficult to support the reasonable setting of the artificial intelligence ratio of digital employees and the dynamic allocation of costs, especially in complex business scenarios where flexible management of artificial intelligence applications is not possible.

Method used

By acquiring the work content data of the target digital employees, breaking it down into the smallest execution unit to form a sequence of atomic task nodes, and then vectorizing it to generate a task rule vector library, matching the target task executor, comparing the consistency between the artificial intelligence execution results and the human execution results, determining the artificial intelligence-capable state of each node, and finally calculating the artificial intelligence achievement rate.

Benefits of technology

It enables quantitative assessment and dynamic calculation of the degree of artificial intelligence implementation in digital employees, providing accurate data-driven decision-making support, operational optimization, and improved operational efficiency and cost management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123117579A_ABST
    Figure CN123117579A_ABST
Patent Text Reader

Abstract

This invention relates to a method, apparatus, device, and medium for evaluating the artificial intelligence (AI) implementation of digital employees. The method involves: acquiring work content data corresponding to the target digital employee; decomposing the work content data into atomic task node sequences using the smallest execution units; vectorizing each atomic task node to obtain a task node set; extracting execution rules for each atomic task node from task flow data and generating rule vectors; clustering all rule vectors to obtain a task rule vector library; matching each atomic task node with a corresponding target task executor based on the task rule vector library; obtaining AI execution results through the target task executors; comparing the consistency between the AI ​​execution results and the manual execution results to determine the AI-capable state of each atomic task node; and counting the number of AI execution nodes based on the AI-capable state to obtain the AI ​​implementation rate of the target digital employee.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of digital employee operation technology, specifically to an artificial intelligence-based evaluation method, apparatus, equipment, and medium for digital employees. Background Technology

[0002] Digital employees are digital laborers that simulate human operations by relying on technologies such as AI (artificial intelligence) and process automation. They are widely used in various business scenarios such as enterprise operation and maintenance scheduling, business processing, and customer service. Digital employee operation is the full lifecycle management of digital employees, covering multiple aspects such as deployment, task scheduling, performance control, efficiency optimization, and cost control. Its core objective is to fully unleash the automation value of digital employees while ensuring stable business operation, and help enterprises improve operational efficiency and optimize human resource costs.

[0003] Current digital employee operations mainly focus on optimizing task execution efficiency and scheduling capabilities. Typically, business tasks are first structurally represented to extract task features, and then the real-time performance status of digital employees and the capability differences of different task execution modules are combined to generate corresponding target operation plans. Through mechanisms such as process orchestration and dynamic resource allocation, digital employees are driven to complete business processing, which can cope with dynamic changes in the operation process to a certain extent and meet the automation processing needs of routine enterprise-level tasks.

[0004] However, existing digital employee operation solutions only focus on optimizing execution processes and improving operational efficiency. They lack quantitative assessment and dynamic measurement methods for the degree of artificial intelligence implementation of digital employees, making it impossible to accurately grasp the level of artificial intelligence in each work link of digital employees. This makes it difficult to support refined operational decisions such as setting a reasonable ratio of artificial intelligence for digital employees and dynamically allocating labor and development costs. In complex business scenarios with high security requirements, such as operator fault handling, it is also impossible to flexibly manage the degree of artificial intelligence application while controlling business risks.

[0005] Therefore, how to quantitatively assess and dynamically calculate the degree of artificial intelligence implementation in digital employees to support the refined operation and optimization of digital employees is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] The technical problem to be solved by this invention is how to achieve quantitative assessment and dynamic calculation of the degree of artificial intelligence implementation of digital employees, so as to support the refined operation and optimization of digital employees.

[0007] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: This application provides an embodiment of an artificial intelligence implementation evaluation method for digital employees, including: Obtain the job content data corresponding to the target number of employees; The work content data is decomposed into the smallest execution unit to obtain an atomic task node sequence. Each atomic task node in the atomic task node sequence is vectorized to obtain a task node set consisting of multiple atomic task nodes carrying task vectors. Collect task flow data corresponding to each atomic task node in the task node set, extract the execution rules of each atomic task node from the task flow data and generate rule vectors, and cluster and organize all rule vectors to obtain a task rule vector library. Based on the task rule vector library, a corresponding target task executor is matched for each atomic task node. The artificial intelligence execution result of each atomic task node is obtained through the target task executor. The manual execution result of each atomic task node is obtained, and the consistency comparison between the artificial intelligence execution result and the manual execution result is performed. Based on the consistency comparison result, the artificial intelligence-capable state corresponding to each atomic task node is determined. Based on the AI-capable state corresponding to each atomic task node, the number of AI execution nodes is counted, and based on the number of AI execution nodes, the AI ​​achievement rate of the target digital employee is obtained.

[0008] The beneficial effects of this invention are as follows: It acquires the work content data corresponding to the target digital employee, breaks it down into the smallest execution units to form a sequence of atomic task nodes, and then performs vectorization processing on each atomic task node to obtain a set of task nodes carrying task vectors. This transforms the originally vague and highly holistic business work into standardized smallest execution units that can be independently measured and accurately labeled, providing a statistically granular carrier for the quantitative evaluation of the degree of artificial intelligence implementation. This fundamentally solves the problems of vague work boundaries for digital employees and the inability to break down and measure the degree of AI implementation in the traditional model. Furthermore, by collecting the task flow data of each atomic task node in the task node set, it extracts execution rules and generates rule vectors, which are then clustered and organized to form a task rule vector library. This transforms the scattered and unstructured task execution logic into a structured, reusable, and uniformly matchable standardized rule system, providing a clear rule benchmark for subsequent task executor matching and ensuring the overall... The consistency and objectivity of the quantitative node evaluation standards avoid the subjective bias caused by manual qualitative judgment, laying a solid rule foundation for quantitative evaluation. Based on the task rule vector library, each atomic task node is matched with a corresponding target task executor. The AI ​​execution results are obtained and compared with the manual execution results. The AI-capable state of each atomic task node is determined based on the comparison results. The objective execution result data comparison is used as the basis for judging whether a node is AI-capable. This ensures that the AI ​​attributes of each atomic task node have verifiable and reproducible data support, achieving accurate quantitative judgment of the AI ​​status of a single node. The number of AI execution nodes is counted based on the AI-capable state of each atomic task node, and the AI ​​implementation rate of the target digital employee is calculated. The abstract AI implementation level is transformed into an intuitive numerical indicator that can be compared horizontally and vertically, completing the full-link quantitative evaluation from the granularity of a single node to the overall indicator of digital employees. Meanwhile, since the entire evaluation process can be repeatedly triggered as work content is updated, task flow rules are iterated, executor capabilities are upgraded, or online operation data fluctuates, it can update the AI-capable status of each node in real time and refresh the AI ​​implementation rate simultaneously. Therefore, it can achieve dynamic measurement of the degree of AI implementation, rather than a one-time static evaluation. The quantified overall AI implementation rate index, coupled with the AI ​​status data refined to each atomic task node, can provide the operations side with accurate data decision-making basis, supporting operations personnel to adjust the configuration of human positions, allocate development and optimization resources, and formulate differentiated human-machine collaboration strategies, ultimately achieving refined operation optimization of digital employees.

[0009] Based on the above technical solution, the present invention can be further improved as follows.

[0010] Furthermore, the work content data is decomposed into the smallest execution unit to obtain an atomic task node sequence. Each atomic task node in the atomic task node sequence is then vectorized to obtain a task node set consisting of multiple atomic task nodes carrying task vectors, including: Using a large language model, semantic parsing is performed on the natural language data in the work content data to obtain semantic parsing results; The semantic parsing result is decomposed into the smallest execution unit to obtain the atomic task node sequence marked with triples, wherein the triples include tools, actions, and target objects; Basic attributes are extracted from each atomic task node in the atomic task node sequence to generate a work description card corresponding to each atomic task node. Based on the attribute information in the job description card, multi-dimensional feature mapping is performed on each atomic task node so that each atomic task node carries an eight-dimensional task vector, resulting in the task node set. The eight-dimensional task vector includes task content description, source system, target system, task target description, task execution plan, task closed-loop receipt, task process, and task dynamic knowledge graph.

[0011] The beneficial effects of adopting the above-mentioned further solution are as follows: Automated semantic parsing and element extraction of work content are achieved through a large language model, replacing manual conversion of unstructured text into structured information. Standardized decomposition of the smallest execution unit is then completed using triplet labeling, ensuring uniform granularity and integrity of core elements in task decomposition. Simultaneously, centralized organization of node attributes is achieved through work description cards, ultimately mapping and generating task vectors covering eight dimensions. This establishes a complete automated conversion link from natural language work descriptions to structured task nodes. On the one hand, this significantly improves the processing efficiency of work content decomposition and vectorization, reducing the cost and subjective bias of manual decomposition. On the other hand, the eight-dimensional task vector comprehensively covers the core attributes of the task, such as content, system, objective, process, and knowledge, making the business representation of atomic task nodes more complete and accurate. This provides a high-quality structured data foundation for subsequent rule extraction, executor matching, and AI-based evaluation, effectively improving the accuracy and reliability of the overall evaluation system.

[0012] Further, the process involves collecting task flow data corresponding to each atomic task node in the task node set, extracting execution rules for each atomic task node from the task flow data and generating rule vectors, and clustering all rule vectors to obtain a task rule vector library, including: For each atomic task node in the task node set, the data flow path, interaction information and historical closed-loop samples in the corresponding task flow are collected. Based on the data flow path, the interaction information and the historical closed-loop samples, the task flow dataset corresponding to each atomic task node is obtained. The execution logic of each atomic task node is parsed from the task flow dataset, and execution rules including tool, action, and target mapping relationship are extracted. Based on the execution rules, a serialized rule vector is generated. The rule vectors are indexed and clustered to integrate and classify rule vectors with the same execution logic, thus obtaining the task rule vector library.

[0013] The beneficial effects of adopting the above-mentioned further solutions are as follows: By collecting three types of data—data flow paths, interaction information, and historical closed-loop samples—a task flow dataset is constructed, comprehensively covering the link logic, interaction logic, and empirical cases of task execution. This ensures that the data foundation for extracting execution rules is comprehensive and closely matches real business scenarios. By analyzing the execution logic through process mining and generating serialized rule vectors, the core execution elements of the task—tools, actions, and objectives—are accurately extracted, while the temporal logic of task execution is fully preserved, making the rule representation more consistent with actual business processes. Furthermore, by constructing vector indexes and clustering, rules are automatically classified and integrated, which not only improves the processing efficiency of rule retrieval and matching but also eliminates duplicate and redundant rule entries, forming a standardized and reusable task rule vector library. This achieves the automated conversion from scattered business operation data to a structured rule system, providing a unified and accurate rule benchmark for subsequent task executor matching, and ensuring the consistency of the evaluation standards for all nodes. This effectively improves the objectivity, processing efficiency, and rule reusability of the entire evaluation system.

[0014] Furthermore, based on the task rule vector library, a corresponding target task executor is matched for each atomic task node. The AI ​​execution result of each atomic task node is obtained through the target task executor. The manual execution result of each atomic task node is also obtained. A consistency comparison is performed between the AI ​​execution result and the manual execution result. Based on the consistency comparison result, the AI-capable state of each atomic task node is determined, including: Based on the execution rules and task characteristics of each atomic task node in the task rule vector library, the atomic task nodes are divided into execution types to obtain task node groups corresponding to different execution types. For each of the task nodes grouped, a target task executor of the corresponding type is matched from a preset executor set; Obtain a task sample, execute the task sample through the target task executor to obtain the artificial intelligence execution result, and obtain the human execution result corresponding to the task sample; The consistency comparison between the artificial intelligence execution result and the manual execution result is performed, and the artificial intelligence-capable state corresponding to each atomic task node is determined based on the consistency comparison result and the preset consistency threshold.

[0015] The beneficial effects of adopting the above-mentioned further solution are as follows: First, nodes are categorized and grouped according to execution rules and task characteristics. Then, task executors of the corresponding types are matched according to the groups, achieving precise adaptation between execution capabilities and task requirements. This avoids deviations in execution effects caused by mismatch between executors and task types. At the same time, the categorized batch matching mode also significantly improves the efficiency of executor configuration for multiple nodes. By conducting human-machine dual-track execution tests using samples of the same source task, the homogeneity and comparability of the comparison data are ensured. Then, using differentiated consistency thresholds as the judgment criteria, the AI-based judgment of different types of nodes has an objective and unified quantitative basis, effectively reducing the subjective error of human judgment. In this embodiment, the solution establishes a complete standardized process from task classification and executor adaptation to human-machine comparison and judgment. This not only ensures the accuracy and rationality of the AI-based status judgment of each node but also improves the evaluation and processing efficiency of batch nodes, providing reliable node status data support for the accurate calculation of the subsequent artificial intelligence achievement rate.

[0016] Furthermore, the consistency comparison between the AI ​​execution result and the manual execution result includes: The output data format of each node in the data stream is unified, and a dual-table storage scheme is used to store the manual execution results and the artificial intelligence execution results respectively, resulting in a dual-track output dataset. The dual-track output dataset is subjected to output compliance detection, and data that does not conform to the preset output format specifications is removed to obtain a valid output dataset; The target AI execution results and target human execution results in the effective output dataset are compared item by item to obtain the consistency comparison results.

[0017] The beneficial effects of adopting the above-mentioned further solution are as follows: First, by unifying the output data format and then using a dual-table storage scheme to construct a dual-track output dataset, the comparison error caused by format differences is eliminated from the data source, ensuring that the two types of execution results have a unified comparison benchmark. At the same time, the dual-table independent storage mode not only ensures the integrity and independence of the two types of data sources, but also adapts to the collection needs of multiple scenarios in testing and production. By eliminating abnormal format data through pre-output compliance detection, the format verification and content comparison are separated, avoiding the interference of format-related issues with the judgment of content consistency, and effectively improving the accuracy of comparison results. Furthermore, by achieving fine-grained verification of each field through item-by-item consistency comparison, not only can the overall consistency level be quantified, but also specific differences can be accurately located, making the comparison results more valuable for reference.

[0018] Furthermore, determining the AI-capable state corresponding to each atomic task node based on the consistency comparison results includes: A four-dimensional evaluation framework including reliability, adaptability, cost-effectiveness, and boundary handling is constructed. Atomic task nodes are hierarchically divided according to task characteristics to obtain multiple task levels. Corresponding accuracy thresholds and alternative strategies are set for different task levels. Based on the four-dimensional evaluation framework, multiple task levels, accuracy thresholds, and alternative strategies, a hierarchical evaluation standard system is formed. Based on the aforementioned hierarchical evaluation standard system, from four dimensions—capability, cost, risk, and data volume—corresponding evaluation indicators and alternative thresholds are configured for atomic task nodes at each task level, resulting in a multi-dimensional evaluation indicator system. The consistency comparison results and the execution data of the corresponding atomic task nodes are input into the multi-dimensional evaluation index system to obtain the scoring results of each dimension. The scores of each dimension are integrated and calculated through the comprehensive judgment matrix to obtain the initial artificial intelligence-capable state corresponding to each atomic task node. Based on a preset veto list, the initial AI-capable state is adjusted to obtain the target AI-capable state.

[0019] The beneficial effects of adopting the above-mentioned further solutions are as follows: By building a four-dimensional evaluation framework and a corresponding hierarchical task system, the one-sidedness of single-indicator evaluation is broken. Differentiated accuracy thresholds and alternative strategies are set for tasks of varying complexity, avoiding a one-size-fits-all evaluation standard and making the evaluation results more aligned with the business characteristics and risk levels of different tasks. By refining the specific evaluation indicators and alternative thresholds for the four dimensions of capability, cost, risk, and data volume, the top-level evaluation framework is implemented as a quantifiable and verifiable indicator system, allowing AI-based evaluation to shift from qualitative experience-based judgment to precise quantitative calculation, significantly improving the objectivity and accuracy of the evaluation. Furthermore, the comprehensive judgment matrix enables the scientific weighted integration of multi-dimensional scores, taking into account factors such as execution effect, operating cost, and operational risk, ensuring the comprehensiveness and rationality of the comprehensive judgment. Finally, the addition of a veto list as a hard constraint safeguards the bottom line of safety and compliance on the basis of conventional evaluation, avoiding the security risks and compliance risks caused by blindly replacing AI in high-risk scenarios.

[0020] Furthermore, the step of counting the number of AI execution nodes based on the AI-capable state corresponding to each atomic task node, and obtaining the AI ​​achievement rate of the target digital employee based on the number of AI execution nodes, includes: Obtain the total number of all task nodes, including AI execution nodes, fixed program execution nodes, and human nodes; Based on the AI-capable state of each atomic task node, task rule matching verification, executor matching verification, and substitutability evaluation verification are performed on each atomic task node to obtain three verification results. Nodes that pass all three verification results are identified as AI execution nodes, and the number of AI execution nodes is determined. The AI ​​implementation rate of the target digital employee is obtained by calculating the ratio of the number of AI execution nodes to the total number.

[0021] The beneficial effects of adopting the above-mentioned further solution are as follows: By clearly defining the total nodes as including three categories—AI execution nodes, fixed program execution nodes, and human nodes—the denominator coverage of the achievement rate calculation is complete and comprehensive, avoiding denominator bias caused by the omission of fixed program nodes. This makes the indicators more aligned with the complete operational system of digital employees, and the statistical caliber is more scientific and reasonable. Through a triple verification process of task rule matching, executor matching, and substitutability assessment, AI execution nodes are screened layer by layer. The identification standards of AI nodes are strictly controlled from three dimensions: rule support, implementation carrier, and business effect. This effectively avoids node misjudgment caused by single-state judgment, ensuring that all nodes included in the numerator have the AI ​​execution capabilities that are implementable, stable, and achieve the expected results, making the achievement rate data more authentic and reliable. Finally, the quantitative achievement rate indicator is output through the ratio of the number of nodes. The calculation logic is clear, intuitive, and highly interpretable. It can accurately quantify the overall AI achievement level of digital employees and can also quickly update the value according to changes in node status. This provides accurate and reliable quantitative decision-making basis for cost allocation, development resource allocation, and human-machine collaboration strategy formulation for digital employees, effectively supporting the refined operation optimization of digital employees.

[0022] This application embodiment also provides an artificial intelligence implementation evaluation device for digital employees, the device comprising: The job content data acquisition module is used to acquire the job content data corresponding to the target number of employees; The task node set acquisition module is used to decompose the work content data into the smallest execution unit to obtain an atomic task node sequence, and to vectorize each atomic task node in the atomic task node sequence to obtain a task node set consisting of multiple atomic task nodes carrying task vectors. The vector library acquisition module is used to collect task flow data corresponding to each atomic task node in the task node set, extract the execution rules of each atomic task node from the task flow data and generate rule vectors, and cluster and organize all rule vectors to obtain a task rule vector library. The consistency comparison module is used to match the corresponding target task executor for each atomic task node based on the task rule vector library, obtain the artificial intelligence execution result of each atomic task node through the target task executor, obtain the manual execution result of each atomic task node, and perform a consistency comparison between the artificial intelligence execution result and the manual execution result. Based on the consistency comparison result, the AI-capable state of each atomic task node is determined. The achievement rate acquisition module is used to count the number of artificial intelligence execution nodes according to the artificial intelligence scalability status corresponding to each atomic task node, and obtain the artificial intelligence achievement rate of the target digital employee based on the number of artificial intelligence execution nodes.

[0023] This application also provides an electronic device, including a processor and a memory, wherein the memory stores multiple instructions; the processor loads instructions from the memory to execute steps in any of the artificial intelligence implementation evaluation methods for digital employees provided in this application.

[0024] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the artificial intelligence implementation evaluation methods for digital employees provided in this application. Attached Figure Description

[0025] Figure 1 A flowchart illustrating the AI-based evaluation method for digital employees provided in this embodiment of the invention; Figure 2 A schematic diagram of the structure of the artificial intelligence implementation evaluation device for digital employees provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0026] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0027] This application provides an artificial intelligence implementation evaluation method, apparatus, electronic device, and medium for digital employees.

[0028] Specifically, the AI-powered evaluation device for the digital employee can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet, smart Bluetooth device, laptop, or personal computer (PC); the server can be a single server or a server cluster consisting of multiple servers.

[0029] In some embodiments, the AI ​​implementation evaluation device for digital employees can also be integrated into multiple electronic devices. For example, the AI ​​implementation evaluation device for digital employees can be integrated into multiple servers, and the AI ​​implementation evaluation method for digital employees of this application can be implemented by multiple servers.

[0030] In some embodiments, the terminal can also be used as a server to implement some or all of the functions of a server.

[0031] In the embodiments of this application, the terms "module" or "unit" can refer to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functions of that module or unit.

[0032] The following sections provide detailed descriptions of each example. It should be noted that the order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0033] This embodiment provides a method for evaluating the artificial intelligence implementation of digital employees. See also Figure 1 , Figure 1 This is a flowchart illustrating the AI-based evaluation method for digital employees provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the specific process of evaluating the artificial intelligence implementation of this digital employee can be as follows: 110. Obtain the job content data corresponding to the target number of employees; 120. Decompose the work content data into the smallest execution unit to obtain an atomic task node sequence. Vectorize each atomic task node in the atomic task node sequence to obtain a task node set consisting of multiple atomic task nodes carrying task vectors. 130. Collect task flow data corresponding to each atomic task node in the task node set, extract the execution rules of each atomic task node from the task flow data and generate rule vectors, and cluster and organize all rule vectors to obtain a task rule vector library. 140. Based on the task rule vector library, match the corresponding target task executor for each atomic task node, obtain the artificial intelligence execution result of each atomic task node through the target task executor, obtain the manual execution result of each atomic task node, and perform a consistency comparison between the artificial intelligence execution result and the manual execution result. Based on the consistency comparison result, determine the artificial intelligence-capable state corresponding to each atomic task node. 150. Based on the AI-capable state corresponding to each atomic task node, count the number of AI execution nodes, and based on the number of AI execution nodes, obtain the AI ​​implementation rate of the target digital employee.

[0034] Digital employees, built upon automation and artificial intelligence technologies, are digital operational entities that can replace or assist humans in completing repetitive and rule-based business tasks. They are widely used in business process automation scenarios in industries such as government, telecommunications, and finance. In this embodiment, the target digital employee refers to a digital operational entity deployed within the enterprise's business system with clearly defined job responsibilities and task boundaries. For example, in a telecommunications scenario, a digital operational entity responsible for specific tasks such as fault scheduling and communication cloud topology querying is the subject of evaluation for the artificial intelligence implementation rate.

[0035] Work content data refers to the information records of all work items in a position. It serves as the foundational input data for process decomposition, efficiency evaluation, and automation transformation. It typically includes work item descriptions, business scenarios, input / output requirements, and related systems. In this embodiment, work content data refers to the descriptive information of all work items undertaken by the target digital employee's position. This includes work content descriptions in natural language, business scenario descriptions, work sources and output requirements, and related business system information. It encompasses both routine, daily tasks and scenario-based specialized tasks, serving as the raw input data for task decomposition and evaluation.

[0036] The smallest execution unit refers to an indivisible independent work segment obtained by breaking down a complete business process according to its execution logic. Each unit has a single execution action and clear inputs and outputs, serving as the basic granularity for process optimization and efficiency measurement. In this embodiment, the smallest execution unit refers to an indivisible independent work segment obtained by breaking down the complete work content according to its business execution logic. Each unit corresponds to a single operation action and clear inputs and outputs, and is the smallest granular work unit constituting the complete workflow. Its execution subject and execution effect can be independently determined.

[0037] In this context, a task node is an independent execution link in the process, and a node sequence is a combination of nodes arranged according to the logical order of business execution, which can completely reconstruct the execution link of the business process. In this embodiment, the atomic task node sequence refers to the sequence of nodes formed by breaking down the work content into multiple smallest execution units and arranging them according to the logical order of business execution. Each node in the sequence corresponds to an atomic task, completely reconstructing the entire execution link of the original work of the target digital employee. Each node has independent business meaning and execution attributes.

[0038] Vectors are a common method for structuring data features, transforming unstructured semantic information into computable numerical features. They are widely used in scenarios such as similarity matching, cluster analysis, and model input. In this embodiment, the task vector is a feature representation formed by structuring the multi-dimensional business attributes of atomic task nodes. It includes eight dimensions of feature information: task content description, source system, target system, task target description, task execution plan, task closed-loop receipt, task flow, and task dynamic knowledge graph. This transforms unstructured task descriptions into computable and matchable structured feature data, supporting subsequent rule extraction and executor matching processes.

[0039] The task node set refers to the set of all atomic task nodes carrying corresponding task vectors, covering all the work nodes after work decomposition for the target digital employee. It is the basic set of objects for subsequent rule extraction, executor matching and artificial intelligence evaluation.

[0040] Task flow data refers to the runtime data generated throughout the entire task execution process. It records the task's flow, interaction information, status changes, and processing results, serving as the data foundation for analyzing task execution logic and optimizing process efficiency. In this embodiment, task flow data refers to the end-to-end runtime data generated by atomic task nodes during actual business operations. This includes the data flow path during task execution, interaction information between nodes, closed-loop samples after historical task completion, and status change records during execution. It is the data source for extracting task execution rules.

[0041] Among them, the execution rules are the standardized logic guiding task execution, clarifying the triggering conditions, execution actions, operating tools, and output requirements of the task. They are the core basis for the automation of task execution. In this embodiment, the execution rules refer to the standardized execution logic of atomic task nodes parsed from task flow data. This includes the mapping relationship between the tools used for task execution, the specific actions performed, and the target object. It clarifies the execution method, triggering conditions, and output requirements of the task nodes and serves as the logical basis for the automation or artificial intelligence execution of tasks.

[0042] In this context, a rule vector is a vector representation obtained by serializing and characterizing the rule logic, enabling similarity calculation and rapid matching of rules. In this embodiment, a rule vector refers to the vector representation obtained by serializing and characterizing the execution rules of atomic task nodes. It carries the execution logic and features of task units in a structured form and can be used for similarity calculation, cluster analysis, and index matching. It is the basic unit for building a rule vector library.

[0043] The vector library is a structured repository for storing and managing vector data. It supports vector indexing, similarity retrieval, and clustering management, enabling efficient reuse and matching of massive amounts of rules and knowledge. In this embodiment, the task rule vector library refers to a set of rules formed by integrating the rule vectors corresponding to all atomic task nodes through vector indexing and clustering analysis. Rules with the same or similar execution logic are categorized and integrated to achieve unified management and reuse of rules for similar tasks, providing standardized rule references for matching task executors.

[0044] The task executor is a functional unit that carries the task execution logic. It can automatically complete task processing and output results according to preset rules. Different types of executors are adapted to task scenarios of different complexities. In this embodiment, the target task executor refers to an automated or artificial intelligence execution unit that is matched with the execution rules and task characteristics of the task node and is used to execute the corresponding atomic task. It includes different types such as pure program orchestration execution units, intelligent agent orchestration execution units, and hybrid orchestration execution units. It can complete the automated processing of tasks and output results according to the corresponding rules.

[0045] Among them, the AI ​​execution result is the processing content and status data output by the intelligent system after processing the task. It is the direct basis for evaluating the execution effect of the intelligent system and usually includes information such as output content, processing time, and execution status. In this embodiment, the AI ​​execution result refers to the task processing result data output after the target task executor runs and processes the corresponding task sample. It includes information such as the task's processing output content, processing time, and execution status, and is the core data for evaluating the AI ​​execution effect of the task.

[0046] In this context, the manual execution result refers to the standard result output by professionals in the business domain after processing the task. It serves as a benchmark for evaluating the performance of automated and artificial intelligence systems, representing the industry standard for this type of task. In this embodiment, the manual execution result refers to the standard task processing result data output by business personnel in the corresponding positions after manually executing the same task sample. This data serves as a benchmark for evaluating the performance of artificial intelligence, representing the human execution standard for this type of task.

[0047] Consistency comparison is a process of comparing two sets of data from the same source item by item according to a unified standard. It is used to quantify the degree of difference between the two sets of data and determine whether the data to be evaluated meets the benchmark requirements. In this embodiment, consistency comparison refers to the process of comparing and verifying the results of artificial intelligence execution and human execution item by item according to a unified output standard. This includes steps such as output format compliance verification, output content consistency comparison, and statistical significance testing. The quantified comparison results determine the degree of difference between the effects of artificial intelligence execution and human execution.

[0048] Among them, the "AI-capable status" is a determination of whether a business process can be independently executed by artificial intelligence, and it serves as the basis for planning intelligent transformation paths. In this embodiment, the AI-capable status refers to the determination result of whether an atomic task node can be independently executed by artificial intelligence after comparison and comprehensive evaluation of human-machine execution effects. It is divided into two categories: AI-capable and non-AI-capable, and serves as the basis for determining the number of AI execution nodes.

[0049] In this context, an AI execution node refers to a business process that can be independently executed by an AI system, and is a core component of the digital workflow. In this embodiment, an AI execution node refers to an atomic task node that has been evaluated and confirmed to be independently executable by AI, and whose actual operational performance meets the evaluation criteria; it is the numerator statistical term for calculating the AI ​​achievement rate.

[0050] Among them, the AI ​​implementation rate is a core indicator for quantifying the level of intelligence of digital employees. It can intuitively reflect the coverage of AI implementation among digital employees and is a core reference data for operational optimization. In this embodiment, the AI ​​implementation rate refers to the ratio of the number of AI execution nodes to the total number of task nodes corresponding to the target digital employee. It quantifies the degree of AI implementation of the target digital employee in numerical form and can be used to compare the AI ​​level of different digital employees horizontally and track the AI ​​progress of the same digital employee vertically.

[0051] For example, the specific method for obtaining the work content data corresponding to the target number of employees can be: By connecting with enterprise business operation systems, job description documents, historical work order systems, and task record platforms, we collect all work-related information for the target digital employees' corresponding positions. This includes natural language descriptions of various business tasks such as daily inspections, fault handling, data queries, and report submissions, as well as related system information, work output requirements, and closed-loop standards. The work information scattered across different channels is then aggregated, deduplicated, and standardized to form a complete work content dataset, which serves as the raw input for subsequent task breakdown.

[0052] For example, the specific method for decomposing and vectorizing the work content data into its smallest execution unit to obtain a set of task nodes can be as follows: First, a large language model is used to semantically parse the natural language descriptions in the work content data, identifying the execution actions, tools, and objects within the work content. The complete work is then broken down into indivisible minimum execution units according to the logical sequence of business execution, forming a sequence of atomic task nodes arranged in execution order. Each atomic task node is labeled with a triplet of tool, action, and target object information. Next, the basic attributes of each atomic task node are extracted to generate corresponding structured work description cards. Based on the attribute information in the cards, multi-dimensional feature mapping is performed to generate a task vector for each atomic task node, encompassing eight dimensions: task content description, source system, target system, task target description, task execution plan, task closure receipt, task flow, and task dynamic knowledge graph. Finally, all atomic task nodes carrying task vectors together constitute a task node set.

[0053] For example, the specific methods for collecting task flow data, extracting execution rules, and constructing a task rule vector library can be as follows: For each atomic task node in the task node set, the entire task execution data, including the data flow path in its corresponding business process, the interaction information between nodes, and closed-loop processing samples of historical tasks, is collected. After cleaning and formatting the data, a task flow dataset corresponding to each node is formed. Then, the complete execution logic of each atomic task node is parsed from the task flow dataset, and standardized execution rules, including the mapping relationship between tools, actions, and target objects, are extracted. The execution rules are then converted into serialized rule vectors. Finally, vector indexing and clustering analysis are performed on all rule vectors. A density-based clustering algorithm is used to integrate and classify rule vectors with the same execution logic and similar features, forming a unified, reusable, and matching task rule vector library.

[0054] For example, the specific method for matching the target task executor and determining the AI-capable state through human-machine consistency comparison can be as follows: First, based on the execution rules and task characteristics of each atomic task node in the task rule vector library, the task nodes are classified into different types of task groups, such as mechanical execution, rule judgment, pattern matching, and understanding reasoning. Then, for different task groups, a suitable target task executor is matched from a preset executor set. For tasks that cannot be matched with a general executor, a customized executor is developed. Subsequently, historical sample data of the corresponding task is selected, and the AI ​​execution results are output by running the target task executor, while the manual execution results are output by the business personnel. Finally, the two types of execution results are compared for consistency according to a unified output standard. First, the output format compliance is checked to remove invalid data, and then the consistency of each item is compared. Combining the preset evaluation threshold and the comparison results, it is determined whether each atomic task node meets the requirements for artificial intelligence, and finally the corresponding AI-capable state of each node is determined.

[0055] For example, the specific method for counting the number of AI execution nodes and calculating the AI ​​achievement rate can be: First, the total number of all task nodes corresponding to the target digital employee is counted, including three types of nodes: AI execution nodes, fixed program execution nodes, and human nodes. Then, based on the AI-capable status of each atomic task node, task rule matching verification, executor matching verification, and substitutability assessment verification are carried out on the nodes in sequence. Nodes that pass all three verifications are identified as AI execution nodes and their numbers are counted. Finally, the ratio of the number of AI execution nodes to the total number of all task nodes is calculated to obtain the AI ​​implementation rate value corresponding to the target digital employee, thus completing the quantitative output of the overall AI implementation level.

[0056] For example, taking digital employees in the regional communication cloud positions of telecom operators as the evaluation object, the process for evaluating their artificial intelligence implementation rate is as follows: First, collect all work content data corresponding to this position, including business descriptions, associated systems, and output requirements for 10 work contents such as outbound topology query, resource data verification, alarm inspection, and report submission; then, use a large language model to perform semantic parsing and decompose the 10 work contents into the smallest execution unit, breaking down each work into independent atomic task nodes. For example, outbound topology query is decomposed into atomic nodes such as logging into the network management system, querying the topology of the target device, exporting topology data, and organizing the topology results. A sequence of atomic task nodes is formed according to the execution order, and each node is marked with a triplet information of tool, action, and target object. Then, eight-dimensional attributes such as the source system, target system, and task objective of each node are extracted to generate a task vector, finally forming a task node set including all nodes; then, collect the task flow data of each atomic task node, such as the data flow path of the network management system for the outbound topology query node, the interaction information of the query operation, and the historical topology. The system retrieves closed-loop work order samples and analyzes the execution rules of each node from the data to generate rule vectors. All rule vectors are then clustered and organized, with rules of the same type (query and inspection) categorized to form a task rule vector library. Next, nodes are classified according to the task characteristics of the rule vector library. For example, topology queries belong to the rule judgment type node, and a corresponding program interface executor is matched to it. 100 historical topology query samples are selected, and the executor automatically queries and outputs the AI ​​execution results. Simultaneously, maintenance personnel manually query and output the manual execution results. The output format of both types of results is validated, and the consistency between the topology node and the link is compared. Based on the evaluation criteria, the node is determined to meet the AI ​​requirements, and its AI-capable status is determined to be AI-capable. The status determination of all 10 nodes is completed similarly. Finally, the total number of all 10 task nodes is counted. After rule matching, executor matching, and substitutability checks, 6 nodes are confirmed as AI execution nodes. The AI ​​implementation rate of this digital employee is calculated as 6 divided by 10, or 60%.

[0057] As can be seen, in this embodiment, the work content data corresponding to the target digital employee is obtained and broken down into the smallest execution unit to form a sequence of atomic task nodes. Then, each atomic task node is vectorized to obtain a set of task nodes carrying task vectors. This transforms the originally vague and highly holistic business work into standardized smallest execution units that can be independently measured and precisely labeled. This provides a statistically granular carrier for the quantitative evaluation of the degree of artificial intelligence implementation, fundamentally solving the problems of vague work boundaries for digital employees and the inability to break down and measure the degree of AI implementation in the traditional model. By collecting the task flow data of each atomic task node in the task node set, execution rules are extracted and rule vectors are generated. These are then clustered and organized to form a task rule vector library. This transforms the scattered and unstructured task execution logic into a structured, reusable, and uniformly matchable standardized rule system. This provides a clear rule benchmark for the matching of subsequent task executors and ensures the overall... The consistency and objectivity of the quantitative node evaluation standards avoid the subjective bias caused by manual qualitative judgment, laying a solid rule foundation for quantitative evaluation. Based on the task rule vector library, each atomic task node is matched with a corresponding target task executor. The AI ​​execution results are obtained and compared with the manual execution results. The AI-capable state of each atomic task node is determined based on the comparison results. The objective execution result data comparison is used as the basis for judging whether a node is AI-capable. This ensures that the AI ​​attributes of each atomic task node have verifiable and reproducible data support, achieving accurate quantitative judgment of the AI ​​status of a single node. The number of AI execution nodes is counted based on the AI-capable state of each atomic task node, and the AI ​​implementation rate of the target digital employee is calculated. The abstract AI implementation level is transformed into an intuitive numerical indicator that can be compared horizontally and vertically, completing the full-link quantitative evaluation from the granularity of a single node to the overall indicator of digital employees. Meanwhile, since the entire evaluation process can be repeatedly triggered as work content is updated, task flow rules are iterated, executor capabilities are upgraded, or online operation data fluctuates, it can update the AI-capable status of each node in real time and refresh the AI ​​implementation rate simultaneously. Therefore, it can achieve dynamic measurement of the degree of AI implementation, rather than a one-time static evaluation. The quantified overall AI implementation rate index, coupled with the AI ​​status data refined to each atomic task node, can provide the operations side with accurate data decision-making basis, supporting operations personnel to adjust the configuration of human positions, allocate development and optimization resources, and formulate differentiated human-machine collaboration strategies, ultimately achieving refined operation optimization of digital employees.

[0058] Preferably, in the embodiment, the step of decomposing the work content data into the smallest execution unit to obtain an atomic task node sequence, and then vectorizing each atomic task node in the atomic task node sequence to obtain a task node set consisting of multiple atomic task nodes carrying task vectors, includes: Using a large language model, semantic parsing is performed on the natural language data in the work content data to obtain semantic parsing results; The semantic parsing result is decomposed into the smallest execution unit to obtain the atomic task node sequence marked with triples, wherein the triples include tools, actions, and target objects; Basic attributes are extracted from each atomic task node in the atomic task node sequence to generate a work description card corresponding to each atomic task node. Based on the attribute information in the job description card, multi-dimensional feature mapping is performed on each atomic task node so that each atomic task node carries an eight-dimensional task vector, resulting in the task node set. The eight-dimensional task vector includes task content description, source system, target system, task target description, task execution plan, task closed-loop receipt, task process, and task dynamic knowledge graph.

[0059] Among them, the large language model is a deep learning model trained on massive text data. It has capabilities such as semantic understanding, information extraction, logical organization, and text generation. It can transform unstructured natural language data into structured element information and is widely used in scenarios such as text parsing and knowledge extraction. In this embodiment, the large language model is used to perform semantic parsing on the natural language description of work content, automatically identifying core information such as execution actions, operation tools, business objects, and process sequence logic. It is the core processing carrier for realizing the automated decomposition of work content, and can replace manual work in semantic recognition and element extraction, ensuring the efficiency and standard consistency of task decomposition.

[0060] The semantic parsing result is a set of structured semantic elements extracted and organized after deep semantic understanding of natural language text. It includes core information such as entities, actions, relationships, and attribute constraints within the text, serving as the foundation for subsequent logical processing and data manipulation. In this embodiment, the semantic parsing result is the structured information output by the large language model after parsing the natural language description of the work content. It includes core semantic elements such as execution actions, operating tools, target objects, execution order, related systems, and output requirements within the work content. This clearly reconstructs the inherent execution logic and constraints of the work content, providing a standardized semantic basis for the subsequent decomposition of the smallest execution unit.

[0061] Among them, the triplet is a standardized representation of the attributes and logical relationships of things through three related elements. It has the characteristics of unified structure and clear semantics, and can realize the structured storage and rapid matching of information. In this embodiment, the triplet is a standardized label for the core execution elements of the atomic task node, specifically including three elements: tool, action, and target object. The tool is the system or operation carrier on which the task execution relies, the action is the specific operation behavior of the task execution, and the target object is the business entity affected by the operation. Through the triplet structure, the core execution logic of each atomic task node can be uniformly represented, so that task nodes of different business types have a consistent description standard.

[0062] The work description card is a structured carrier that centrally presents the core attributes of a single work step. It can standardize the recording of various basic information of the work step, enabling unified management and rapid retrieval of work step information. In this embodiment, the work description card is a structured information card generated for each atomic task node. It centrally records various basic attributes of the atomic task node, including core information such as task content, related systems, execution requirements, closed-loop standards, and process logic. It is a standardized presentation of the attribute information of the atomic task node, providing a unified attribute data source for subsequent task vectorization processing.

[0063] Among them, the multi-dimensional task vector is a structured vector formed by feature mapping of multiple attributes of a task. Each dimension corresponds to a core attribute of the task, which can comprehensively and accurately represent the business characteristics and execution attributes of the task. In this embodiment, the eight-dimensional task vector is a task vector formed by feature mapping of eight core business attributes of the atomic task node. The eight dimensions are task content description, source system, target system, task target description, task execution plan, task closed-loop receipt, task process, and task dynamic knowledge graph. It covers all dimensions of features such as task content attributes, system association, execution target, execution plan, closed-loop requirements, process logic, and knowledge system. It can completely represent all business attributes of the atomic task node and support subsequent processing processes such as rule extraction, executor matching, and effect evaluation.

[0064] For example, by using a large language model to perform semantic parsing on the natural language data in the work content data, the specific method to obtain the semantic parsing results can be as follows: The collected work content data, described in natural language, is input into a large language model. Leveraging the model's semantic understanding and information extraction capabilities, core semantic elements such as business actions, operational tools, target objects, execution sequence logic, related systems, and output timeliness requirements are identified. Simultaneously, the model analyzes the internal connections and constraints of the work content, transforming the unstructured natural language work description into a structured set of semantic elements. The resulting semantic analysis provides a clear semantic basis for subsequent task unit decomposition, avoiding issues such as inconsistent standards and missing elements that can occur with manual decomposition.

[0065] For example, the method of decomposing the semantic parsing result into the smallest execution unit to obtain the sequence of atomic task nodes marked with triples can be as follows: Based on the execution logic and element information extracted from the semantic parsing results, and following the granularity standard of "single execution action, indivisible", the complete work content is broken down into multiple independent minimum execution units, each unit corresponding to an atomic task node. At the same time, all atomic task nodes are arranged according to the logical order of business execution to form a complete sequence of atomic task nodes. Each atomic task node in the sequence is labeled with the corresponding tool, action, and target object triple information, clarifying the core execution elements of each node, so that each atomic task node has a standardized execution logic representation.

[0066] For example, the method of extracting basic attributes from each atomic task node in the atomic task node sequence and generating a corresponding job description card for each atomic task node can be as follows: For each node in the atomic task node sequence, basic attribute information is extracted from the semantic parsing results, original work content data, and corresponding business system metadata. This includes the specific content of the task, the business system from which the data comes, the target system for the output results, the business objectives to be achieved by the task, the execution frequency and timeliness requirements of the task, the closed-loop feedback method after the task is completed, the detailed execution process steps of the task, and the business knowledge system associated with the task. All attribute information is classified, organized, and summarized to generate a structured work description card for each atomic task node, thereby achieving standardized and centralized management of single-node attribute information.

[0067] For example, based on the attribute information in the job description card, multi-dimensional feature mapping is performed on each atomic task node so that each atomic task node carries an eight-dimensional task vector. The specific method for obtaining the task node set can be as follows: Based on the attribute information in the job description card, and according to the preset eight-dimensional classification standard, the attribute information in the card is mapped to eight dimensions: task content description, source system, target system, task target description, task execution plan, task closure feedback, task process, and task dynamic knowledge graph. The information in each dimension is characterized and encoded to generate vector features for the corresponding dimension. Finally, these features are integrated to form an eight-dimensional task vector for each atomic task node. All atomic task nodes carrying eight-dimensional task vectors together constitute a task node set, completing the full transformation from unstructured job content to a structured, computable set of task nodes.

[0068] In this embodiment, a large language model is used to automate semantic parsing and element extraction of work content, replacing manual conversion of unstructured text into structured information. Then, a standardized decomposition of the smallest execution unit is achieved using triplet labeling, ensuring uniform granularity and completeness of core elements in task decomposition. Simultaneously, work description cards are used to centrally organize node attributes, ultimately mapping and generating task vectors covering eight dimensions. This establishes a complete automated conversion link from natural language work descriptions to structured task nodes. On the one hand, this significantly improves the efficiency of work content decomposition and vectorization, reducing the cost and subjective bias of manual decomposition. On the other hand, the eight-dimensional task vector comprehensively covers the core attributes of the task, such as content, system, objective, process, and knowledge, making the business representation of atomic task nodes more complete and accurate. This provides a high-quality structured data foundation for subsequent rule extraction, executor matching, and AI-based evaluation, effectively improving the accuracy and reliability of the overall evaluation system.

[0069] Preferably, in this embodiment, the step of collecting task flow data corresponding to each atomic task node in the task node set, extracting the execution rules of each atomic task node from the task flow data and generating rule vectors, and clustering and organizing all rule vectors to obtain a task rule vector library includes: For each atomic task node in the task node set, the data flow path, interaction information and historical closed-loop samples in the corresponding task flow are collected. Based on the data flow path, the interaction information and the historical closed-loop samples, the task flow dataset corresponding to each atomic task node is obtained. The execution logic of each atomic task node is parsed from the task flow dataset, and execution rules including tool, action, and target mapping relationship are extracted. Based on the execution rules, a serialized rule vector is generated. The rule vectors are indexed and clustered to integrate and classify rule vectors with the same execution logic, thus obtaining the task rule vector library.

[0070] The data flow path refers to the complete link of business data transmission and processing between different systems and processing nodes during process execution. It clearly records the data's origin, processing stages, and final destination, serving as the core basis for understanding business process logic and locating data processing nodes. In this embodiment, the data flow path refers to the transmission link of business data between various related business systems and processing stages during the execution of atomic task nodes. For example, alarm data originates from the intelligent monitoring system, passes through the fault prediction system and the one-click scheduling system, and finally flows into the work order system and fault management system—a complete transmission route that can intuitively reconstruct the data flow and node connection relationships during task execution.

[0071] Interactive information refers to the data exchanged between different systems, nodes, and operators during process execution, including instruction transmission, status feedback, and operation triggers. It records the triggering conditions and status change signals for process advancement. In this embodiment, interactive information refers to information such as instruction exchange data, operation trigger signals, execution status receipts, and node transition triggering conditions between related systems during atomic task execution. Examples include the level judgment result sent to the scheduling node after fault prediction, the notification instruction sent to repair personnel after work order dispatch, and the status receipt after task node completion. This information is crucial for reconstructing the task execution triggering logic.

[0072] Historical closed-loop samples refer to historical task cases that have completed the entire process and formed a complete business closed loop. This includes full-link information such as task input conditions, complete processing procedures, final output results, and acceptance confirmation status. These samples form the foundational data for mining task execution patterns and extracting standardized rules. In this embodiment, historical closed-loop samples refer to all records corresponding to atomic tasks that have historically completed the entire process and formed a complete business closed loop. This includes complete information such as task input parameters, execution procedures, output results, and final acceptance status. For example, complete processing records of two hundred fault scheduling tasks that have completed closed loops within the past three months provide empirical data for extracting task execution rules.

[0073] The task flow dataset is a standardized dataset formed by aggregating various data related to the entire process of a specific task, after cleaning and normalization. It covers the entire chain of task execution information and can support subsequent analysis work such as rule extraction and process optimization. In this embodiment, the task flow dataset refers to a unified dataset formed by aggregating the corresponding data flow path, node interaction information, historical closed-loop samples, and other full-dimensional task flow data for a single atomic task node, after deduplication, cleaning, and format standardization. It is the core data source for parsing task execution logic and extracting execution rules.

[0074] The serialized rule vector is generated by arranging rule elements into an ordered sequence according to the chronological order of business execution, followed by feature encoding. This process fully preserves the temporal logic and execution order of the rules, making it more closely aligned with the execution logic of real business processes. In this embodiment, the serialized rule vector is generated by organizing the execution rules of atomic tasks into an ordered sequence of rule units according to the chronological order of business execution, and then encoding the features of each element in the sequence. This process fully preserves the temporal logic of task execution and the flow of nodes, making the rule representation more closely resemble the actual business execution process.

[0075] Vector index construction is the process of establishing an efficient retrieval index structure for massive vector data. An optimized index structure can significantly improve the processing efficiency of vector similarity calculation and near nearest neighbor retrieval, making it a core technology supporting rapid matching of large-scale vector data. In this embodiment, vector index construction refers to establishing an efficient retrieval index structure for the rule vectors corresponding to all atomic task nodes. Index optimization improves the speed of similarity calculation and matching retrieval of rule vectors, providing efficient retrieval capabilities for subsequent rule clustering and executor matching stages.

[0076] Clustering and sorting involves using clustering algorithms to automatically classify data into different categories based on feature similarity. Data within the same category has high feature similarity, while data between different categories shows significant feature differences, enabling automatic data classification and redundancy merging. In this embodiment, clustering and sorting refers to using a density-based clustering algorithm. Based on the feature similarity of rule vectors, rule vectors with the same execution logic and similar features are automatically grouped into the same category, merging duplicate and redundant rule entries, and standardizing and organizing rules of the same category to achieve automatic classification and integration of task rules.

[0077] For example, for each atomic task node in the task node set, the data flow path, interaction information, and historical closed-loop samples in the corresponding task flow are collected. Based on the data flow path, the interaction information, and the historical closed-loop samples, the task flow dataset corresponding to each atomic task node can be obtained in the following specific way: For each atomic task node in the task node set, it connects to the corresponding business system logs, process engine data, work order system history records, operation audit data, and other data sources. It collects the complete flow path of business data during the execution of the node, the interaction instructions and status feedback information between each related system and the node, and all task case data that have completed the closed loop in the past. Then, the collected multi-source heterogeneous data is deduplicated, cleaned, and formatted to remove invalid abnormal data and duplicate records. It is then organized according to a unified data structure to finally form a task flow dataset that covers the entire link information for each atomic task node.

[0078] For example, the specific method for parsing the execution logic of each atomic task node from the task flow dataset, extracting execution rules including tool, action, and target mapping relationships, and generating a serialized rule vector based on the execution rules can be as follows: By employing a process mining approach, the complete execution logic chain of atomic tasks is reconstructed from the task flow dataset. The triggering conditions, operation steps, flow conditions, and output requirements of task execution are identified. The mapping relationship between the operation tool, execution action, and target object corresponding to each execution step is extracted to form standardized execution rule units. Then, according to the chronological order of business execution, each execution rule unit is arranged into an ordered rule sequence. Finally, feature encoding transformation is performed on various elements in the rule sequence to generate a serialized rule vector that retains the complete temporal logic, realizing the transformation from execution logic to a computable vector.

[0079] For example, the method of constructing and clustering the rule vectors to integrate and classify rule vectors with the same execution logic to obtain the task rule vector library can be as follows: First, a vector retrieval index is constructed for the rule vectors corresponding to all atomic task nodes to optimize the retrieval efficiency of vector similarity calculation. Then, a density-based clustering algorithm is adopted, using the feature similarity of rule vectors as the judgment criterion, to automatically group rule vectors with consistent execution logic and similar business characteristics into the same rule category, merge duplicate and redundant rule entries, and standardize and unify each category of rules. Finally, all the classified and standardized rule vectors are uniformly stored and managed to form a task rule vector library with clear classification and fast retrieval and reuse.

[0080] In this embodiment, a task flow dataset is constructed by collecting three types of data: data flow paths, interaction information, and historical closed-loop samples. This comprehensively covers the link logic, interaction logic, and empirical cases of task execution, ensuring that the data foundation for extracting execution rules is comprehensive and closely matches real business scenarios. By analyzing the execution logic through process mining and generating serialized rule vectors, the core execution elements of the task—tools, actions, and objectives—are accurately extracted, while the temporal logic of task execution is fully preserved, making the rule representation more consistent with actual business processes. Furthermore, the automatic classification and integration of rules are achieved through vector index construction and clustering, which not only improves the processing efficiency of rule retrieval and matching but also eliminates duplicate and redundant rule entries, forming a standardized and reusable task rule vector library. This realizes the automatic conversion from scattered business operation data to a structured rule system, providing a unified and accurate rule benchmark for subsequent task executor matching, and ensuring the consistency of the evaluation standards for all nodes, effectively improving the objectivity, processing efficiency, and rule reusability of the entire evaluation system.

[0081] Preferably, in an embodiment, the step of matching a corresponding target task executor for each atomic task node based on the task rule vector library, obtaining the artificial intelligence execution result of each atomic task node through the target task executor, obtaining the manual execution result of each atomic task node, and performing a consistency comparison between the artificial intelligence execution result and the manual execution result, and determining the AI-capable state of each atomic task node based on the consistency comparison result, includes: Based on the execution rules and task characteristics of each atomic task node in the task rule vector library, the atomic task nodes are divided into execution types to obtain task node groups corresponding to different execution types. For each of the task nodes grouped, a target task executor of the corresponding type is matched from a preset executor set; Obtain a task sample, execute the task sample through the target task executor to obtain the artificial intelligence execution result, and obtain the human execution result corresponding to the task sample; The consistency comparison between the artificial intelligence execution result and the manual execution result is performed, and the artificial intelligence-capable state corresponding to each atomic task node is determined based on the consistency comparison result and the preset consistency threshold.

[0082] The execution type classification, based on core characteristics such as the complexity of task execution logic, the degree of decision-making involvement, the clarity of rules, and the reusability of scenarios, categorizes task nodes into different types. This process serves as a prerequisite for matching and adapting execution schemes and establishing differentiated evaluation standards. In this embodiment, the execution type classification is based on execution rule data in the task rule vector library. It extracts features such as the degree of operational fixity, the proportion of judgment and decision-making, and the frequency of scenario changes for each atomic task node, classifying nodes into different types such as mechanical execution, rule judgment, pattern matching, and reasoning. This provides a classification basis for subsequent executor matching and evaluation standard setting.

[0083] Task node grouping is a centralized management method that groups task nodes with similar execution characteristics together. Tasks within the same group are matched with the same type of execution scheme and evaluation criteria, enabling batch configuration and unified optimization, thus improving processing efficiency. In this embodiment, task node grouping is a set of nodes formed by grouping atomic task nodes with the same execution type, similar complexity, and similar execution logic characteristics into the same group after dividing them by execution type. Nodes in the same group can be matched with the same type of task executor, and the optimization and iteration of the executor can cover the entire group of nodes, significantly reducing the cost of repetitive configuration.

[0084] The executor set is a collection of various task execution units with different execution capabilities, covering all levels of task requirements from simple fixed operations to complex reasoning and decision-making. It can be matched and invoked as needed based on task characteristics. In this embodiment, the executor set is a pre-organized and integrated collection of multiple types of execution units, including four categories: pure program orchestration execution units, intelligent agent orchestration execution units, hybrid orchestration execution units, and custom-developed execution units. Different types are adapted to task scenarios of varying complexity, providing standardized execution carrier options for various atomic task nodes.

[0085] The task samples are a set of representative test cases selected from historically completed tasks, covering common scenarios, common anomalies, and boundary conditions. They are used to verify the actual operational effectiveness of the execution plan and serve as the foundational data for comparative testing. In this embodiment, the task samples are selected from historical closed-loop tasks corresponding to the atomic tasks using stratified sampling. They cover common business scenarios, common anomaly scenarios, and typical boundary scenarios, possessing clear input conditions and standard output results. This allows for simultaneous support of both AI-driven and human-driven testing, ensuring the objectivity and comprehensiveness of the verification results.

[0086] The consistency threshold is a pre-set quantitative critical standard. When the consistency index obtained from the comparison reaches or exceeds this critical value, the object to be evaluated is determined to meet the qualification requirements. It is the core basis for eliminating subjective judgment bias and unifying the evaluation scale. In this embodiment, the consistency threshold is a qualification judgment standard set according to the business requirements and risk levels of different task types. It includes critical values ​​of multiple core indicators such as output content consistency rate, accuracy rate, and availability rate. Different thresholds are corresponding to different types of tasks such as mechanical execution, rule judgment, and understanding reasoning. It is a quantitative benchmark for determining whether a node can be artificially automated.

[0087] For example, based on the execution rules and task characteristics of each atomic task node in the task rule vector library, the atomic task nodes are divided into execution types, and the specific method for grouping task nodes corresponding to different execution types can be as follows: Using rule data from the task rule vector library as input, core features such as the degree of fixed operation, the proportion of decision judgment, the clarity of rules, and the frequency of scene changes for each atomic task node are extracted. Based on the classification criteria of mechanical execution, rule judgment, pattern matching, understanding and reasoning, and innovative decision-making, all atomic task nodes are classified. Nodes with similar execution features and complexity are grouped into the same group, resulting in multiple task node groups corresponding to different execution types, thus achieving hierarchical classification management of task nodes.

[0088] For example, the specific method for matching the target task executor of the corresponding type from a preset executor set for each task node group can be as follows: For each task node group, the task executor with the highest suitability is selected from the executor set based on the characteristics of the task group, such as execution type, operation complexity, and system integration requirements. For task groups with fixed operations and clear rules, a general-purpose executor of pure program orchestration or hybrid orchestration is directly matched. For task groups with complex reasoning and no general adaptation solution, they are marked as pending custom development, and a custom executor will be customized according to the task rules. Finally, the corresponding target task executor is determined for each task node group to ensure that the execution capability and task requirements are accurately matched.

[0089] For example, the specific method for obtaining a task sample, executing the task sample through a target task executor to obtain the artificial intelligence execution result, and obtaining the corresponding human execution result of the task sample can be as follows: For each atomic task node's business scenario, a stratified sampling method is used to select a sufficient number of representative task samples from historical closed-loop tasks, ensuring that the samples cover common scenarios, common anomalies, and boundary conditions. All task samples are batch-input into the matched target task executor for processing, and the output content, processing time, execution status, and other data are fully recorded to obtain the artificial intelligence execution results. At the same time, identical task samples are given to business personnel in the corresponding positions, who manually process them according to standard operating procedures and record the results to obtain the same source of manual execution results, ensuring that the two sets of data have a basis for comparison.

[0090] For example, the method of comparing the consistency of AI execution results and human execution results, and determining the AI-capable state of each atomic task node based on the consistency comparison results and a preset consistency threshold, can be as follows: The AI ​​execution results and human execution results from the same source are verified item by item according to a unified data standard. Quantitative consistency indicators such as output content consistency rate, result accuracy rate, and output usability rate are calculated. Each indicator is compared with the preset consistency threshold corresponding to the task type. If all core indicators meet or exceed the threshold requirements, the atomic task node is determined to be in an AI-enabled state. If any core indicator does not meet the threshold, it is determined to be in a temporarily unenabled state. Finally, the state determination of all atomic task nodes is completed.

[0091] In this embodiment, nodes are first categorized and grouped according to execution rules and task characteristics. Then, task executors of the corresponding types are matched according to the groups. This achieves precise adaptation between execution capabilities and task requirements, avoiding deviations in execution effects caused by mismatches between executors and task types. The categorized batch matching mode also significantly improves the efficiency of executor configuration for multiple nodes. Human-machine dual-track execution testing using source task samples ensures the homogeneity and comparability of comparison data. Furthermore, using differentiated consistency thresholds as the judgment standard provides objective and unified quantitative basis for the AI-based judgment of different types of nodes, effectively reducing the subjective errors of manual judgment. This embodiment establishes a complete standardized process from task classification and executor adaptation to human-machine comparison and judgment. This ensures the accuracy and rationality of the AI-based status judgment of each node and improves the evaluation and processing efficiency of batch nodes, providing reliable node status data support for the accurate calculation of subsequent artificial intelligence achievement rates.

[0092] Preferably, in the embodiment, the consistency comparison between the artificial intelligence execution result and the manual execution result includes: The output data format of each node in the data stream is unified, and a dual-table storage scheme is used to store the manual execution results and the artificial intelligence execution results respectively, resulting in a dual-track output dataset. The dual-track output dataset is subjected to output compliance detection, and data that does not conform to the preset output format specifications is removed to obtain a valid output dataset; The target AI execution results and target human execution results in the effective output dataset are compared item by item to obtain the consistency comparison results.

[0093] The output data format is a unified specification for the presentation, field structure, data type, and encoding rules of task execution results. It serves as the fundamental standard for data interaction, comparison, and storage between different systems. In this embodiment, the output data format is a unified format specification for the execution results of each atomic task node. It clarifies the requirements for the field composition, data type, hierarchical structure, and naming rules of the result data. For example, the topology query results are uniformly specified to include fixed fields such as node name, link relationship, port status, and data center, ensuring that the AI ​​execution results and human execution results have a unified comparison benchmark and eliminating comparison errors caused by format differences.

[0094] The dual-table storage scheme is a storage strategy that stores two sets of comparison data from the same source but different origins in two independent data tables. The two tables use completely identical table structures and field specifications, ensuring that the two sets of data are independent and do not interfere with each other, while also supporting convenient correlation and comparison. This approach is commonly used in scenarios such as human-machine comparison and dual-system verification. In this embodiment, the dual-table storage scheme refers to storing the results of manual execution and the results of artificial intelligence execution in two data tables with completely identical structures. One table stores all the results data from manual execution, and the other table stores the corresponding results data from artificial intelligence execution. The field definitions, data types, and primary key rules of the two tables are completely unified, ensuring the independence and integrity of the two data sources, supporting accurate correlation and comparison based on the primary key of the task samples, and adapting to both synchronous comparison in the testing environment and asynchronous collection in the production environment.

[0095] The dual-track output dataset is a complete dataset acquired through a dual-track acquisition method, comprising two sets of comparative data from the same source. These two sets of data correspond to different execution entities, supporting subsequent difference analysis and effect evaluation. In this embodiment, the dual-track output dataset is a complete comparative dataset formed by storing the results of manual execution and the results of artificial intelligence execution separately using a dual-table storage scheme. It includes all output data obtained from the same batch of task samples by both manual and artificial intelligence processing. The data are one-to-one and structurally consistent, serving as the fundamental data source for consistency comparison.

[0096] The output compliance check is a preliminary threshold verification of the output data according to preset format specifications and data standards. It filters out valid data that meets the format requirements and removes invalid data with format errors, missing fields, or abnormal data types, ensuring the data quality for subsequent comparative analysis. In this embodiment, the output compliance check is a pre-verification step for consistency comparison. According to preset output format specifications, each result data in the dual-track output dataset is verified, checking for field completeness, data type compliance, and format standardization. Abnormal data that does not meet the specifications is removed to avoid format-related issues interfering with subsequent content consistency judgments.

[0097] The output format specification is a written stipulation of the format requirements for data output, including specific requirements such as field list, required fields, data types, value ranges, encoding rules, and structural hierarchy. It serves as the basis for data compliance verification. In this embodiment, the output format specification is a format standard formulated for the output results of different types of atomic task nodes. It clarifies the required fields, optional fields, data types, and value constraints for various task outputs. For example, the fault level determination result must include four required fields: fault number, determination level, determination basis, and determination time. The level value is limited to level one to four, serving as the benchmark for conducting output compliance testing.

[0098] The effective output dataset is the set of qualified data obtained after the original dataset has undergone quality verification and the removal of invalid and abnormal data. The data quality meets the requirements for subsequent analysis and processing and can be directly used for comparison, calculation, and other operations. In this embodiment, the effective output dataset is the set of all remaining qualified data after the dual-track output dataset has undergone output compliance testing and invalid data such as missing fields, format errors, and type anomalies has been removed. The results of manual execution and artificial intelligence execution still maintain a one-to-one correspondence, and the data quality meets the requirements for item-by-item consistency comparison. It is the direct processing object for subsequent content comparison.

[0099] The item-by-item consistency comparison is a process of matching and verifying each item of corresponding data in two sets according to preset comparison dimensions. This can accurately quantify the degree of consistency between the two sets of data and locate specific differences. In this embodiment, the item-by-item consistency comparison performs content matching and verification on each field of the output result for each corresponding sample in the valid output dataset. For example, it compares the number of nodes, the number of links, the port status, and the data center affiliation in the topology query result, and counts the proportion of consistent items. Finally, a quantitative consistency comparison result is obtained, which accurately reflects the degree of agreement between the artificial intelligence execution result and the human benchmark.

[0100] For example, to unify the output data format of each node in the data stream and use a dual-table storage scheme to store the results of manual execution and artificial intelligence execution respectively, the specific method for obtaining a dual-track output dataset can be as follows: First, a unified output data format standard is established for the output content of each atomic task node, clarifying the field composition, data type, structural hierarchy and naming rules, and standardizing the output format of both human and artificial intelligence execution subjects. Then, a dual-table storage scheme is adopted to establish two data tables with completely identical structures, which are used to store the human execution results and the artificial intelligence execution results respectively. The two types of execution results of the same batch of task samples are stored in the two tables according to the sample primary key, and finally summarized to form a dual-track output dataset with a unified structure and one-to-one correspondence.

[0101] For example, the specific method for performing output compliance checks on the dual-track output dataset and removing data that does not conform to the preset output format specifications to obtain a valid output dataset can be as follows: Using the preset output format specifications as the verification standard, each piece of data in the dual-track output dataset is pre-verified, checking the completeness of fields, compliance of data types, reasonableness of value range, and format specifications of each piece of data. Invalid data with problems such as missing fields, incompatible data types, and disordered formats are marked and removed. At the same time, the other set of same source data corresponding to the invalid data is removed to ensure the pairing integrity of the remaining data. Finally, a valid output dataset that meets all format requirements is obtained, eliminating the interference of format-related issues on subsequent content comparison.

[0102] For example, the consistency comparison of the target AI execution result and the target human execution result in the valid output dataset can be performed item by item to obtain the consistency comparison result. Using task samples as units, the corresponding AI execution results and human execution results in the effective output dataset are paired. Content matching and verification are carried out one by one according to each field of the output results to compare whether the content of each sub-item is consistent. The number and proportion of consistent items in all compared sub-items are counted, and the specific sub-items with differences and their contents are recorded. Finally, a consistency comparison result including the overall consistency rate, sub-item consistency rate, and difference details is generated to quantitatively present the degree of conformity between the AI ​​execution effect and the human benchmark.

[0103] In this embodiment, a unified output data format is first implemented, and a dual-table storage scheme is used to construct a dual-track output dataset. This eliminates comparison errors caused by format differences from the data source, ensuring that the two types of execution results have a unified comparison benchmark. At the same time, the independent storage mode of the two tables not only ensures the integrity and independence of the two types of data sources, but also adapts to the collection needs of multiple scenarios in testing and production. By eliminating abnormal format data through pre-output compliance detection, format verification and content comparison are separated, avoiding interference from format-related issues in judging content consistency and effectively improving the accuracy of comparison results. Furthermore, fine-grained verification of each field is achieved through item-by-item consistency comparison, which not only quantifies the overall consistency level, but also accurately locates specific differences, making the comparison results more valuable.

[0104] Preferably, in the embodiment, determining the AI-capable state corresponding to each atomic task node based on the consistency comparison result includes: A four-dimensional evaluation framework including reliability, adaptability, cost-effectiveness, and boundary handling is constructed. Atomic task nodes are hierarchically divided according to task characteristics to obtain multiple task levels. Corresponding accuracy thresholds and alternative strategies are set for different task levels. Based on the four-dimensional evaluation framework, multiple task levels, accuracy thresholds, and alternative strategies, a hierarchical evaluation standard system is formed. Based on the aforementioned hierarchical evaluation standard system, from four dimensions—capability, cost, risk, and data volume—corresponding evaluation indicators and alternative thresholds are configured for atomic task nodes at each task level, resulting in a multi-dimensional evaluation indicator system. The consistency comparison results and the execution data of the corresponding atomic task nodes are input into the multi-dimensional evaluation index system to obtain the scoring results of each dimension. The scores of each dimension are integrated and calculated through the comprehensive judgment matrix to obtain the initial artificial intelligence-capable state corresponding to each atomic task node. Based on a preset veto list, the initial AI-capable state is adjusted to obtain the target AI-capable state.

[0105] The four-dimensional evaluation framework is a top-level evaluation architecture that comprehensively assesses the feasibility of AI-based tasks from multiple core dimensions. It avoids the one-sidedness of evaluation based on a single indicator and ensures the comprehensiveness of the evaluation results. In this embodiment, the four-dimensional evaluation framework includes four evaluation dimensions: reliability, adaptability, cost-effectiveness, and boundary handling. These correspond to four core evaluation directions: the accuracy and stability of AI execution, adaptability to scene changes, cost-effectiveness, and ability to handle extreme scenarios. It fully covers the four core aspects of AI application implementation: effectiveness, adaptability, cost, and security, and serves as the top-level guiding framework for the entire AI-capable state assessment.

[0106] Reliability measures the accuracy and stability of AI task execution, focusing on whether the AI ​​output is correct and whether repeated executions are consistent. It is a fundamental prerequisite for determining whether AI can handle the corresponding task. In this embodiment, the reliability dimension corresponds to specific evaluation indicators such as task success rate, decision consistency rate, output availability rate, and repeatability consistency. Quantitative data is used to verify whether the AI ​​execution results meet the acceptable standards for human execution, ensuring the execution quality after AI replacement.

[0107] Adaptability measures the ability of artificial intelligence to maintain performance despite changes in business scenarios and fluctuations in input data distribution. Its core function is to examine whether the AI ​​can still consistently output satisfactory results after environmental changes, preventing significant performance degradation due to scenario drift after deployment. In this embodiment, the adaptability dimension corresponds to specific evaluation indicators such as boundary stability, drift sensitivity, and throughput elasticity, verifying the AI's tolerance to input fine-tuning, business volume fluctuations, and scenario changes, ensuring the long-term operational stability of the AI ​​in a real production environment.

[0108] Cost-effectiveness is used to measure the input and output of artificial intelligence replacing human labor. It compares the cost differences between AI and human labor from a full life cycle perspective to determine whether AI replacement has economic value, and is a core reference for operational decisions. In this embodiment, the cost-effectiveness dimension corresponds to specific evaluation indicators such as single task operation cost, fixed investment payback period, error loss cost, and human resource release value, covering costs in all aspects of development, operation, and risk, providing data support for cost allocation and resource allocation.

[0109] Boundary handling measures the ability of artificial intelligence to handle extreme and abnormal scenarios and edge input cases. Its core focus is on whether AI can correctly handle or smoothly transition to human intervention in unconventional scenarios, preventing serious errors and losses caused by edge situations. In this embodiment, the boundary handling dimension corresponds to specific evaluation indicators such as fatal error rate, anomaly handling rate, and edge use case accuracy rate, controlling the risk threshold of AI applications and ensuring business security in extreme scenarios.

[0110] The task hierarchy is based on core characteristics such as decision complexity, rule clarity, scenario uncertainty, and innovation requirements, classifying task nodes into different levels. The AI ​​difficulty and risks vary significantly across different task levels, requiring different evaluation standards and implementation strategies. In this embodiment, the task hierarchy is divided into five levels from low to high: mechanical execution, rule judgment, pattern matching, understanding and reasoning, and innovative decision-making. Higher levels involve a greater proportion of decision-making and stronger scenario uncertainty, leading to differentiated evaluation standards and alternative strategies, thus achieving precise evaluation through hierarchical classification.

[0111] The accuracy threshold is a minimum accuracy threshold set for different task levels to determine whether the AI's execution effect is acceptable. It is a core quantitative benchmark for evaluating the quality of AI execution, and the threshold level is matched with the risk level and complexity of the task. In this embodiment, the accuracy threshold is the highest for mechanical execution tasks, followed by rule judgment and pattern matching tasks, and then the threshold is further relaxed for understanding and reasoning tasks. For innovative decision-making tasks, high accuracy is not pursued as a substitute. Differentiated thresholds ensure that the evaluation standard is adapted to the task's risk and characteristics.

[0112] The substitution strategy is a set of AI application strategies tailored to different task levels. It clarifies the AI ​​application methods and the degree of human involvement required for each task level, guiding the specific implementation of AI. In this embodiment, the substitution strategy is set separately for different task levels: a full substitution strategy is used for mechanical execution tasks; a substitution plus sampling inspection strategy is used for rule-based judgment tasks; a human-machine collaboration strategy is used for understanding and reasoning tasks; and a strategy of only human assistance without substitution is used for innovative decision-making tasks, ensuring a high degree of matching between AI application methods and task characteristics.

[0113] The hierarchical evaluation standard system is a standardized evaluation system composed of a top-level four-dimensional evaluation framework, multi-level task hierarchy, and accuracy thresholds and alternative strategies for each level. This system addresses the problem of using the same evaluation standard for tasks of varying complexity. In this embodiment, the system sets evaluation scales hierarchically according to task characteristics, ensuring both the uniformity of evaluation standards and the differences between different tasks. It serves as the general basis for subsequent refinement of evaluation indicators and the implementation of specific evaluation work.

[0114] The multi-dimensional evaluation indicator system is a set of specific indicators further refined from the hierarchical evaluation standard system in four directions: capability, cost, risk, and data volume. Each indicator corresponds to a clear calculation formula, measurement method, and alternative threshold value, thus grounding the top-level evaluation framework into a quantifiable and verifiable benchmark. In this embodiment, the capability dimension covers execution effect and efficiency indicators, the cost dimension covers full lifecycle cost indicators, the risk dimension covers safety and anomaly indicators, and the data volume dimension covers the sample size requirements for different evaluation stages, providing comprehensive quantitative support for AI-based status determination.

[0115] The comprehensive judgment matrix is ​​a decision-making tool that integrates multiple indicators with weighted averages. By setting matching weight coefficients for different evaluation dimensions, it integrates the scattered scores from multiple dimensions into a single comprehensive score, avoiding the bias of judgment based on a single dimension. In this embodiment, the comprehensive judgment matrix sets corresponding weights for core indicators such as task success rate, time consumption ratio, cost ratio, and fatal error rate. It then performs weighted calculations on the scores of each dimension to obtain a comprehensive evaluation result, achieving scientific integration of multiple indicators and ensuring the comprehensiveness and rationality of the final judgment.

[0116] The initial AI-capable state is a preliminary assessment of the AI-capable status of atomic task nodes obtained after multi-dimensional indicator scoring and comprehensive judgment matrix operations. It only reflects the AI ​​feasibility of the task under conventional evaluation dimensions and has not yet undergone hard constraint verification such as security and compliance. In this embodiment, the initial state is only derived from the comprehensive score of dimensions such as execution effect, cost, and conventional risks. It serves as the basis for the subsequent final judgment but is not directly used as the final AI-capable state.

[0117] The veto list is a pre-defined set of high-risk and compliance red-line scenarios. If a task falls within a scenario on this list, regardless of how excellent other evaluation results are, it cannot be considered a complete AI replacement. This serves as a hard safety and compliance constraint for AI applications. In this embodiment, the veto list includes six categories: scenarios involving personal safety, scenarios requiring human approval due to compliance red lines, scenarios requiring human decision-making by law, scenarios requiring empathy, scenarios with unverifiable outputs, and scenarios where users explicitly reject AI. This aims to prevent the safety and compliance risks arising from blindly replacing AI in high-risk scenarios from the outset.

[0118] The target AI-capable status is the final AI-capable status determination result after adjustment and verification through a veto list. It takes into account both the feasibility of AI under conventional dimensions and adheres to the hard bottom line of security and compliance, and serves as the final judgment basis for statistically analyzing AI execution nodes. In this embodiment, the target status integrates considerations of effectiveness, cost, risk, and compliance, resulting in a more rigorous judgment that better aligns with the security operation requirements of actual business.

[0119] For example, a four-dimensional evaluation framework can be constructed by hierarchically dividing atomic task nodes according to task characteristics, and setting corresponding accuracy thresholds and alternative strategies for different task levels to form a hierarchical evaluation standard system. First, a top-level evaluation framework is established, encompassing four dimensions: reliability, adaptability, cost-effectiveness, and boundary handling, clarifying the evaluation direction and core content of each dimension. Then, characteristics such as the operational fixity, decision complexity, scenario uncertainty, and innovation requirements of each atomic task node are extracted, dividing the task nodes into multiple task levels ranging from mechanical execution to innovative decision-making. For each level, corresponding accuracy thresholds and suitable alternative strategies are matched. Finally, the above content is integrated to form a hierarchical evaluation standard system covering the top-level framework, level division, threshold standards, and alternative strategies, providing a hierarchical classification standard basis for subsequent evaluations.

[0120] For example, based on a hierarchical evaluation standard system, from four dimensions—capability, cost, risk, and data volume—corresponding evaluation indicators and alternative thresholds are configured for atomic task nodes at each task level. The specific method for obtaining a multi-dimensional evaluation indicator system can be as follows: Based on the hierarchical evaluation standard system, specific evaluation indicators are broken down into four dimensions—capability, cost, risk, and data volume—for each task level. The corresponding calculation formula, measurement method, and quantitative alternative threshold value are clearly defined for each indicator to ensure that each indicator can be calculated and verified. Finally, a complete multi-dimensional evaluation indicator system is formed, which transforms the top-level evaluation framework into a quantifiable and executable evaluation benchmark to support subsequent quantitative scoring and judgment work.

[0121] For example, the consistency comparison results and the execution data of the corresponding atomic task nodes are input into a multi-dimensional evaluation index system to obtain the scoring results of each dimension. The initial AI-capable state corresponding to each atomic task node is obtained by integrating and calculating the scores of each dimension through a comprehensive judgment matrix. The consistency comparison results obtained in the early stage, as well as the actual execution data such as the running time, cost data, error records, and sample data of the atomic task node, are input into the multi-dimensional evaluation index system. The actual values ​​of each index under the four dimensions of capability, cost, risk, and data volume are calculated respectively. The scores of each dimension are obtained by comparing them with the corresponding level of substitution threshold. Then, through the comprehensive judgment matrix, matching weight coefficients are set for different evaluation dimensions, and the scores of each dimension are weighted and integrated to obtain the comprehensive evaluation result of the task node. The comprehensive result is compared with the qualification standard to determine the initial artificial intelligence-capable state of the node.

[0122] For example, the method of adjusting the initial AI-capable state according to a preset veto list to obtain the target AI-capable state can be as follows: By comparing each atomic task node with a pre-defined veto list, the business scenario to which it belongs is checked to determine whether it falls under the veto scenario on the list. If the task involves high-risk veto scenarios such as personal safety, compliance red lines, or legal human decision-making, the initial AI-capable state is adjusted and determined to be unsuitable for full AI-capability, regardless of whether the initial assessment result meets the standard. If the task does not involve any veto scenarios, the initial judgment result is retained. The final result is the target AI-capable state that balances execution effectiveness, cost-effectiveness, and safety compliance.

[0123] In this embodiment, by establishing a four-dimensional evaluation framework and a corresponding hierarchical task system, the one-sidedness of single-indicator evaluation is broken. Differentiated accuracy thresholds and alternative strategies are set for tasks of varying complexity, avoiding a one-size-fits-all evaluation standard and making the evaluation results more aligned with the business characteristics and risk levels of different tasks. By refining the specific evaluation indicators and alternative thresholds for the four dimensions of capability, cost, risk, and data volume, the top-level evaluation framework is implemented as a quantifiable and verifiable indicator system, shifting AI-based evaluation from qualitative experience-based judgment to precise quantitative calculation, significantly improving the objectivity and accuracy of the evaluation. Furthermore, a comprehensive judgment matrix is ​​used to achieve scientific weighted integration of multi-dimensional scores, taking into account factors such as execution effect, operating cost, and operational risk, ensuring the comprehensiveness and rationality of the overall judgment. Finally, a hard constraint of a veto list is added, safeguarding the bottom line of safety and compliance on the basis of conventional evaluation, and avoiding the security risks and compliance risks caused by blindly replacing AI in high-risk scenarios.

[0124] Preferably, in an embodiment, the step of counting the number of AI execution nodes based on the AI-capable state corresponding to each atomic task node, and obtaining the AI ​​achievement rate of the target digital employee based on the number of AI execution nodes, includes: Obtain the total number of all task nodes, including AI execution nodes, fixed program execution nodes, and human nodes; Based on the AI-capable state of each atomic task node, task rule matching verification, executor matching verification, and substitutability evaluation verification are performed on each atomic task node to obtain three verification results. Nodes that pass all three verification results are identified as AI execution nodes, and the number of AI execution nodes is determined. The AI ​​implementation rate of the target digital employee is obtained by calculating the ratio of the number of AI execution nodes to the total number.

[0125] Fixed program execution nodes refer to task nodes that are automatically executed through pre-written fixed programs, scripts, or process logic. The execution process requires no intelligent reasoning or human decision-making, and runs stably according to preset logic, adapting only to business scenarios where the rules are completely fixed and unchanging. In this embodiment, fixed program execution nodes are one of the constituent categories of all task nodes. They refer to atomic task nodes that run automatically through fixed program logic and do not involve artificial intelligence reasoning or decision-making. These nodes have fixed execution logic, zero operational decision-making, and high operational stability, but lack scenario adaptation and reasoning capabilities. They belong to automated execution nodes but are not included in the statistical scope of artificial intelligence execution nodes.

[0126] Among them, manual nodes refer to task nodes that must be manually executed by human business personnel. These typically correspond to high-risk decision-making, emotional interactions, compliance requirements necessitating human involvement, and business scenarios where automation is not yet feasible. They are a crucial component of ensuring security and compliance in business processes. In this embodiment, manual nodes are one category of all task nodes, referring to atomic task nodes that, after evaluation, are deemed not to meet the conditions for artificial intelligence and must be manually executed by business personnel. Reasons for this include execution results not meeting thresholds, belonging to veto scenarios, and the lack of suitable executors. These nodes are both the current accounting target for labor costs and a potential optimization direction for future AI-driven transformation.

[0127] Task rule matching verification is a preliminary verification step to confirm whether a task node has corresponding standardized execution rules. It confirms that the task's execution logic is supported by clear and reusable rules, serving as a fundamental verification to ensure standardized task execution. In this embodiment, task rule matching verification is the first verification step in identifying AI execution nodes. It verifies whether an atomic task node can be matched with a corresponding standardized execution rule in the task rule vector library, ensuring that the node has a clear and implementable execution logic basis. Nodes that fail rule matching do not have the foundation for standardized AI execution and cannot be included in the statistics of AI execution nodes.

[0128] Among them, executor matching verification is a connection verification step that verifies whether a task node has a suitable execution carrier to carry its execution logic. It is a key verification step that transforms abstract rules into actual execution capabilities. In this embodiment, executor matching verification is the second verification step in identifying artificial intelligence execution nodes. It is used to verify whether an atomic task node has a compatible target task executor that can carry the automated intelligent execution of the corresponding rules. Nodes without a compatible executor cannot actually implement AI operation and cannot be counted as artificial intelligence execution nodes.

[0129] The substitutability assessment and verification is a core verification step that verifies whether a task meets the AI ​​substitution qualification standards from multiple dimensions, including execution effectiveness, cost, and risk. It ensures that the business quality and safety meet requirements after AI substitution. In this embodiment, the substitutability assessment and verification is the third verification step in identifying AI execution nodes. Based on human-machine consistency comparison results and a multi-dimensional evaluation system, it verifies whether the node meets the corresponding level of AI-based evaluation standards. Verification is considered successful only if all dimensions, including execution effectiveness, cost-effectiveness, and risk control, meet the standards, thus safeguarding the quality baseline of AI substitution from a business effectiveness perspective.

[0130] For example, the specific method for obtaining the total number of all task nodes, including AI execution nodes, fixed program execution nodes, and human nodes, can be as follows: First, a full review and classification of all atomic task nodes corresponding to the target number of employees is conducted, clearly distinguishing between three types of nodes: AI execution nodes, fixed program execution nodes, and manual nodes. Each node is checked to avoid duplicate or omissions in the statistics. Finally, the total number of all nodes is obtained as the denominator base for calculating the AI ​​achievement rate, ensuring that the total number of nodes covers all work processes of the target number of employees and ensuring that the base for the indicator calculation is complete and accurate.

[0131] For example, the specific method for obtaining the number of AI execution nodes can be: Based on the initial AI-capable status of each node, three checks are carried out in sequence: first, it is checked whether the node can be matched with the corresponding standardized execution rule in the task rule vector library; second, it is checked whether there is a suitable target task executor that can carry the intelligent execution of the node; and finally, it is checked whether the node meets the multi-dimensional substitutability evaluation criteria. Only nodes that pass all three checks are finally identified as AI execution nodes. The total number of such nodes is counted as the numerator for calculating the AI ​​implementation rate. By using three checks to control the identification criteria of AI execution nodes, the distortion of indicators caused by the wrong inclusion of nodes is avoided.

[0132] For example, the specific method for calculating the AI ​​achievement rate of the target digital employee by comparing the number of AI execution nodes to the total number can be as follows: The number of AI execution nodes obtained from statistics is used as the numerator, and the total number of all task nodes is used as the denominator. The ratio is calculated and converted into a percentage to obtain the AI ​​implementation rate value corresponding to the target digital employee. This value intuitively quantifies the degree of AI coverage of the target digital employee. The calculation logic is clear and interpretable. It can be directly used to compare the intelligence level of different digital employees horizontally, and can also be used to track the AI ​​transformation progress of the same digital employee vertically.

[0133] In this embodiment, by clearly defining the total nodes as including three categories—AI execution nodes, fixed program execution nodes, and human nodes—the denominator coverage for the achievement rate calculation is complete and comprehensive, avoiding denominator bias caused by omitting fixed program nodes. This makes the indicators more aligned with the complete operational system of digital employees, and the statistical caliber more scientific and reasonable. AI execution nodes are screened layer by layer through a triple verification process of task rule matching, executor matching, and substitutability assessment. The identification standards for AI nodes are strictly controlled from three dimensions: rule support, implementation carrier, and business effect. This effectively avoids node misjudgment caused by single-state determination, ensuring that all nodes included in the numerator possess AI execution capabilities that are implementable, stable, and achieve the required results, making the achievement rate data more authentic and reliable. Finally, a quantitative achievement rate indicator is output through the ratio of the number of nodes. The calculation logic is clear, intuitive, and highly interpretable. It can accurately quantify the overall AI achievement level of digital employees and quickly update the values ​​according to changes in node status. This provides accurate and reliable quantitative decision-making basis for cost allocation, development resource allocation, and human-machine collaboration strategy formulation for digital employees, effectively supporting the refined operation optimization of digital employees.

[0134] To better implement the above methods, this application also provides an artificial intelligence implementation evaluation device for digital employees. This device can be integrated into electronic devices, such as servers or terminals. The terminal may include tablet computers, laptops, and / or personal computers.

[0135] For example, see Figure 2 , Figure 2 This is a schematic diagram of the structure of the artificial intelligence implementation evaluation device for digital employees provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the AI ​​implementation evaluation device for the digital employee may include a work content data acquisition module 201, a task node set acquisition module 202, a vector library acquisition module 203, a consistency comparison module 204, and an achievement rate acquisition module 205, as follows: The work content data acquisition module 201 is used to acquire the work content data corresponding to the target number of employees. The task node set acquisition module 202 is used to decompose the work content data into the smallest execution unit to obtain an atomic task node sequence, and to vectorize each atomic task node in the atomic task node sequence to obtain a task node set composed of multiple atomic task nodes carrying task vectors. The vector library acquisition module 203 is used to collect the task flow data corresponding to each atomic task node in the task node set, extract the execution rules of each atomic task node from the task flow data and generate rule vectors, and cluster and organize all rule vectors to obtain a task rule vector library. The consistency comparison module 204 is used to match the corresponding target task executor for each atomic task node based on the task rule vector library, obtain the artificial intelligence execution result of each atomic task node through the target task executor, obtain the manual execution result of each atomic task node, and perform a consistency comparison between the artificial intelligence execution result and the manual execution result. Based on the consistency comparison result, the artificial intelligence-capable state corresponding to each atomic task node is determined. The achievement rate acquisition module 205 is used to count the number of artificial intelligence execution nodes according to the artificial intelligence scalability status corresponding to each atomic task node, and obtain the artificial intelligence achievement rate of the target digital employee based on the number of artificial intelligence execution nodes.

[0136] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0137] As can be seen from the above, this application embodiment obtains the work content data corresponding to the target digital employee, breaks it down into the smallest execution unit to form a sequence of atomic task nodes, and then performs vectorization processing on each atomic task node to obtain a set of task nodes carrying task vectors. This transforms the originally vague and highly holistic business work into standardized smallest execution units that can be independently measured and accurately labeled, providing a statistically granular carrier for the quantitative evaluation of the degree of artificial intelligence implementation. This fundamentally solves the problems of vague work boundaries of digital employees and the inability to decompose and measure the degree of AI implementation in the traditional model. By collecting the task flow data of each atomic task node in the task node set, execution rules are extracted and rule vectors are generated. These are then clustered and organized to form a task rule vector library, transforming the scattered and unstructured task execution logic into a structured, reusable, and uniformly matchable standardized rule system. This provides a clear rule benchmark for the matching of subsequent task executors and also ensures... The consistency and objectivity of the full-node evaluation standards avoid the subjective bias caused by manual qualitative judgment, laying a solid rule foundation for quantitative evaluation. Based on the task rule vector library, each atomic task node is matched with a corresponding target task executor. The AI ​​execution results are obtained and compared with the manual execution results. The AI-capable state of each atomic task node is determined based on the comparison results. The objective execution result data comparison is used as the basis for judging whether a node is AI-capable. This ensures that the AI ​​attributes of each atomic task node have verifiable and reproducible data support, achieving accurate quantitative judgment of the AI ​​status of a single node. The number of AI execution nodes is counted based on the AI-capable state of each atomic task node, and the AI ​​implementation rate of the target digital employee is calculated. The abstract AI implementation level is transformed into an intuitive numerical indicator that can be compared horizontally and vertically, completing the full-link quantitative evaluation from the granularity of a single node to the overall indicator of digital employees. Meanwhile, since the entire evaluation process can be repeatedly triggered as work content is updated, task flow rules are iterated, executor capabilities are upgraded, or online operation data fluctuates, it can update the AI-capable status of each node in real time and refresh the AI ​​implementation rate simultaneously. Therefore, it can achieve dynamic measurement of the degree of AI implementation, rather than a one-time static evaluation. The quantified overall AI implementation rate index, coupled with the AI ​​status data refined to each atomic task node, can provide the operations side with accurate data decision-making basis, supporting operations personnel to adjust the configuration of human positions, allocate development and optimization resources, and formulate differentiated human-machine collaboration strategies, ultimately achieving refined operation optimization of digital employees.

[0138] In some embodiments, the AI ​​implementation evaluation device for digital employees of the present invention can be implemented in a combination of hardware and software. As an example, the AI ​​implementation evaluation device for digital employees of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the AI ​​implementation evaluation method for digital employees of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0139] The modules described in the embodiments of this invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0140] An electronic device according to an embodiment of the present invention includes a processor and a memory, wherein the memory stores multiple instructions; the processor loads instructions from the memory to execute the steps in the artificial intelligence implementation evaluation method for digital employees as described above. That is, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the artificial intelligence implementation evaluation method for digital employees as described in any embodiment of the present invention by calling the computer program.

[0141] In one alternative embodiment, an electronic device is provided, such as Figure 3 As shown, Figure 3 The illustrated electronic device 3000 includes a processor 3001 and a memory 3003. The processor 3001 and the memory 3003 are connected, for example, via a bus 3002. Optionally, the electronic device 3000 may further include a transceiver 3004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 3004 is not limited to one type, and the structure of the electronic device 3000 does not constitute a limitation on the embodiments of the present invention.

[0142] Processor 3001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 3001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0143] Bus 3002 may include a path for transmitting information between the aforementioned components. Bus 3002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 3002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus 3002 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0144] The memory 3003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0145] The memory 3003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 3001. The processor 3001 executes the application code stored in the memory 3003 to implement the content shown in the foregoing method embodiments.

[0146] Among them, electronic devices can also be terminal devices, which can be any device that can install applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0147] It should be noted that, Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0148] An embodiment of the present invention provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to perform the steps in the above-described method for evaluating the artificial intelligence implementation of digital employees.

[0149] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0150] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the steps in the AI ​​implementation evaluation method for digital employees described above.

[0151] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0152] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the implementable architecture, functions, and operations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which includes one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0153] The computer-readable storage medium provided in this invention can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EEPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium that includes or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0154] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0155] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0156] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0157] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product in one or more computer-readable media, the computer-readable media including computer-readable program code.

[0158] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for evaluating the artificial intelligence implementation of digital employees, characterized in that, include: Obtain the job content data corresponding to the target number of employees; The work content data is decomposed into the smallest execution unit to obtain an atomic task node sequence. Each atomic task node in the atomic task node sequence is vectorized to obtain a task node set consisting of multiple atomic task nodes carrying task vectors. Collect task flow data corresponding to each atomic task node in the task node set, extract the execution rules of each atomic task node from the task flow data and generate rule vectors, and cluster and organize all rule vectors to obtain a task rule vector library. Based on the task rule vector library, a corresponding target task executor is matched for each atomic task node. The artificial intelligence execution result of each atomic task node is obtained through the target task executor. The manual execution result of each atomic task node is obtained, and the consistency comparison between the artificial intelligence execution result and the manual execution result is performed. Based on the consistency comparison result, the artificial intelligence-capable state corresponding to each atomic task node is determined. Based on the AI-capable state corresponding to each atomic task node, the number of AI execution nodes is counted, and based on the number of AI execution nodes, the AI ​​achievement rate of the target digital employee is obtained.

2. The method of claim 1, wherein, The process involves decomposing the work content data into the smallest execution unit to obtain a sequence of atomic task nodes. Each atomic task node in this sequence is then vectorized to obtain a set of task nodes consisting of multiple atomic task nodes carrying task vectors, including: Using a large language model, semantic parsing is performed on the natural language data in the work content data to obtain semantic parsing results; The semantic parsing result is decomposed into the smallest execution unit to obtain the atomic task node sequence marked with triples, wherein the triples include tools, actions, and target objects; Basic attributes are extracted from each atomic task node in the atomic task node sequence to generate a work description card corresponding to each atomic task node. Based on the attribute information in the job description card, multi-dimensional feature mapping is performed on each atomic task node so that each atomic task node carries an eight-dimensional task vector, resulting in the task node set. The eight-dimensional task vector includes task content description, source system, target system, task target description, task execution plan, task closed-loop receipt, task process, and task dynamic knowledge graph.

3. The method of claim 1, wherein, The process involves collecting task flow data corresponding to each atomic task node in the task node set, extracting execution rules for each atomic task node from the task flow data and generating rule vectors, and clustering all rule vectors to obtain a task rule vector library, including: For each atomic task node in the task node set, the data flow path, interaction information and historical closed-loop samples in the corresponding task flow are collected. Based on the data flow path, the interaction information and the historical closed-loop samples, the task flow dataset corresponding to each atomic task node is obtained. The execution logic of each atomic task node is parsed from the task flow dataset, and execution rules including tool, action, and target mapping relationship are extracted. Based on the execution rules, a serialized rule vector is generated. The rule vectors are indexed and clustered to integrate and classify rule vectors with the same execution logic, thus obtaining the task rule vector library.

4. The method of claim 1, wherein, Based on the task rule vector library, a corresponding target task executor is matched for each atomic task node. The AI ​​execution result of each atomic task node is obtained through the target task executor. The manual execution result of each atomic task node is also obtained. A consistency comparison is performed between the AI ​​execution result and the manual execution result. Based on the consistency comparison result, the AI-capable state of each atomic task node is determined, including: Based on the execution rules and task characteristics of each atomic task node in the task rule vector library, the atomic task nodes are divided into execution types to obtain task node groups corresponding to different execution types. For each of the task nodes grouped, a target task executor of the corresponding type is matched from a preset executor set; Obtain a task sample, execute the task sample through the target task executor to obtain the artificial intelligence execution result, and obtain the human execution result corresponding to the task sample; The consistency comparison between the artificial intelligence execution result and the manual execution result is performed, and the artificial intelligence-capable state corresponding to each atomic task node is determined based on the consistency comparison result and the preset consistency threshold.

5. The method of claim 1, wherein, The consistency comparison between the AI ​​execution result and the manual execution result includes: The output data format of each node in the data stream is unified, and a dual-table storage scheme is used to store the manual execution results and the artificial intelligence execution results respectively, resulting in a dual-track output dataset. The dual-track output dataset is subjected to output compliance detection, and data that does not conform to the preset output format specifications is removed to obtain a valid output dataset; The target AI execution results and target human execution results in the effective output dataset are compared item by item to obtain the consistency comparison results.

6. The method of claim 1, wherein, The process of determining the AI-capable state of each atomic task node based on the consistency comparison results includes: A four-dimensional evaluation framework including reliability, adaptability, cost-effectiveness, and boundary handling is constructed. Atomic task nodes are hierarchically divided according to task characteristics to obtain multiple task levels. Corresponding accuracy thresholds and alternative strategies are set for different task levels. Based on the four-dimensional evaluation framework, multiple task levels, accuracy thresholds, and alternative strategies, a hierarchical evaluation standard system is formed. Based on the aforementioned hierarchical evaluation standard system, from four dimensions—capability, cost, risk, and data volume—corresponding evaluation indicators and alternative thresholds are configured for atomic task nodes at each task level, resulting in a multi-dimensional evaluation indicator system. The consistency comparison results and the execution data of the corresponding atomic task nodes are input into the multi-dimensional evaluation index system to obtain the scoring results of each dimension. The scores of each dimension are integrated and calculated through the comprehensive judgment matrix to obtain the initial artificial intelligence-capable state corresponding to each atomic task node. Based on a preset veto list, the initial AI-capable state is adjusted to obtain the target AI-capable state.

7. The method of claim 1, wherein, The step of counting the number of AI execution nodes based on the AI-capable state corresponding to each atomic task node, and obtaining the AI ​​achievement rate of the target digital employee based on the number of AI execution nodes, includes: Obtain the total number of all task nodes, including AI execution nodes, fixed program execution nodes, and human nodes; Based on the AI-capable state of each atomic task node, task rule matching verification, executor matching verification, and substitutability evaluation verification are performed on each atomic task node to obtain three verification results. Nodes that pass all three verification results are identified as AI execution nodes, and the number of AI execution nodes is determined. The AI ​​implementation rate of the target digital employee is obtained by calculating the ratio of the number of AI execution nodes to the total number.

8. A digital employee artificial intelligence implementation evaluation device, characterized by, include: The job content data acquisition module is used to acquire the job content data corresponding to the target number of employees; The task node set acquisition module is used to decompose the work content data into the smallest execution unit to obtain an atomic task node sequence, and to vectorize each atomic task node in the atomic task node sequence to obtain a task node set consisting of multiple atomic task nodes carrying task vectors. The vector library acquisition module is used to collect task flow data corresponding to each atomic task node in the task node set, extract the execution rules of each atomic task node from the task flow data and generate rule vectors, and cluster and organize all rule vectors to obtain a task rule vector library. The consistency comparison module is used to match the corresponding target task executor for each atomic task node based on the task rule vector library, obtain the artificial intelligence execution result of each atomic task node through the target task executor, obtain the manual execution result of each atomic task node, and perform a consistency comparison between the artificial intelligence execution result and the manual execution result. Based on the consistency comparison result, the AI-capable state of each atomic task node is determined. The achievement rate acquisition module is used to count the number of artificial intelligence execution nodes according to the artificial intelligence scalability status corresponding to each atomic task node, and obtain the artificial intelligence achievement rate of the target digital employee based on the number of artificial intelligence execution nodes.

9. An electronic device, comprising: It includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform the steps in the artificial intelligence implementation evaluation method for digital employees as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps in the evaluation method for the artificial intelligence implementation of digital employees according to any one of claims 1 to 7.