Data braiding system implementation method and device based on dialogue and multi-agent collaboration

Abstract

The invention provides a dialogue and multi-agent collaboration-based data knitting system implementation method, which comprises the following steps that: a main agent performs user intention recognition, splits and distributes a data processing task according to the user intention, and issues a data retrieval task to a data retrieval agent in a domain agent; the data retrieval agent executes a data retrieval task on the managed data domain to obtain a retrieval result, the main agent issues a data calculation task to the data calculation agent in the domain agent, and the data calculation agent executes the data calculation task to obtain a result data set; and the main agent executes a data fusion task on the retrieval result and / or the result data set, and returns the fused data set to the user. According to the invention, through the precise division and efficient cooperation mechanism of the main agent and the domain agent in the multi-agent system, the implementation process of the data knitting system is simplified, the system is easy to expand, and the high efficiency and intelligence of the enterprise-level data knitting process are realized.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of network technology, and more particularly, to a method and device for implementing a data weaving system based on dialogue and multi-agent collaboration. Background Art

[0002] With the continuous improvement of the informatization level of various industries, enterprises and organizations have accumulated various types of heterogeneous data, such as structured and unstructured data, which have become important assets for enterprises and organizations. The key to managing and utilizing this data is to break the data silos within enterprises and organizations and promote the rapid integration and interoperability of data. Traditional data management methods have problems such as low processing efficiency and insufficient security when dealing with large-scale, multi-source heterogeneous data. The centralized data supply model also encounters problems such as massive resource requirements, real-time computing bottlenecks, and poor adaptability to multiple data application scenarios.

[0003] In order to solve these challenges, data weaving has emerged as a new cross-platform data integration and management concept, which focuses on data connection, integration and unified view, aiming to improve the seamless flow and integration of data. However, the implementation process of the current data weaving system is still relatively complicated, requiring the construction of a global knowledge graph and data federation, and the expansion and maintenance are relatively cumbersome. The automation and intelligence capabilities are insufficient, which limits its effectiveness in practical applications.

[0004] Application Contents

[0005] This application describes a method and device for implementing a data weaving system based on dialogue and multi-agent collaboration, which can solve the above-mentioned technical problems.

[0006] According to a first aspect, a method for implementing a data weaving system based on dialogue and multi-agent collaboration is provided, wherein the data weaving system includes a main agent and at least one domain agent, wherein a domain agent is a collection of sub-agents that manage data assets of a data domain, wherein a domain agent includes a data retrieval agent and a data calculation agent;

[0007] The method comprises:

[0008] The main agent performs user intention recognition, and disassembles and distributes data processing tasks according to the user intention, wherein the disassembled data processing tasks include data retrieval tasks, data calculation tasks and data fusion tasks;

[0009] The master agent issues a data retrieval task to the data retrieval agent in the domain agent, wherein the data retrieval task is used to obtain storage information of the field to be retrieved in the data domain;

[0010] The data retrieval agent performs a data retrieval task on the managed data domain, obtains a retrieval result, the retrieval result includes the table structure where the field to be retrieved is located and the storage information in the table structure, and returns the retrieval result to the main agent;

[0011] The master agent issues a data computing task to a data computing agent in a domain agent, wherein the data computing task is used to perform calculations based on the search results and / or one or more fields in the data domain;

[0012] The data computing agent executes the data computing task to obtain a result data set, wherein the result data set includes a two-dimensional table structure calculated based on the search result or the fields in the data domain, and returns the result data set to the main agent;

[0013] The master agent performs a data fusion task on the retrieval results and / or the result data set, and returns the fused data set to the user.

[0014] According to a second aspect, a data weaving system based on dialogue and multi-agent collaboration is provided, wherein the data weaving system includes a main agent and at least one domain agent, wherein a domain agent is a collection of sub-agents that manage a data domain;

[0015] The main agent is used to identify the user's intention and to disassemble and allocate the data processing tasks according to the user's intention, wherein the disassembled data processing tasks include data retrieval tasks, data calculation tasks and data fusion tasks;

[0016] The master agent is further used to issue a data retrieval task to the data retrieval agent in the domain agent, wherein the data retrieval task is used to obtain storage information of the field to be retrieved in the data domain;

[0017] The data retrieval agent is used to perform data retrieval tasks on the managed data domain, obtain retrieval results, including the table structure where the field to be retrieved is located and the storage information in the table structure, and return the retrieval results to the main agent;

[0018] The master agent is used to issue data computing tasks to the data computing agent in the domain agent, wherein the data computing tasks are used to perform calculations based on the search results and / or one or more fields in the data domain;

[0019] The data computing agent is used to execute the data computing task, obtain a result data set, wherein the result data set includes a two-dimensional table structure calculated based on the search results or the fields in the data domain, and return the result data set to the main agent;

[0020] The main agent is used to perform a data fusion task on the retrieval results and / or the result data set, and return the fused data set to the user.

[0021] According to a third aspect, a data weaving system implementation device based on dialogue and multi-agent collaboration is provided, wherein the data weaving system includes a main agent and at least one domain agent, wherein a domain agent is a collection of sub-agents that manage data assets of a data domain, wherein a domain agent includes a data retrieval agent and a data calculation agent;

[0022] The first processing device is used for the main agent to identify the user's intention, and to disassemble and allocate the data processing tasks according to the user's intention, wherein the disassembled data processing tasks include data retrieval tasks, data calculation tasks and data fusion tasks; the main agent sends the data retrieval task to the data retrieval agent in the domain agent, and the data retrieval task is used to obtain the storage information of the field to be retrieved in the data domain;

[0023] The second processing device is used for the data retrieval agent to perform a data retrieval task on the managed data domain, obtain a retrieval result, the retrieval result includes the table structure where the field to be retrieved is located and the storage information in the table structure, and return the retrieval result to the main agent;

[0024] A third processing device, configured for the master agent to issue a data computing task to a data computing agent in a domain agent, wherein the data computing task is used to perform calculation based on the search result and / or one or more fields in the data domain;

[0025] A fourth processing device is used for the data computing agent to execute the data computing task, obtain a result data set, the result data set including a two-dimensional table structure calculated based on the search result or the fields in the data domain, and return the result data set to the main agent;

[0026] The fifth processing device is used for the main agent to perform a data fusion task on the retrieval results and / or the result data set, and return the fused data set to the user.

[0027] In the above-mentioned system and method provided in the embodiments of this specification, by utilizing the precise division of labor and efficient collaboration mechanism between the main agent and the domain agent in the multi-agent system, the implementation process of the data weaving system is simplified, the data weaving system has good scalability, and the efficiency and intelligence of the enterprise-level data weaving process are realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 A system schematic diagram of a data weaving system based on dialogue and multi-agent collaboration provided by an embodiment of this specification is shown;

[0030] Figure 2 A schematic diagram of a main agent in a data weaving system based on dialogue and multi-agent collaboration provided by an embodiment of this specification is shown;

[0031] Figure 3 A schematic diagram of a data retrieval agent in a data weaving system based on dialogue and multi-agent collaboration provided by an embodiment of this specification is shown;

[0032] Figure 4 A schematic diagram of a data computing agent in a data weaving system based on dialogue and multi-agent collaboration provided by an embodiment of this specification is shown;

[0033] Figure 5 A schematic diagram of a data governance agent in a data weaving system based on dialogue and multi-agent collaboration provided by an embodiment of this specification is shown;

[0034] Figure 6 A schematic diagram of the process of agent NL2SQL in a data weaving system based on dialogue and multi-agent collaboration provided by an embodiment of this specification is shown;

[0035] Figure 7 A flow chart of a data weaving system based on dialogue and multi-agent collaboration provided by an embodiment of this specification is shown;

[0036] Figure 8 A flow chart showing a method for implementing a data weaving system based on dialogue and multi-agent collaboration provided in an embodiment of this specification;

[0037] Fig. 9 A schematic diagram of a device for implementing a data weaving system based on dialogue and multi-agent collaboration provided in an embodiment of this specification is shown. DETAILED DESCRIPTION

[0038] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0039] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0040] As a data management architecture, the Data Weaving System provides enterprises with a data management method to effectively manage and utilize their data assets, and realize the connection, integration and unified view of enterprise data. The Data Weaving System is committed to providing consistent interfaces and services between different data sources, so that data can flow seamlessly between different systems; using artificial intelligence and machine learning technology to automatically discover data relationships, perform data governance, and optimize data access paths; ensuring data security and privacy protection while meeting various regulatory compliance requirements; and being able to easily expand and adjust as data volume grows or business needs change.

[0041] In order to achieve the above goals, the construction of the data weaving system relies on a variety of underlying technologies and tools. Since it involves a variety of existing technologies and new development methods, the technical implementation is difficult. The current implementation of the data weaving system requires the construction of a global knowledge graph. The construction of a global knowledge graph requires the organization and standardization of data from different sources, and the accuracy and consistency of these data must be ensured. It is also necessary to build data federation to achieve cross-domain data sharing and analysis without moving data, which puts higher requirements on data access control and privacy protection. With the increase in data volume or changes in business needs, expansion and maintenance are cumbersome, and the lack of automation and intelligence capabilities limits its effectiveness in practical applications.

[0042] Depend on Figure 1 The system schematic diagram of the data weaving system based on dialogue and multi-agent collaboration shows that the present invention realizes the data weaving system through a master agent and multiple domain agents. The data weaving system includes a master agent and a domain agent. Each domain agent autonomously manages a data domain or data center, and the master agent and the domain agent adopt a master-slave collaboration mode to achieve the fusion of target data. The master agent receives user requests in a dialogue mode, performs intent recognition, task decomposition and allocation, result fusion and status monitoring, etc., to ensure that the system collaborates and runs efficiently as planned. The domain agent is a collection of agents composed of multiple sub-agents, which receives task instructions assigned by the master agent and feeds back the results. The domain agent includes a data retrieval agent, a data computing agent, and a data governance agent.

[0043] Depend on Figure 2 The schematic diagram of the main agent in the data weaving system based on dialogue and multi-agent collaboration shows that the main agent receives user requests through natural language dialogue mode, uses large models and RAG retrieval enhanced generation technology to perform intent recognition, task decomposition and allocation, data fusion and status monitoring, etc., and coordinates communication and collaboration between various agents, recovers from abnormal situations, and ensures efficient and stable operation of the entire system.

[0044] The main agent uses a large language model to perform multi-level semantic analysis on data requests in natural language form in the dialogue mode. Through context analysis and contextual reasoning, it identifies user intentions, such as the specific scope of data retrieval, required content details, and whether data fusion calculations are involved.

[0045] The main agent determines the task to be performed based on the identified user intention and assigns the task to the corresponding agent according to the task type. For example, if the task type is data retrieval, the task is assigned to the data directory search agent; if the task type is data calculation, the task is assigned to the data calculation agent. At the same time, combined with the asset directory and data permissions maintained by each data domain, the security of task assignment and data access is ensured. The main agent needs to verify the permissions requested by the user and control data access and operations based on the permissions.

[0046] The master agent coordinates the communication between agents in each domain, manages the communication protocols between different agents, and ensures the security and reliability of data transmission. The communication between agents in each domain includes message passing, data sharing, and task collaboration, so as to realize information exchange and collaborative work between agents in each domain.

[0047] The main agent integrates a fusion data warehouse to cache the result data processed by each domain agent. According to the user's intention and the result data received from the domain agent, it determines the data fusion algorithm, executes the fusion calculation task, and returns the fused result data to the user.

[0048] The main agent monitors the system status in real time and detects failures or abnormal behaviors of domain agents. Once a failure is detected, it reallocates tasks or starts backup agents to ensure the stable operation of the system. Through a variety of fault-tolerant strategies, such as task retry, task transfer, and backup agent startup, resource allocation is optimized to cope with different types of failure situations and enhance system robustness.

[0049] The master agent receives the task execution results of the domain agent, which include various information such as task completion status, data processing results, and external interface calls. The master agent evaluates the task execution results to see whether the goal is achieved as expected. If the evaluation results show that the task does not meet expectations, the master agent activates the reflection mechanism to analyze potential problems in the execution process. The master agent adjusts strategies or optimizes tool usage accordingly to improve the efficiency and accuracy of subsequent task execution. Specifically, the reflection mechanism is a method of self-evaluation and self-improvement, which is used to analyze potential problems in the execution process and optimize future actions. It has a wide range of applications in fields such as artificial intelligence, robotics, and software engineering, including the process of collecting data, analyzing data, evaluating results, generating reports, formulating improvement plans, implementing improvements, and iterating cycles.

[0050] A domain agent is a collection of sub-agents built for a certain data domain. Each sub-agent in the sub-agent collection receives task instructions from the main agent. There are usually multiple data domains or data centers within an enterprise, such as production data domain, sales data domain, financial data domain, and human resources data domain. A corresponding domain agent is built for each data domain or data center. When it comes to cross-domain business scenarios, the relevant domain agents and the main agent work together in a master-slave distributed mode. The main agent clarifies the collaboration strategy between the domain agents based on the user's intention, disassembles and assigns orchestration tasks to each domain agent, and orchestrates or integrates the feedback results of each domain agent and outputs them to the user. Each domain agent receives and executes the assigned tasks, and feeds back the execution results of the domain agent to the main agent.

[0051] When adding a new data domain, you only need to add the corresponding domain agent and incorporate it into the main agent for collaboration, making it easy to expand and adjust.

[0052] like Figure 3 The data retrieval agent in the data weaving system based on dialogue and multi-agent collaboration shows that the data retrieval agent receives the query intent sent by the main agent, and after translating the intent in the managed data domain, it searches and retrieves multiple data sources in the data domain and returns the results to the main agent. The data retrieval agent builds the table structure and view structure based on the characteristics and capability constraints of the big model. Specifically, the core table, associated table and transaction table in the table structure are designed considering the computing resource requirements, data processing speed and data consistency of the big model. The view structure is used to simplify complex query logic and provide data views that are easy to understand and use. Simplify the table structure and view structure, specifically including removing redundant fields, using enumeration types for fields with limited options, uniformly using standard data types, and reducing the association levels between tables. Use the simplified table structure and view structure to build the big model prompt words.

[0053] Large model prompt words include:

[0054] 1. Table / view description;

[0055] 2. Field descriptions and attributes related to the user's question, including data type, granularity, sorting direction, and unit;

[0056] 3. Table / view SQL statement specifications, such as filtering conditions for time or dimension information.

[0057] Sort out the business semantic knowledge system and build a business data semantic knowledge framework, where the business semantic system includes the above-mentioned large model prompt words.

[0058] The business data semantic knowledge framework is uniformly vectorized to obtain a data semantic layer knowledge base.

[0059] Based on user intent and the data semantic layer knowledge base, vector retrieval and recall operations are performed to extract relevant information from the data semantic layer knowledge base, and the information is optimized and sorted using re-ranking technology to achieve fast and accurate search and retrieval in multiple data sources.

[0060] It should be noted that due to the capacity constraints of large models in multi-table association, in order to improve the accuracy of NL2SQL, a large model prompt word project is built. By using simplified table and view structures, complex association problems can be turned into a problem that only needs to focus on user questions and table field extraction. The prompt word content includes: table / view description; field description and attributes (data type, granularity, sorting direction, unit, etc.) related to user questions; table / view SQL statement specifications, such as filtering conditions for time and dimension information.

[0061] like Figure 6 As shown, the algorithm flow of NL2SQL for a specific large model is as follows:

[0062] A. Users enter query requests using natural language.

[0063] B. Use advanced semantic understanding models (such as pre-trained language models) to parse user input and accurately identify user intent.

[0064] C. For compound questions containing multiple query elements, they should be split into several single query intents to achieve more accurate understanding and processing.

[0065] For situations where the intent depends on the context to fully understand, coreference resolution or lexical replacement operations are required to ensure that the intent can be accurately parsed.

[0066] Perform standardization of synonyms and terminology to ensure consistency and accuracy within the context of your specific business domain.

[0067] Select the most appropriate semantic library based on context relevance.

[0068] D. Based on the business type, extract the core indicators and keywords to be queried, and load the semantic layer definition of the indicators or fields.

[0069] Through the SQL template matching mechanism, vector retrieval is used to identify the "Few-shot" instances that are closest to the existing sample SQL structure.

[0070] If a specific complex business scenario is encountered, query guidance suitable for the scenario will be retrieved from the predefined prompt word template set.

[0071] E. With the help of the code generation model, generate accurate SQL query statements based on the previous intent parsing and template matching results.

[0072] F. Ensure that the generated SQL code is valid and executable. Generate the final executable SQL query and present the query results through test cases or database-related verification.

[0073] like Figure 4 The data computing agent in the data weaving system based on dialogue and multi-agent collaboration shows that the data computing agent receives the computing task issued by the main agent. The task information will carry the user's intention, the required data asset identification, etc. The data computing agent will generate detailed information of the task according to the user's intention, such as: data asset identification, calculation dimensions (time dimension, geographical dimension, other dimensions, etc.), calculation indicators, etc.

[0074] According to the data asset identifier, data asset information is obtained from the domain's data management system, such as: data source type, data source connection method, order of magnitude of data assets, metadata information of data assets, etc.

[0075] Collect relevant data of the original data that meets the conditions from the data source, such as data volume, maximum / minimum value of indicators, number of dimensions, etc.

[0076] Combined with the order of magnitude, data situation, and type of data source, determine whether it is necessary to sink the calculation, as well as the size of the original data and the size of the result data. Depending on the situation, determine whether a distributed computing engine (such as Spark) is needed. In the case of data source types that cannot be sunk for data calculation, and the amount of original data or the amount of result data is relatively large, use a distributed computing engine for calculation. At this time, the computing intelligence will generate Spark's SQL through NL2SQL technology based on the user's intention and detailed information of the task, and schedule the Spark program for calculation.

[0077] If it is a data source type that can be sunk (such as MySQL, DAMO, ClickHouse), the computing intelligence will generate the standard SQL of the data source type through NL2SQL according to the user's intention and detailed information of the task, and sink the data calculation to the data source for calculation. If it is another NoSQL data source type (such as Redis), the corresponding syntax is generated through a special natural language conversion command model similar to NL2SQL (such as NL2Redis) to perform relevant data calculations.

[0078] The data agent obtains the data calculation results and determines whether to return the data results directly or output them to the fusion database and return relevant information to the main agent based on the size of the results. If the data volume is large, it is output to the fusion database; if the data volume is small, it is directly returned to the main agent.

[0079] The data computing agent integrates the general language model and the code language model, builds NL2SQL (Natural Language to SQL) technology, has the ability to automatically identify data sources, analyze data formats and structures, perform SQL splicing through the semantics of target tables and fields, and dynamically select and adjust data processing strategies according to needs. Through self-learning and optimization mechanisms, it can efficiently perform data retrieval, data processing, and data computing tasks in complex and changing data environments.

[0080] During the processing, different strategies are adopted for heterogeneous data. For structured data, SQL queries or DataFrame operations (such as Pandas, Spark DataFrame) are used for processing; for unstructured data, natural language processing (NLP) technologies are selected, such as text classification, sentiment analysis, and entity recognition; and for semi-structured data, appropriate parsing and conversion technologies are selected, such as JSON parsing or XML parsing.

[0081] In terms of ensuring the timeliness of processing, the processing strategy is dynamically adjusted according to the characteristics of the data. For large data volume scenarios, a distributed computing framework (such as Spark) is used for efficient processing; for real-time data streams, a stream processing framework (such as Flink) is selected for immediate processing. After processing is completed, the result data is aggregated and output.

[0082] like Figure 5 The data governance agent in the data weaving system based on dialogue and multi-agent collaboration shows that the data governance agent uses the task splitting and tool calling capabilities of the big model to monitor data quality and security, including using algorithms and rules to check whether there are missing values, duplicate values, format errors, and sensitive data in important data. Through data association analysis, it verifies whether the logical relationship between data is correct, and determines through security policies whether sensitive data has been desensitized or encrypted, and whether users have data access rights.

[0083] The data governance agent handles data quality and security issues, including format conversion or filling in missing values ​​for issues that can be automatically fixed, such as incorrect or missing data; automatically desensitizing or encrypting output sensitive data according to security policies when sensitive data is not desensitized or encrypted; providing feedback of insufficient permissions for data to which no access permissions are granted; and providing feedback of potential data problems for complex quality or security issues and notifying administrators for manual processing.

[0084] The data governance intelligent body conducts real-time monitoring and continuous improvement, including real-time monitoring of data input and output, detecting fluctuations in data volume, optimizing abnormal fluctuations through a combination of manual and intelligent means, and ensuring the stability and normality of data flow. It detects whether there is sensitive data in the data, whether it has been processed in accordance with security policies, and audits user access rights to ensure data security. The above process realizes comprehensive management and continuous maintenance of data quality and data security, enhancing data reliability, availability, and security.

[0085] like Figure 7 As shown, the dialog request data weaving process includes the following steps:

[0086] Step 1: The main agent receives the user request.

[0087] In step 2, the main agent splits the user request into one or more tasks through intent recognition and task decomposition, and determines the relevant tasks of each domain agent, including data retrieval tasks, data calculation tasks, and data governance tasks.

[0088] Step 3: The master agent issues the corresponding data retrieval task to the data retrieval agent in the domain agent to which the data retrieval task needs to be issued.

[0089] Step 4: The data retrieval agent performs data retrieval tasks on the managed data domain.

[0090] Step 5: If the retrieval result is obtained, the data retrieval agent returns the data retrieval result to the main agent.

[0091] Step 6: The master agent sends the corresponding data computing task to the data computing agent in the domain agent to which the data computing task needs to be sent.

[0092] Step 7: The data computing agent extracts data from the data domain to perform computing tasks.

[0093] Step 8: After the data computing agent completes the computing task, if there is no need to trigger the data governance task, it returns the data computing result to the main agent;

[0094] If a data governance task needs to be triggered, the data governance agent will be triggered to check the quality and security of the data, and perform corresponding desensitization, encryption or permission control on the data according to the existing quality rules and security policies.

[0095] Step 9: The data governance agent feeds back the processing results to the main agent.

[0096] Step 10: The master agent caches the data of the domain agents that need to be fused into the fusion data warehouse, receives the relevant data, performs fusion calculations, generates the final results and feeds them back to the user.

[0097] Step 11: The main intelligent agent desensitizes, encrypts or controls the permissions of the final result again according to the overall security strategy and then feeds it back to the user.

[0098] The present invention relies on the intelligent collaboration capability of the master agent, combined with the autonomous and intelligent management capabilities of the domain agent, and the parallel processing capabilities between domain agents, to achieve rapid retrieval of enterprise-wide data, efficient and intelligent data calculation and management, and significantly improve the overall efficiency of data weaving. The master agent is also responsible for monitoring the working status of each domain agent, dynamically adjusting tasks according to actual conditions, achieving load balancing, handling and recovering from faults, etc., to ensure stable and efficient operation of the system.

[0099] By allocating tasks from the master agent to each domain agent, distributed processing of tasks is achieved. Each domain agent can quickly feedback processing results based on its active and intelligent management of local data, effectively reducing the latency of data retrieval, improving the overall response speed and operating efficiency of the system, and enhancing the flexibility of the system.

[0100] Each data domain of an enterprise can manage its asset catalog and data permissions independently through its own domain intelligence, without the need to aggregate data assets within the enterprise into a centralized catalog or knowledge graph, or synchronize their respective security policies, thus reducing complexity. This approach also enables each data domain to independently update and maintain its data assets, and allows them to flexibly control and output data according to their own security policies, thereby ensuring the real-time, accuracy and security of the data.

[0101] The managers of each data domain of the enterprise have a deeper understanding of their data assets and permissions, and can allocate and manage permissions more accurately. By performing fine-grained permission control on data within the domain, it is ensured that only authorized users can access their data assets. This approach improves the accuracy and security of data permissions and avoids problems such as inaccurate or untimely permission allocation that may be caused by centralized management.

[0102] In the present invention, different types of sub-agents in the domain agent are specially designed to perform different tasks, using large models, RAG and other technologies to complete the retrieval, calculation and governance of data in the domain, such as the data retrieval agent is responsible for enhanced data retrieval, the data calculation agent is responsible for intelligent data extraction and processing, and the data governance agent is responsible for efficient data governance. This specialized solution method can improve the solution ability and work efficiency of the domain agent.

[0103] The data computing agent in the present invention integrates advanced large models and machine learning technologies, can automatically identify data source types and data formats, and then intelligently optimize data access paths, generate the most appropriate data processing strategies, and greatly improve data processing efficiency.

[0104] like Figure 8 It is shown that, according to an embodiment of the first aspect, a method for implementing a data weaving system based on dialogue and multi-agent collaboration is provided, wherein the data weaving system includes a main agent and at least one domain agent, a domain agent is a collection of sub-agents that manage a data domain, wherein a domain agent includes a data retrieval agent and a data computing agent.

[0105] The following steps are involved:

[0106] 110. The main intelligent agent identifies the user's intention and decomposes and allocates the data processing tasks according to the user's intention. The decomposed data processing tasks include data retrieval tasks, data calculation tasks and data fusion tasks.

[0107] 120. The main agent sends a data retrieval task to the data retrieval agent in the domain agent. The data retrieval task is used to obtain the storage information of the field to be retrieved in the data domain.

[0108] 130. The data retrieval agent performs data retrieval tasks on the data domain it manages and obtains retrieval results, which include the table structure where the field to be retrieved is located and the storage information in the table structure, and returns the retrieval results to the main agent.

[0109] 140. The master agent sends a data computing task to the data computing agent in the domain agent, where the data computing task is used to perform calculations based on the retrieval results and / or one or more fields in the data domain.

[0110] 150. The data computing agent executes the data computing task and obtains a result data set, which includes a two-dimensional table structure calculated based on the search results or fields in the data domain, and returns the result data set to the main agent.

[0111] 160. The main agent performs a data fusion task on the retrieval results and / or result data sets, and returns the fused data sets to the user.

[0112] In some embodiments, step 130 specifically includes:

[0113] 131. The data retrieval agent builds the table structure and view structure of the data domain based on the large language model, and simplifies the table structure and view structure;

[0114] 132. Use the simplified table structure and view structure to construct a large model prompt word, which is used to describe the table structure and view structure;

[0115] 133. Based on the big language model, sort out the business semantic knowledge system of the data domain, where the business semantic system is used to describe the business terminology information of the data domain, including the big model prompt words;

[0116] 134. Use the business semantic knowledge system to build a business data semantic knowledge framework. The business data semantic knowledge framework includes metadata of the data domain. The metadata includes the definition of the table structure and the definition of the field.

[0117] 135. Perform unified knowledge vectorization processing on the business data semantic knowledge framework to obtain a data semantic layer knowledge base of the data domain. The data semantic layer knowledge base is a repository for managing and organizing metadata of the data domain.

[0118] 136. Based on the field information to be retrieved, a vector search and recall operation is performed on the data semantic layer knowledge base to obtain the table structure where the field to be retrieved is located and the storage information in the table structure.

[0119] Step 150 specifically includes:

[0120] 151. The data computing agent generates computing task information according to the user intention in the data computing task, and the computing task information includes a data asset identifier;

[0121] 152. According to the data asset identifier in the computing task information, obtain data asset information from the data domain, the data asset information including the type of data source, the connection method of the data source and the data volume of the data asset;

[0122] 153. When the amount of data in the data source is less than the preset amount of data and the data involved in the calculation are all in the data source, and the conditions for sinking to the data source for calculation are met, a data query statement applicable to the data source type and / or a grammatical statement applicable to the data source type are generated according to the user's intention and the calculation task information, and the data calculation task, as well as the data query statement applicable to the data source type and / or the grammatical statement applicable to the data source type are sunk to the data source for calculation, and a result data set of the data source is obtained;

[0123] 154. When the amount of data in the data source is greater than the preset amount of data or other data in the domain is required to participate in the calculation, and the conditions for sinking to the data source for calculation are not met, it is necessary to rely on the high-performance computing engine of the data computing agent to perform data calculations. According to the user's intention and computing task information, a data query statement and / or a grammatical statement suitable for the computing engine is generated, and the data that meets the conditions is extracted to the computing engine. The computing engine executes the data computing task and obtains the result data set;

[0124] 155. Return the result data set to the main agent.

[0125] In some embodiments, the domain agent also includes a data governance agent, and the data processing task also includes a data governance task;

[0126] Also includes:

[0127] The master agent issues data governance tasks to the data governance agent in the domain agent. The data governance tasks are used to manage and maintain the quality, security and permissions of the data in the data domain or result data set.

[0128] The data governance agent performs data governance tasks on the managed data domain and / or result data set, obtains the governed result data set, and returns the governed result data set to the main agent.

[0129] In some embodiments, it also includes:

[0130] The data governance agent determines the data to be governed according to the data governance task. The data to be governed includes the data in the data domain or the result data set returned by the data computing agent.

[0131] Check whether there are data quality issues in the data to be governed, and automatically repair the data quality issues that can be automatically repaired, including missing values, duplicate values ​​and / or format errors. For data quality issues that cannot be automatically repaired, feedback indicates that the data has hidden problems;

[0132] If sensitive data in the data to be governed is not desensitized or encrypted, desensitize or encrypt the sensitive data according to the security policy. For sensitive data that does not have access rights, feedback will be given indicating insufficient permissions.

[0133] Monitor the input and output of data in the data domain in real time, detect abnormal fluctuations in the amount of data in the data domain, and feedback abnormal fluctuations.

[0134] In some embodiments, it also includes:

[0135] The main agent communicates with the user through natural language to obtain the user's intention, which includes the user's query requirements. The query requirements include the query time range, time granularity, data objects, and output requirements.

[0136] Decompose the user's query requirements into tasks and issue data retrieval tasks to the data retrieval agent in the domain agent;

[0137] The main agent receives the storage information of the to-be-retrieved field fed back by the data retrieval agent in the domain agent;

[0138] Obtaining fields related to the user's query requirements from among the to-be-retrieved fields fed back by the data retrieval agent;

[0139] Determine the relevant data domains where the fields related to the user's query requirements are located;

[0140] Generate an execution plan based on the user's query requirements;

[0141] According to the execution plan, data computing tasks are issued to data computing agents in related data domains.

[0142] In some embodiments, it also includes:

[0143] The master agent receives the result data set fed back by the data calculation agents from multiple domain agents, and caches the result data set into the fusion data warehouse;

[0144] The result data set is extracted from the fusion data warehouse for fusion calculation to generate a fused result data set.

[0145] In some more specific embodiments, it also includes:

[0146] The data computing agent determines whether the data type of the search result or the field in the data domain is structured data, and processes it using a general database query or a data table DataFrame operation;

[0147] The data type of the fields in the search results or data domain is unstructured data, which is processed using natural language processing methods;

[0148] The data type of the fields in the search results or data domain is semi-structured data, which is processed after being parsed and converted into structured data or unstructured data.

[0149] According to another aspect of the embodiment, there is also provided a data weaving system based on dialogue and multi-agent collaboration, the data weaving system comprising a main agent and at least one domain agent, a domain agent being a collection of sub-agents that manage a data domain;

[0150] The main agent is used to identify user intentions and to decompose and allocate data processing tasks according to user intentions. The decomposed data processing tasks include data retrieval tasks, data calculation tasks, and data fusion tasks.

[0151] The main agent is also used to issue data retrieval tasks to the data retrieval agent in the domain agent. The data retrieval task is used to obtain the storage information of the field to be retrieved in the data domain;

[0152] The data retrieval agent is used to perform data retrieval tasks on the managed data domain, obtain retrieval results, including the table structure where the field to be retrieved is located and the storage information in the table structure, and return the retrieval results to the main agent;

[0153] The main agent is used to issue data computing tasks to the data computing agents in the domain agents. The data computing tasks are used to perform calculations based on the search results or one or more fields in the data domain.

[0154] The data computing agent is used to execute the data computing task, obtain the result data set, which includes a two-dimensional table structure calculated based on the search results or the fields in the data domain, and return the result data set to the main agent;

[0155] The main agent is used to perform data fusion tasks on the retrieval results and / or result data sets, and return the fused data sets to the user.

[0156] This embodiment utilizes intelligent agent technology, and users can make data weaving requests through a dialogue mode. The system has the ability to automatically identify user intentions and intelligently disassemble and reorganize the requests, thereby greatly improving the convenience of user use.

[0157] By using multi-agent collaborative technology, the data weaving process has been significantly simplified and optimized. There is no need to build a global knowledge graph for centralized management, which reduces the risk of data inconsistency. Through close collaboration and reasonable division of labor among multiple agents, tasks can be completed efficiently. At the same time, this collaborative mechanism enhances the fault tolerance of the data weaving process, and can also effectively integrate and allocate various resources to ensure that the data weaving system can operate continuously and stably.

[0158] With the autonomy and intelligence of the agent, the domain agent can autonomously and intelligently manage the data directory, data quality, and data security of its domain. It provides intelligent data enhancement retrieval capabilities, can automatically generate and execute SQL statements for data processing based on the optimal strategy and NL2SQL technology to complete data computing tasks, and can also implement intelligent security strategies for data within the domain.

[0159] like Fig. 9It is shown that according to a third aspect, a data weaving system implementation device based on dialogue and multi-agent collaboration is provided, the data weaving system includes a main agent and at least one domain agent, a domain agent is a collection of sub-agents that manage data assets of a data domain, wherein a domain agent includes a data retrieval agent and a data calculation agent;

[0160] The first processing device is used to identify the user's intention through dialogue with the user, and to disassemble and allocate the data processing tasks according to the user's intention, wherein the disassembled data processing tasks include data retrieval tasks, data calculation tasks and data fusion tasks; and to issue the data retrieval tasks to the data retrieval agent in the domain agent, wherein the data retrieval tasks are used to obtain the storage information of the field to be retrieved in the data domain;

[0161] The second processing device is used for the data retrieval agent to perform a data retrieval task on the managed data domain, obtain a retrieval result, which includes the table structure where the field to be retrieved is located and the storage information in the table structure, and return the retrieval result to the main agent;

[0162] The third processing device is used for the master agent to issue a data computing task to the data computing agent in the domain agent, and the data computing task is used to perform calculation based on the search result or one or more fields in the data domain;

[0163] A fourth processing device is used for the data computing agent to perform the data computing task, obtain a result data set, the result data set includes a two-dimensional table structure calculated based on the search results or the fields in the data domain, and return the result data set to the main agent;

[0164] The fifth processing device is used for the main agent to perform data fusion tasks on the retrieval results and / or result data sets, and return the fused data sets to the user.

[0165] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation methods of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the scope of protection of the present application.

Claims

1. A method for implementing a data weaving system based on dialogue and multi-agent collaboration, characterized in that: The data weaving system includes a main agent and at least one domain agent, a domain agent is a collection of sub-agents that manage data assets of a data domain, wherein a domain agent includes a data retrieval agent and a data calculation agent; The method comprises: The main agent performs user intention recognition, and disassembles and distributes data processing tasks according to the user intention, wherein the disassembled data processing tasks include data retrieval tasks, data calculation tasks and data fusion tasks; The master agent issues a data retrieval task to the data retrieval agent in the domain agent, wherein the data retrieval task is used to obtain storage information of the field to be retrieved in the data domain; The data retrieval agent performs a data retrieval task on the managed data domain, obtains a retrieval result, the retrieval result includes the table structure where the field to be retrieved is located and the storage information in the table structure, and returns the retrieval result to the main agent; The master agent issues a data computing task to a data computing agent in a domain agent, wherein the data computing task is used to perform calculations based on the search results and / or one or more fields in the data domain; The data computing agent executes the data computing task to obtain a result data set, wherein the result data set includes a two-dimensional table structure calculated based on the search result or the fields in the data domain, and returns the result data set to the main agent; The master agent performs a data fusion task on the retrieval results and / or the result data set, and returns the fused data set to the user.

2. The method according to claim 1, characterized in that: The data retrieval agent performs a data retrieval task on the managed data domain, obtains a retrieval result, and the retrieval result includes the table structure of the field to be retrieved and the storage information in the table structure, and returns the storage information of the field to be retrieved to the main agent, specifically including: The data retrieval agent constructs a table structure and a view structure of the data domain based on a large language model, and simplifies the table structure and the view structure; Using the simplified table structure and the view structure, construct a large model prompt word, wherein the large model prompt word is used to describe the table structure and the view structure; Based on the large language model, sorting out the business semantic knowledge system of the data domain, wherein the business semantic system is used to describe the business terminology information of the data domain, including the large model prompt words; Using the business semantic knowledge system to construct a business data semantic knowledge framework, the business data semantic knowledge framework includes metadata of the data domain, and the metadata includes a definition of the table structure and a definition of the field; Performing unified knowledge vectorization processing on the business data semantic knowledge framework to obtain a data semantic layer knowledge base of the data domain, wherein the data semantic layer knowledge base is a repository for managing and organizing metadata of the data domain; Based on the field information to be retrieved, a vector search and recall operation is performed on the data semantic layer knowledge base to obtain the table structure where the field to be retrieved is located and the storage information in the table structure.

3. The method according to claim 1, characterized in that The data computing agent executes the data computing task to obtain a result data set, wherein the result data set includes a two-dimensional table structure calculated based on the search result or the fields in the data domain, and returns the result data set to the main agent, specifically including: The data computing agent generates computing task information according to the user intention in the data computing task, wherein the computing task information includes a data asset identifier; According to the data asset identifier in the computing task information, data asset information is obtained from the data domain, wherein the data asset information includes the type of data source, the connection mode of the data source, and the data volume of the data asset; When the amount of data in the data source is less than the preset amount of data and the data involved in the calculation are all in the data source, and the conditions for being sunk to the data source for calculation are met, a data query statement applicable to the data source type and / or a grammatical statement applicable to the data source type are generated according to the user intention and the calculation task information, the data calculation task, and the data query statement applicable to the data source type and / or the grammatical statement applicable to the data source type are sunk to the data source for calculation, and a result data set of the data source is obtained; When the amount of data in the data source is greater than the preset amount of data or other data in the domain is required to participate in the calculation, and the conditions for sinking to the data source for calculation are not met, it is necessary to rely on the high-performance computing engine of the data computing agent to perform data calculations, and generate data query statements and / or grammatical statements suitable for the computing engine according to the user intention and the computing task information, extract qualified data into the computing engine, and let the computing engine execute the data computing task to obtain the result data set; The result data set is returned to the master agent.

4. The method according to claim 1, characterized in that: The domain agent also includes a data governance agent, and the data processing task also includes a data governance task; The method further comprises: The master agent issues data governance tasks to the data governance agent in the domain agent, and the data governance tasks are used to manage and maintain the quality, security and permissions of the data in the data domain or the result data set; The data governance agent performs data governance tasks on the managed data domain and / or the result data set, obtains the governed result data set, and returns the governed result data set to the main agent.

5. The method according to claim 4, characterized in that The data governance agent performs data governance tasks on the managed data domain and / or the result data set to obtain a governed result data set, and returns the governed result data set to the master agent, specifically including: The data governance agent determines the data to be governed according to the data governance task, and the data to be governed includes the data in the data domain or the result data set returned by the data computing agent; Check whether there are data quality problems in the data to be managed, and automatically repair the data quality problems that can be automatically repaired, including missing values, duplicate values ​​and / or format errors. For data quality problems that cannot be automatically repaired, feedback indicates that the data has hidden dangers; When the sensitive data in the data to be managed has not been desensitized or encrypted, the sensitive data is desensitized or encrypted according to the security policy, and for sensitive data without access rights, insufficient permission information is fed back; The input and output of data in the data domain are monitored in real time, abnormal fluctuations in the amount of data in the data domain are detected, and abnormal fluctuations are fed back.

6. The method according to claim 1, characterized in that The method further comprises: The main agent and the user communicate through natural language to obtain the user's intention, which includes the user's query requirements, and the query requirements include the query time range, time granularity, data objects and output requirements; Decomposing the query requirements of the user into tasks, and issuing data retrieval tasks to the data retrieval agent in the domain agent; The master agent receives the storage information of the to-be-retrieved field fed back by the data retrieval agent in the domain agent; Obtaining fields related to the user's query requirements from among the to-be-searched fields fed back by the data search agent; Determine the relevant data domain where the fields related to the user's query requirements are located; Generate an execution plan based on the query requirements of the user; According to the execution plan, data computing tasks are issued to data computing agents in relevant data domains.

7. The method according to claim 1, characterized in that The method further comprises: The master agent receives a result data set fed back by data calculation agents in the plurality of domain agents, and caches the result data set into a fusion data warehouse; The result data set is extracted from the fusion data warehouse for fusion calculation to generate a fused result data set.

8. The method according to claim 3, characterized in that The method further comprises: The data computing agent determines that the data type of the search result or the field in the data domain is structured data, and processes it using a general database query or a data table DataFrame operation; The data type of the search result or the field in the data domain is unstructured data, which is processed by a natural language processing method; The data type of the search result or the field in the data domain is semi-structured data, which is processed after being parsed and converted into structured data or unstructured data.

9. A data weaving system based on dialogue and multi-agent collaboration, characterized in that: The data weaving system includes a main agent and at least one domain agent, and a domain agent is a collection of sub-agents that manage a data domain; The main agent is used to identify the user's intention and to disassemble and allocate the data processing tasks according to the user's intention, wherein the disassembled data processing tasks include data retrieval tasks, data calculation tasks and data fusion tasks; The master agent is further used to issue a data retrieval task to the data retrieval agent in the domain agent, wherein the data retrieval task is used to obtain storage information of the field to be retrieved in the data domain; The data retrieval agent is used to perform data retrieval tasks on the managed data domain, obtain retrieval results, including the table structure where the field to be retrieved is located and the storage information in the table structure, and return the retrieval results to the main agent; The master agent is used to issue data computing tasks to the data computing agent in the domain agent, wherein the data computing tasks are used to perform calculations based on the search results and / or one or more fields in the data domain; The data computing agent is used to execute the data computing task, obtain a result data set, wherein the result data set includes a two-dimensional table structure calculated based on the search results or the fields in the data domain, and return the result data set to the main agent; The main agent is used to perform a data fusion task on the retrieval results and / or the result data set, and return the fused data set to the user.

10. A data weaving system implementation device based on dialogue and multi-agent collaboration, characterized in that: The data weaving system includes a main agent and at least one domain agent, a domain agent is a collection of sub-agents that manage data assets of a data domain, wherein a domain agent includes a data retrieval agent and a data calculation agent; The first processing device is used for the main agent to identify the user's intention, and to disassemble and allocate the data processing tasks according to the user's intention, wherein the disassembled data processing tasks include data retrieval tasks, data calculation tasks and data fusion tasks; the main agent sends the data retrieval task to the data retrieval agent in the domain agent, and the data retrieval task is used to obtain the storage information of the field to be retrieved in the data domain; The second processing device is used for the data retrieval agent to perform a data retrieval task on the managed data domain, obtain a retrieval result, the retrieval result includes the table structure where the field to be retrieved is located and the storage information in the table structure, and return the retrieval result to the main agent; A third processing device, configured for the master agent to issue a data computing task to a data computing agent in a domain agent, wherein the data computing task is used to perform calculation based on the search result and / or one or more fields in the data domain; A fourth processing device is used for the data computing agent to execute the data computing task, obtain a result data set, the result data set including a two-dimensional table structure calculated based on the search result or the fields in the data domain, and return the result data set to the main agent; The fifth processing device is used for the main agent to perform a data fusion task on the retrieval results and / or the result data set, and return the fused data set to the user.

Images