A data analysis system based on multi-agent cooperation

Abstract

The application discloses a data analysis system based on multi-agent cooperation, comprising a role definition module, a workflow definition module and a large language model driving module; wherein the role definition module is used for defining multi-agents; the workflow definition module is used for constructing a standard workflow based on the role definition module; and the large language model driving module is used for driving the multi-agents to process complex problems based on the standard workflow. The application utilizes the code writing processing capability of a large language model (LLM) to replace the related work of a Python engineer and an ETL data engineer, and greatly reduces the labor input cost of a data analysis task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a data analysis system based on multi-agent cooperation. BACKGROUND

[0002] The traditional data analysis process usually follows a top-down method, such as shown in the following figure, starting from a business problem, first determining the analysis target and requirements, including business indicator definition, etc., and then collecting the required data from the corresponding data source. The collected data will be preprocessed, including cleaning and conversion, to ensure data quality. Then, a suitable data model is designed according to the business indicator definition to process the data. After the model is established and the data processing is completed, the analysis results need to be presented to the stakeholders in the form of reports or dashboards according to the business indicators, to support business decision-making and action. Figure 1

[0003] In the prior art process, a data analyst is usually needed to analyze the business, define the indicators, a data engineer (ETL / Python) is needed to access and clean the data, build the model and process the data, and then hand it over to the data analyst to build the board / dashboards (BI / reporting tools, etc.) and write the report. That is, the traditional data analysis process relies too much on manual processing and lacks system self-processing capability, so the present application proposes a data analysis system based on multi-agent cooperation. SUMMARY

[0004] The purpose of the present application is to provide a data analysis system based on multi-agent cooperation to solve the problems existing in the prior art.

[0005] The data analysis system based on multi-agent cooperation provided by the present application comprises a role definition module, a workflow definition module and a large language model driven module.

[0006] The role definition module is used to define multi-agents.

[0007] The workflow definition module is used to build a standard workflow based on the role definition module.

[0008] The large language model driven module is used to drive the multi-agents to process complex problems based on the standard workflow.

[0009] Optionally, the role definition module comprises a user interaction assistant agent, a data analyst agent, a Python engineer agent, an ETL data engineer agent and a data report writing agent.

[0010] The user interaction assistant agent is used to interact with the user through the interface to obtain the initial user analysis requirements. ​

[0011] The data analyst agent is configured to formulate an analysis plan based on the initial user analysis requirement;

[0012] The Python engineer agent is configured to convert the analysis plan into an executable Python script;

[0013] The ETL data engineer agent is configured to complete data extraction based on the executable Python script and generate a corresponding data analysis result;

[0014] The data report writing agent is configured to generate a data analysis report based on the data analysis result.

[0015] Optionally, the role definition module further comprises a business consultant agent; when the initial user analysis requirement is not clear, the business consultant agent is started;

[0016] The business consultant agent is configured to query business indicators based on the initial user analysis requirement that is not clear to obtain a data analysis indicator suggestion result.

[0017] Optionally, the workflow definition module comprises an analysis requirement collection unit, a data analysis unit and an analysis result generation unit;

[0018] The analysis requirement unit is configured to obtain an initial user analysis requirement based on the user interaction assistant agent;

[0019] The data analysis unit is configured to perform data extraction and analysis on the obtained initial user analysis requirement to obtain a data analysis result;

[0020] The analysis result generation unit is configured to generate a data analysis report based on the data report writing agent and the data analysis result.

[0021] Optionally, the data analysis unit is configured to perform data extraction and analysis on the obtained initial user analysis requirement to obtain a data analysis result, wherein the process of obtaining a line-of-sight analysis result by the data analysis unit comprises:

[0022] determining whether the initial user analysis requirement is clear;

[0023] When the initial user analysis requirement is not clear, the business consultant agent queries business indicators based on an internal database to obtain a data analysis indicator suggestion result, and the user interaction assistant agent feeds back the data analysis indicator suggestion result to the user and generates a new analysis requirement, and when the user confirms, the data analyst agent outputs an analysis plan based on the new analysis requirement, and notifies the Python engineer agent and the ETL data engineer agent to perform data extraction and analysis calculation work based on the analysis plan to obtain a data analysis result.

[0024] When the initial user analysis requirement is clear, the data analyst agent directly outputs an analysis plan based on the initial user analysis requirement, and notifies the Python engineer agent and the ETL data engineer agent to perform data extraction and analysis calculation work based on the analysis plan to obtain a data analysis result.

[0025] Optionally, the large language model driving module comprises: a Prompt 2.1 unit, a Prompt 2.2 unit, a Prompt 2.3 unit, a Prompt 2.5 unit, a Prompt 2.6 unit, a Prompt 2.7 unit, a Prompt 2.8 unit, and a Prompt 2.9 unit.

[0026] The Prompt 2.1 unit is configured to drive the large language model to infer the initial user analysis requirement and determine whether the initial user analysis requirement is clear.

[0027] The Prompt 2.3 unit is configured to summarize the data analysis indicator suggestion result to form a new analysis requirement.

[0028] The Prompt 2.5 unit is configured to drive the large language model to split the new analysis requirement into an analysis plan.

[0029] The Prompt 2.6 unit is configured to drive the large language model to construct Python code.

[0030] The Prompt 2.7 unit is configured to drive the large language model to construct SQL code.

[0031] The Prompt 2.8 unit is configured to drive the large language model to generate an analysis chart and an analysis report based on the data analysis result.

[0032] The Prompt 2.9 unit is configured to drive the large language model to polish the analysis report and feed it back to the user.

[0033] Optionally, the large language model driving module further comprises a Prompt 2.2 unit, when the initial user analysis requirement is not clear, a business consulting expert agent is started, and the business consulting expert agent performs index query based on the Prompt 2.2 unit;

[0034] The Prompt 2.2 unit is used for the business consulting expert agent to assemble index list information based on built-in database query, and obtain a data analysis index suggestion result by reasoning the index list information based on a large language model.

[0035] The present application has the following technical effects:

[0036] The present scheme uses a large language model (LLM) driven multi-agent (Multi-Agent) to complete a complex data analysis task that cannot be processed by a single large language model (LLM). At the same time, the industry knowledge pre-trained by the large language model (LLM) is combined with the business management index library to replace the work of data analysts. The code writing processing capability of the large language model (LLM) is used to replace the related work of Python engineers and ETL data engineers, greatly reducing the human labor cost of data analysis tasks.

[0037] The present scheme has strong adaptability. By adjusting the business management index library and part of the setting of Prompt, it can be applied to various industry data analysis scenarios. It replaces the work of junior data analysts, data engineers and Python engineers, reduces the labor cost, and improves the overall data analysis effect. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application and their description are used to explain the application and are not intended to limit the application. In the drawings:

[0039] Figure 1 It is a schematic diagram of the conventional data analysis process in the background art of the present application;

[0040] Figure 2 It is a schematic diagram of the agent in the embodiment of the present application;

[0041] Figure 3 It is a data analysis system diagram of the multi-agent cooperation in the embodiment of the present application. DETAILED DESCRIPTION

[0042] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0043] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown.

[0044] Embodiment one

[0045] Traditional data analysis systems are formed by complex organization, screening, filtering of business knowledge, index definition, model and data, etc. to form analysis results and present feedback to analysis decision makers. The business index analysis, data model building and data processing require a large amount of manpower and cost, and the results are relatively fixed, which is increasingly unsuitable for the current complex and changing business environment.

[0046] The present application simulates multiple agent roles to organize the data analysis process, disassembles a complex problem, and gradually integrates data, knowledge and models according to the disassembled ideas to form the final analysis results and present them to decision makers based on a large language model as a reasoning engine.

[0047] As shown in Figure 2 In the present technical solution, the concept of agent (Agent) under a large language model (LLM) is used. Agent is a virtual role driven by LLM, which has human-like thinking and planning ability and memory, and can use tools to complete specific tasks as needed.

[0048] In the present solution, multiple agents (Multi-Agent) are used to collaborate to complete complex data analysis tasks. Through multi-agent collaboration, complex tasks can be disassembled and completed by professional roles, avoiding the shortcomings of large language models (LLM) in handling complex problems and automating the original manual data analysis process.

[0049] The present embodiment discloses a data analysis system based on multi-agent collaboration, comprising a role definition module, a workflow definition module and a large language model driving module.

[0050] The role definition module is used to define multiple agents.

[0051] The workflow definition module is used to construct a standard workflow based on the role definition module.

[0052] The large language model driving module is used to drive the multiple agents to process complex problems based on the standard workflow.

[0053] The role definition module includes: User Interaction Assistant Agent, Business Consulting Expert Agent, Data Analyst Agent, Python Engineer Agent, ETL Data Engineer Agent, and Data Report Writer Agent.

[0054] The specific Agent roles are defined as follows:

[0055] UserAssistant (User Interaction Assistant Agent) is responsible for interface interaction with users, collecting their needs, and guiding them through the system for a more optimized experience.

[0056] BusinessConsultant (Business Consulting Expert Agent) provides professional knowledge of business processes according to the user's business field and recommends meaningful Key Performance Indicators (KPIs) for analysis.

[0057] Analyst (Data Analyst Agent) develops a robust and detailed analysis plan based on business needs, including data analysis methods, processes, and frameworks.

[0058] PythonEngineer (Python Engineer Agent) translates data analysis plans into executable Python scripts, focusing on accuracy, efficiency, and scalability.

[0059] DataEngineer (ETL Data Engineer Agent) builds models through SQL scripts based on data analysis plans to generate high-quality data for data analysis.

[0060] Reporter (Data Report Writer Agent) turns data analysis results into charts and summarizes them into clear and effective reports, and conveys the story behind the data through visualization.

[0061] For the above Agent roles, we need to define Profile (description), Goal (target), and Constraints (constraints) for each role. For example, the Data Analyst role:

[0062] Profile: Analytical Strategy Developer / / Definition: Analysis Strategy Developer

[0063] Goal: To devise robust analytical plans including data analysis methodologies, procedures, and frameworks based on business requirements.

[0064] Constraints: Must develop plans that are feasible for implementation with available tools and data.

[0065] In the construction of the Agent, a large language model (LLM) session is initialized, and the Profile, Goal, and Constraints are organized in the template (""You are a {profile}, named {name}, your goal is {goal}. The constraint is {constraints}.""). The organized content is injected into the LLM session in the SystemPrompt format ({"role": "system", "content": msg}).

[0066] When the Agent drives the large language model (LLM) for reasoning through Prompt, in order to prevent users from commanding LLM through natural language, the Prompt adopts the following standard format, which clearly distinguishes between input content (Content) and prompt words (Prompt).

[0067]

[0068] The above content is translated as follows:

[0069]

[0070]

[0071] The workflow definition module includes: analysis requirement collection unit, data analysis unit and analysis result generation unit;

[0072] Among them, the analysis requirement unit is used to obtain the initial user analysis requirement based on the user interaction assistant agent.

[0073] The data analysis unit is used for data extraction and analysis of the obtained initial user analysis requirements, to obtain data analysis results;

[0074] The analysis result generation unit is used for generating data analysis reports based on the data report writing agent. The specific analysis process is as follows:

[0075] As shown in Figure 3 The Multi-Agent (Multi-Agent) collaborative data analysis system obtains the user's input analysis requirement, and performs task decomposition and execution through the following process, to complete the data analysis process:

[0076] The UserAssistant role drives the large language model LLM to reason about the user's analysis requirement through Prompt 2.1, to determine whether the analysis requirement is clear.

[0077] In the case where the analysis requirement is not clear, the BusinessConsultant role is online, and the Agent queries the business indicator database through the built-in database query tool (Tools) with the user input keyword as the query condition, to obtain an indicator list (including indicator name, description, calculation range, data source, etc.). The indicator list information is assembled through Prompt 2.2 and input into the large language model (LLM) for reasoning, and the LLM returns the corresponding data analysis indicator suggestion result.

[0078] The UserAssistant role summarizes the analysis suggestions given by the BusinessConsultant role through Prompt 2.3, and forms a new analysis requirement in combination with the previous analysis requirement, to feedback to the user for communication confirmation.

[0079] After the user confirms the analysis requirement, if the requirement is clear enough, the Analyst role is online.

[0080] The Analyst role drives the large language model (LLM) to reason through Prompt 2.5 according to the data analysis requirement and the data basis, to split and list detailed and executable data analysis plans. The data analysis requirement includes: indicator requirement, analysis dimension (time, region, etc.), limitation condition, etc.; the data basis includes: data table name, field name, model relationship; Prompt 2.5 organizes the above content, and requires the large language model (LLM) to output a standard analysis plan. The analysis plan includes: step-by-step listing of analysis steps (data extraction, analysis calculation), visual presentation form, etc.

[0081] According to the steps in the analysis plan of the Analyst role, notify the Python engineer and the ETL data engineer respectively to perform data extraction and analysis calculation work.

[0082] The Python engineer / ETL engineer constructs corresponding Python code / SQL code according to the analysis requirements by driving a large language model (LLM) through Prompt 2.6 / 2.7, and iteratively runs to complete data extraction and generate corresponding analysis result data.

[0083] After the data analysis result is generated, the Reporter role is online, and according to the analysis requirements, analysis plan and data result, analysis charts and analysis reports are formed through Prompt 2.8.

[0084] The UserAssistant role refines the analysis report through Prompt 2.9 and feeds back to the end user.

[0085] In the above process, each Agent role is driven by a large language model through specific prompts (Prompts) to obtain reasoning ability and task decomposition execution ability. Information sharing between multiple roles is completed through environmental components (Env), such as data link storage information, etc. Message passing between roles is completed through Env's Message broadcast.

[0086] With the reasoning ability of the latest large language model (LLM), the data analysis process is automatically completed through the cooperation of multiple Agents, reducing the dependence on data analysts, ETL data engineers, and Python data engineers, reducing data analysis costs, improving data analysis efficiency, and improving data analysis decision-making ability.

[0087] The above is only the preferred specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data analysis system based on multi-agent collaboration, characterized in that: include: Role definition module, workflow definition module and large language model driving module; Among them, the role definition module is used to define multiple agents; the role definition module includes: a user interaction assistant agent, a data analyst agent, a Python engineer agent, an ETL data engineer agent and a data report writing agent; wherein, the user interaction assistant agent is used to interact with the user through an interface to obtain initial user analysis requirements; the data analyst agent is used to formulate an analysis plan based on the initial user analysis requirements; the Python engineer agent is used to convert the analysis plan into an executable Python script; the ETL data engineer agent is used to complete data extraction and generate corresponding data analysis results based on the executable Python script; the data report writing agent is used to generate a data analysis report based on the data analysis results; The role definition module also includes a business consulting expert agent; when the initial user analysis requirements are unclear, the business consulting expert agent is activated; The business consulting expert agent is used to query business indicators based on the unclear initial user analysis requirements to obtain data analysis indicator recommendation results; The workflow definition module is used to construct a standard workflow based on the role definition module; the workflow definition module includes: an analysis requirement acquisition unit, a data analysis unit and an analysis result generation unit; wherein the analysis requirement unit is used to obtain initial user analysis requirements based on the user interaction assistant agent; the data analysis unit is used to extract and analyze the data of the obtained initial user analysis requirements to obtain data analysis results; the analysis result generation unit is used to generate a data analysis report based on the data analysis results of the data report writing agent; The data analysis unit is used to extract and analyze data from the obtained initial user analysis requirements to obtain data analysis results, wherein the process of the data analysis unit obtaining the data analysis results includes: Determining whether the initial user analysis requirements are clear; When the initial user analysis requirements are unclear, the business consulting expert agent queries the business indicators based on the built-in database to obtain data analysis indicator recommendation results. The user interaction assistant agent feeds back the data analysis indicator recommendation results to the user and generates new analysis requirements. After the user confirms, the data analyst agent outputs an analysis plan based on the new analysis requirements, and based on the analysis plan, notifies the Python engineer agent and the ETL data engineer agent to perform data extraction and analysis calculations to obtain data analysis results. When the initial user analysis requirements are clear, the data analyst agent directly outputs an analysis plan based on the initial user analysis requirements, and notifies the Python engineer agent and the ETL data engineer agent to perform data extraction and analysis calculations based on the analysis plan to obtain data analysis results; The large language model driving module is used to drive the multi-agent to process complex problems based on the standard workflow; the large language model driving module includes: Prompt 2.1 unit, Prompt 2.2 unit, Prompt 2.3 unit, Prompt 2.5 unit, Prompt 2.6 unit, Prompt 2.7 unit, Prompt 2.8 unit, and Prompt 2.9 unit; The Prompt 2.1 unit is used to drive the large language model to infer the initial user analysis requirements and determine whether the initial user analysis requirements are clear; The Prompt 2.3 unit is used to summarize the data analysis indicator suggestion results to form new analysis requirements; The Prompt 2.5 unit is used to drive the large language model to split the new analysis requirements into analysis plans; The Prompt 2.6 unit is used to drive the large language model to build Python code; The Prompt 2.7 unit is used to drive the large language model to build SQL code; The Prompt 2.8 unit is used to drive the large language model to generate analysis charts and analysis reports based on the data analysis results; The Prompt 2.9 unit is used to drive the large language model to polish the analysis report and then feed back to the user; The large language model driving module further includes a Prompt 2.2 unit, which activates a business consulting expert agent when the initial user analysis requirement is unclear, and the business consulting expert agent performs an indicator query based on the Prompt 2.2 unit; The Prompt 2.2 unit is used by the business consulting expert agent to assemble the indicator list information based on the built-in database query, and to infer the indicator list information based on the large language model to obtain the data analysis indicator recommendation results; When the intelligent agent is built, a large language model (LLM) session is initialized, and the data analysis roles are organized into templates. The organized content is injected into the LLM session in the form of System Prompt. When the intelligent agent drives the Large Language Model (LLM) for reasoning, to prevent users from injecting commands into the LLM through natural language, Prompt uniformly adopts the following standard format, clearly marking the difference between input content and prompt words.

Images