Business-driven data analysis method and system based on large model agent
By integrating domain knowledge and multi-source data processing into a large language model, the accuracy and adaptability issues of large language model agents in data analysis are solved, enabling efficient and accurate business-driven data analysis and lowering the barrier to entry.
Patent Information
- Application Number
- CN202511770720.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-02-27
AI Technical Summary
Existing large language model agents suffer from problems in data analysis, such as lack of domain knowledge, insufficient multi-hop reasoning ability, mismatch between static knowledge and the dynamic world, rigid design, high maintenance cost, lack of cognitive and planning ability, weak memory module, and difficulty in evaluation. These issues lead to inaccurate analysis results and difficulty in adapting to complex business scenarios.
By employing defined unstructured data processing methods, domain knowledge is deeply integrated into a large language model. Through multi-source data fusion, entity processing, dimensional analysis, and iterative optimization, transparent control and refined management are achieved, supporting business-driven data analysis processes and optimizing data mining efficiency using vector space computation.
It improves the accuracy and efficiency of data analysis, lowers the barrier to entry, enables business experts to efficiently use AI to solve problems, and provides a general and efficient solution for unstructured data analysis.
Smart Images
Figure CN121579642A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, more particularly, to a business-driven data analysis method and system based on a large model agent. BACKGROUND
[0002] At present, the large language model agent scheme is basically through the RAG technology or the workflow-driven scheme.
[0003] I. Problems and challenges of RAG (Retrieval Augmented Generation) scheme
[0004] The core idea of RAG is "retrieving relevant information from an external knowledge base and providing it as context to the LLM, so as to generate more accurate and reliable answers". But it is far from perfect.
[0005] Retrieval quality is highly sensitive, and retrieval accuracy is a problem: if the retrieval system returns irrelevant, inaccurate or redundant information, the LLM's generation result will be directly "polluted", resulting in factual errors or irrelevant answers. This is very dependent on the quality of the embedding model: can the model really understand the semantics of the query and document fragments; chunking (text segmentation) strategy: unreasonable segmentation will lead to information fragmentation and loss of key context; failure of the "needle in a haystack" test: when you need to find a very specific fact in a huge document library, RAG systems sometimes fail because they may not be able to accurately locate a key sentence from a huge amount of text.
[0006] Limitations and efficiency of context window: even if GPT-4-Turbo has a context window of 128K, when the knowledge base is very large, it is impossible to fit all relevant information into it; information dilution: when too much content is retrieved, the really key information may be overwhelmed by a large amount of secondary text, making it difficult for the LLM to grasp the key points; delay and cost: retrieving and processing a large amount of text and long context will significantly increase API call delay and computing cost.
[0007] Weak multi-hop reasoning ability: for complex problems that require multiple-step reasoning, RAG may need to perform multiple retrievals and integrations, and the current RAG system is still clumsy in how to autonomously plan multiple retrievals, integrate information from different sources and conduct in-depth comparisons, and is easy to lose in the middle steps.
[0008] Static knowledge vs dynamic world: the knowledge base of RAG needs to be updated and maintained continuously, if not updated in time, the answer given by the LLM will still be outdated, which brings additional operation and maintenance cost.
[0009] II. Problems and challenges of workflow-driven scheme
[0010] Workflow-driven agents complete tasks by predefining a series of steps (calling tools, judging conditions, loops, etc.), and their core problem is rigidity and lack of true intelligence.
[0011] Rigid and inflexible: Workflows are pre-designed, like a fixed script. Once encountering situations beyond the script, the agent will be at a loss or perform incorrectly. It cannot handle ambiguous or open-ended goals. For example, the user instruction is "help me plan an interesting weekend activity", such tasks without explicit steps will make it difficult for the workflow engine to start.
[0012] High design and maintenance cost: Designing a reliable workflow for each complex task requires a lot of manual programming and testing. It is a highly specialized and tedious process. When business logic changes, the entire workflow may need to be rebuilt, with poor scalability and maintainability.
[0013] Low "intelligence" level, essentially an automated script: This solution is more like an upgrade of traditional "automation script" or "IFTTT (if-then)", with the core decision logic being artificially pre-set, rather than emerging from the model itself.
[0014] LLM often only plays a "classifier" or "parameter filler" role (e.g., determining which API to call at this step, and filling user input into a certain parameter of the API), without fully utilizing its core reasoning, planning, and creative abilities.
[0015] Complex troubleshooting chain: When a workflow composed of multiple steps fails, it is very difficult to locate the problem. Is it the LLM's understanding error, is it a certain API call failure, or is it a bug in the process logic itself? It's like a detective solving a case.
[0016] High technical threshold: It requires understanding of workflow configuration methods, understanding of large model capability boundaries, and how to set hard logic judgments based on data. This increases the use threshold and reduces the feasibility of agent landing.
[0017] Three, common underlying core problems
[0018] Lack of true "cognition" and "planning" ability: Current agents are mostly "stimulus-response" type, lacking deep understanding of human goals and long-term, hierarchical planning to achieve goals. They are good at executing the next step, but not good at looking at the big picture.
[0019] Weak memory module: the intelligent agent lacks an efficient and structured way to store and utilize historical interaction experience. Each conversation starts almost from scratch, and cannot maintain consistency and learn user preferences over a long period of time.
[0020] Evaluation and verification dilemma: how to evaluate the performance of an intelligent agent in completing complex tasks, lack of reliable and automated evaluation benchmarks and evaluation methods, making iterative optimization subjective and difficult.
[0021] Lack of background knowledge of specific business scenarios: large models themselves lack in-depth background knowledge and processing solutions of business scenarios. Cannot understand how to handle, handle what and other issues.
[0022] Therefore, a business-driven data analysis method and system based on a large model intelligent agent are proposed to solve the difficulties existing in the prior art, which is a problem that needs to be solved by those skilled in the art. SUMMARY
[0023] Therefore, the present application provides a business-driven data analysis method and system based on a large model intelligent agent, which integrates domain knowledge deeply on a large language model through a defined unstructured data processing method. Not only does it complement the short board of professional background knowledge, but also realizes transparent control and fine management of the model reasoning process through a configurable process. Users can start from business and automatically build a full data analysis process through natural language description by the intelligent agent of the present application.
[0024] In order to achieve the above purpose, the present application provides the following technical scheme: A business-driven data analysis method based on a large model intelligent agent, comprising the following steps: S1, multi-source data fusion: accessing multi-source heterogeneous data in unstructured text documents, interaction logs and structured basic databases, and uniformly formatting the multi-source heterogeneous data to generate standard processing source data; S2, entity processing: performing entity recognition and entity linking processing on the standard processing source data, and performing embedding processing on the recognized and linked entities to generate entity vectors; S3, dimension processing: performing parallel multi-dimensional analysis processing on the generated entity vectors; S4, analysis and evaluation: integrated analysis of the results of multi-dimensional analysis processing to generate the final knowledge representation; S5, data storage: storing the knowledge representation and related analysis data into a vector database and a structured query language database at the same time; S6, iterative optimization: accuracy verification is performed on the knowledge representation generated in S4; if the verification fails, S3-S5 are re-executed based on the data stored in the vector database and / or the structured query language database until the knowledge representation that passes the accuracy verification is generated.
[0025] Optionally, the entity recognition processing in S2 is configured to identify named entities from the standard processing source data using natural language processing techniques, and classify the named entities into predefined entity types. The entity linking processing is configured to associate and disambiguate the identified entities with existing standard entities in the knowledge base to determine the unique identification of the entities.
[0026] Optionally, the multi-dimensional analysis processing in S3 includes dimension matching processing, dimension clustering processing, and dimension sorting processing. The dimension matching processing is configured to calculate the semantic similarity between different entity vectors, and associate entities with similar semantics according to a preset threshold. The dimension clustering processing is configured to perform unsupervised learning on the entity vectors using a clustering algorithm, and automatically merge entities with similar characteristics into multiple groups. The dimension sorting processing is configured to quantitatively sort the importance of entities based on their attributes, correlation, frequency of occurrence, or a combination of multiple indicators.
[0027] Optionally, the knowledge representation stored in the vector database in S5 is used to support similarity search and intelligent recommendation applications, and the knowledge representation stored in the structured query language database is used to support traditional query and report applications.
[0028] A business-driven data analysis system based on a large model agent, which applies any one of the above-mentioned business-driven data analysis methods based on a large model agent, includes a multi-source data fusion module, an entity processing module, a dimension processing module, an analysis and evaluation module, a data storage module, and an iterative optimization module. The multi-source data fusion module is connected to the input end of the entity processing module, configured to access multi-source heterogeneous data in unstructured text documents, interaction logs, and structured basic databases, and perform unified formatting processing on the multi-source heterogeneous data to generate standard processing source data. The entity processing module is connected to the input end of the dimension processing module, configured to perform entity recognition and entity linking processing on the standard processing source data, and perform embedding processing on the recognized and linked entities to generate entity vectors. The dimension processing module is connected to the input end of the analysis and evaluation module, configured to perform parallel multi-dimensional analysis processing on the generated entity vectors. An analysis and evaluation module is connected to the input end of the data storage module, configured to perform integrated analysis on the results of the multi-dimensional analysis and processing, and generate a final knowledge representation; A data storage module is connected to the input end of the iterative optimization module, configured to store the knowledge representation and related analysis data into a vector database and a structured query language database at the same time; An iterative optimization module is configured to verify the accuracy of the generated knowledge representation; if the verification fails, the dimension processing module, the analysis and evaluation module and the data storage module are re-executed based on the data stored in the vector database and / or the structured query language database until the knowledge representation that passes the accuracy verification is generated.
[0029] Compared with the prior art, the technical scheme provided by the present application has the following beneficial effects: (1) The present application uses a defined unstructured data processing method to deeply integrate domain knowledge into a large language model, which not only fills the gap in professional background knowledge, but also realizes transparent control and fine management of the model reasoning process through a configurable process; (2) The present application uses a business-driven design concept, allowing users to define the goals and behaviors of intelligent agents through natural language or graphical interfaces, greatly reducing the customization and use threshold of AI agents, so that business experts can efficiently use AI to solve problems without deep technical background; (3) The present application innovatively uses vector space calculation to optimize the underlying logic of large language models in processing statistical reasoning and unstructured data mining, significantly improving the efficiency and accuracy of such tasks, and providing a general and efficient solution for big data analysis of unstructured data. BRIEF DESCRIPTION OF DRAWINGS
[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.
[0031] Figure 1 A processing data flow diagram is provided for the present application; Figure 2 A left application diagram supported by a knowledge space is provided for the present application; Figure 3 A data diagram showing that the local knowledge space can obtain the global knowledge space is provided for the present application; Figure 4The user provided by the present application creates an account, a knowledge space, an entity, a dimension, and a flowchart of various data analysis tasks. DETAILED DESCRIPTION
[0032] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.
[0033] Referring to Figure 1 As shown in the drawings, the present application discloses a business-driven data analysis method based on a large model intelligent agent, comprising the following steps: S1, multi-source data fusion: accessing multi-source heterogeneous data in unstructured text documents, interactive logs and structured basic databases, and uniformly formatting the multi-source heterogeneous data to generate standard processing source data; S2, entity processing: performing entity recognition and entity linking processing on the standard processing source data, and performing embedding processing on the recognized and linked entities to generate entity vectors; S3, dimension processing: performing parallel multi-dimensional analysis processing on the generated entity vectors; S4, analysis and evaluation: performing integrated analysis on the results of multi-dimensional analysis processing to generate final knowledge representation; S5, data storage: storing the knowledge representation and related analysis data to a vector database and a structured query language database at the same time; S6, iterative optimization: verifying the accuracy of the knowledge representation generated in S4; if the verification fails, re-executing S3-S5 based on the data stored in the vector database and / or the structured query language database until the knowledge representation that passes the accuracy verification is generated.
[0034] Further, the entity recognition processing in S2 is used to recognize named entities from the standard processing source data using natural language processing technology, and classify the named entities into predefined entity types; The entity linking processing is used to associate and disambiguate the recognized entities with the existing standard entities in the knowledge base to determine the unique identification of the entities.
[0035] Further, the multi-dimensional analysis processing in S3 includes dimension matching processing, dimension clustering processing and dimension sorting processing; The dimension matching processing is used to calculate the semantic similarity between different entity vectors, and associate the entities with similar semantics according to a preset threshold; Dimension clustering processing is used for unsupervised learning of entity vectors by using a clustering algorithm, and entities with similar features are automatically merged into multiple groups; Dimension ranking processing is used for quantitatively ranking the importance of entities based on the attributes, correlation, frequency of occurrence or combination of multiple indexes of the entities.
[0036] Specifically, the results of the multi-dimensional analysis processing are integrated and analyzed in S4 to generate the final knowledge representation, which can provide analysis evaluation reasons while providing analysis results, break the data analysis black box problem, and the analysis results can be directly obtained from the knowledge space through API, thereby facilitating the integration of the business system itself, in addition, the analysis results can also be used to build a dialogue robot through a dialogue intelligent agent without code.
[0037] Further, the knowledge representation stored in the vector database in S5 is used to support similarity search and intelligent recommendation applications, and the knowledge representation stored in the structured query language database is used to support traditional query and report applications.
[0038] Specifically, since the analysis process in the application uses vector operations, the data filtering and noise caused by hard logic judgment are reduced. At the same time, the structured query language can better perform statistics and analysis.
[0039] In one specific embodiment, as shown in Figure 2 , Figure 3 , Figure 4 includes the following content: CREAM is the acronym of Clustering, Ranking, Embedding, Analyzing, Matching. It is a data analysis platform for developers and non-developers built by Agent. It includes structured data and unstructured data analysis.
[0040] CREAM solves the problem of large-scale unstructured data analysis in the big data era. At the same time, unlike other workflow or agent platforms that solve generalization problems, CREAM focuses more on data analysis, builds agents and various specific data analysis methods, helps users reduce the use threshold, and solves the problem of unreliable general agent data analysis. At the same time, it truly realizes that business personnel can customize their own data analysis according to business needs.
[0041] CREAM uses the universality and flexibility of Agent, taking into account the accuracy and other characteristics required for data analysis. Separate data analysis and requirements, and put data analysis first. Users can use the analysis results of CREAM at any time to complete business functions according to their needs. Simplify the use threshold of data analysis and improve work efficiency.
[0042] When facing non-developers, through the CREAM platform, users can configure their own business domain data model through page, and automatically process imported data through tasks to obtain the required results.
[0043] When facing developers, through the API of the CREAM platform, developers can configure their own business domain data model, upload data through various ways (API, upload file database connection, etc.), and after data analysis, developers can obtain the results of various data analysis through API.
[0044] Users define the entity (entity), dimension (dimension) and related business data analysis tasks of the field to build a business data processing flow: Step 1: Access unstructured text documents, interactive logs, and structured basic databases of multi-source heterogeneous data, and uniformly format the multi-source heterogeneous data to generate standard processing source data; Step 2: Perform entity recognition and entity linking processing on the standard processing source data, and perform embedding processing on the recognized and linked entities to generate entity vectors; Entity recognition processing is used to identify named entities from standard processing source data using natural language processing technology, and classify named entities into predefined entity types; Entity linking processing is used to associate and disambiguate the recognized entities with existing standard entities in the knowledge base to determine the unique identification of the entity; Step 3: Perform parallel multi-dimensional analysis processing on the generated entity vectors; Multi-dimensional analysis processing at least includes dimension matching processing, dimension clustering processing and dimension sorting processing; Dimension matching processing is used to calculate the semantic similarity between different entity vectors, and associate entities with similar semantics according to a pre-set threshold; Dimension clustering processing is used to perform unsupervised learning on entity vectors using clustering algorithms, and automatically merge entities with similar characteristics into multiple groups; Dimension sorting processing is used to quantify and sort the importance of entities based on their attributes, correlation, frequency of occurrence, or a combination of multiple indicators; Step 4: Perform integrated analysis on the results of multi-dimensional analysis processing to generate the final knowledge representation; Step 5: Store the knowledge representation and related analysis data to the vector database and the structured query language database simultaneously; The knowledge representation stored in the vector database is used to support similarity retrieval and intelligent recommendation applications, and the knowledge representation stored in the structured query language database is used to support traditional query and report applications; Step 6: accuracy verification is performed on the generated knowledge representation; if the verification fails, steps 3-5 are re-executed based on the data stored in the vector database and / or the structured query language database until a knowledge representation that passes the accuracy verification is generated.
[0045] A business-driven data analysis system based on a large model agent, applying any one of the above business-driven data analysis methods based on a large model agent, comprising: a multi-source data fusion module, an entity processing module, a dimension processing module, an analysis and evaluation module, a data storage module and an iterative optimization module; The multi-source data fusion module is connected with the input end of the entity processing module, used for accessing multi-source heterogeneous data in unstructured text documents, interactive logs and structured basic databases, and performing unified formatting processing on the multi-source heterogeneous data to generate standard processing source data; The entity processing module is connected with the input end of the dimension processing module, used for performing entity recognition and entity linking processing on the standard processing source data, and performing embedding processing on the recognized and linked entities to generate entity vectors; The dimension processing module is connected with the input end of the analysis and evaluation module, used for performing parallel multi-dimensional analysis processing on the generated entity vectors; The analysis and evaluation module is connected with the input end of the data storage module, used for performing integrated analysis on the results of multi-dimensional analysis processing to generate the final knowledge representation; The data storage module is connected with the input end of the iterative optimization module, used for storing the knowledge representation and related analysis data into the vector database and the structured query language database at the same time; The iterative optimization module is used for accuracy verification of the generated knowledge representation; if the verification fails, the dimension processing module, the analysis and evaluation module and the data storage module are re-executed based on the data stored in the vector database and / or the structured query language database until a knowledge representation that passes the accuracy verification is generated.
[0046] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.
[0047] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and, while certain modifications are discussed, it is desired to be protected in accordance with the spirit and scope of the application. Therefore, the application is not limited to the specific embodiments shown and described, but only by the scope of the appended claims, unless otherwise specified.
Claims
1. A business-driven data analysis method based on a large-scale intelligent agent model, characterized in that, Includes the following steps: S1. Multi-source data fusion: Access multi-source heterogeneous data from unstructured text documents, interaction logs, and structured basic databases, and perform unified formatting on the multi-source heterogeneous data to generate standard processed source data; S2, Entity Processing: Perform entity recognition and entity linking on the standard processing source data, and embed the recognized and linked entities to generate entity vectors; S3, Dimension Processing: Perform parallel multi-dimensional analysis on the generated entity vectors; S4. Analysis and Evaluation: Integrate and analyze the results of multi-dimensional analysis to generate the final knowledge representation; S5. Data storage: Store knowledge representation and related analysis data simultaneously in a vector database and a structured query language database; S6. Iterative optimization: Verify the accuracy of the knowledge representation generated in S4; If the verification fails, S3-S5 are re-executed based on the data stored in the vector database and / or the structured query language database until a knowledge representation that passes the accuracy verification is generated.
2. The business-driven data analysis method based on a large-scale intelligent agent according to claim 1, characterized in that, The entity recognition processing in S2 is used to identify named entities from standard processing source data using natural language processing techniques, and classify the named entities into predefined entity types; Entity linking is used to associate and disambiguate identified entities with existing standard entities in the knowledge base in order to determine the unique identifier of the entity.
3. The business-driven data analysis method based on a large-scale intelligent agent according to claim 1, characterized in that, Multidimensional analysis processing in S3 includes dimension matching processing, dimension clustering processing, and dimension ranking processing; Dimensional matching is used to calculate the semantic similarity between different entity vectors and associate semantically similar entities according to a preset threshold. Dimensional clustering is used to perform unsupervised learning on entity vectors using clustering algorithms, automatically grouping entities with similar features into multiple groups; Dimensional ranking is used to quantify and rank the importance of entities based on their attributes, relevance, frequency of occurrence, or a combination of multiple indicators.
4. The business-driven data analysis method based on a large-scale intelligent agent according to claim 1, characterized in that, In S5, knowledge representations stored in a vector database are used to support similarity retrieval and intelligent recommendation applications, while knowledge representations stored in a structured query language database are used to support traditional query and reporting applications.
5. A business-driven data analysis system based on a large-scale intelligent agent model, characterized in that, The business-driven data analysis method based on a large model intelligent agent according to any one of claims 1-4 includes: a multi-source data fusion module, an entity processing module, a dimension processing module, an analysis and evaluation module, a data storage module, and an iterative optimization module; The multi-source data fusion module is connected to the input end of the entity processing module. It is used to access multi-source heterogeneous data from unstructured text documents, interactive logs, and structured basic databases, and to perform unified formatting processing on the multi-source heterogeneous data to generate standard processing source data. The entity processing module is connected to the input end of the dimension processing module. It is used to perform entity recognition and entity linking processing on the standard processing source data, and to embed the recognized and linked entities to generate entity vectors. The dimension processing module, connected to the input of the analysis and evaluation module, is used to perform parallel multi-dimensional analysis on the generated entity vectors. The analysis and evaluation module, connected to the input of the data storage module, is used to integrate and analyze the results of multi-dimensional analysis and processing to generate the final knowledge representation. The data storage module, connected to the input of the iterative optimization module, is used to simultaneously store knowledge representation and related analysis data into the vector database and the structured query language database. The iterative optimization module is used to verify the accuracy of the generated knowledge representation. If the verification fails, the dimension processing module, analysis and evaluation module, and data storage module are re-executed based on the data stored in the vector database and / or structured query language database until a knowledge representation that passes the accuracy verification is generated.