Method and device for intelligently acquiring data based on large model

Abstract

The invention provides an intelligent data acquisition method and device based on a large model. The method comprises the following steps: deploying an RAG model framework by adopting a private environment; collecting and preprocessing table structure information in a business system database, converting the table structure information into a large model acceptable data document, and mounting the large model acceptable data document to a knowledge base in the RAG model framework; maintaining and updating a data source and a database type of report data in the RAG model framework; according to a search request and search conditions submitted by a user side, searching and analyzing the content to be searched by the user side in the knowledge base; and converting the analyzed search content into a report file according to a report downloading request submitted by a user side. By combining a large model technology and an RAG framework, cooperative work of intelligent data report demand identification, information retrieval, query statement generation, configuration management, report generation and downloading modules is realized; and a rapid, accurate and personalized report generation and downloading process is realized.

Description

Technical Field

[0001] The present invention belongs to the field of data reporting, and in particular, relates to a method and device for intelligently acquiring data based on a large model. Background Art

[0002] In e-commerce systems, users need personalized sales data reports when entering the market. These reports are usually one-time requirements, but developing new reports takes a long time and is costly, and lacks flexibility and quick response.

[0003] In the prior art, the traditional data report download demand processing method requires developers to spend a lot of time and resources to design and generate reports, which increases development costs and time; secondly, users usually need to wait for developers to complete report generation, which leads to slow response speed; in addition, since each user's needs may be different, traditional data reports are difficult to meet personalized needs and need to continuously develop new reports. In summary, the traditional method is inefficient and inflexible, unable to quickly adapt to changes in user needs, and increases system maintenance and development costs. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention proposes a data intelligent acquisition method based on a large model, which comprises:

[0005] Adopt privatized environment to deploy RAG model framework;

[0006] Collect and pre-process the table structure information in the business system database, convert it into a data document acceptable to the large model, and mount it into the knowledge base in the RAG model framework;

[0007] Maintain and update the data source and database type of report data in the RAG model framework;

[0008] According to the search request and search conditions submitted by the user, searching and parsing the content to be searched by the user in the knowledge base;

[0009] According to the report download request submitted by the user, the parsed search content is converted into a report file.

[0010] Furthermore, the deployment of the RAG model framework includes:

[0011] Building a hardware model that supports high-intensity computing, and installing basic software on the hardware model;

[0012] Obtain the RAG model and corresponding code base, and build a corresponding exclusive knowledge database based on the application scenario;

[0013] Containerize applications using Docker and deploy the RAG model framework in a private environment.

[0014] Furthermore, the collection and preprocessing of table structure information in the business system database includes:

[0015] The table structure information includes table name, field name, field type, primary key, foreign key and index metadata;

[0016] The preprocessing is to construct a knowledge graph based on the table structure information, and then convert the knowledge graph into a data document acceptable to the large model;

[0017] The conversion process is to encode the relational structure into a sequence, a vector or a specific text description;

[0018] The RAG model framework knowledge base also contains user historical query data.

[0019] Furthermore, the data source includes an address and port, a database name, authentication information, a database driver and a dynamic data source loader;

[0020] The address and port are the network address and port number of the database server;

[0021] The database name is the specific database instance name where the report database is located;

[0022] The authentication information is a username and password that is stored in encrypted form, and a key management system is used to ensure security;

[0023] The database driver automatically identifies the database type and loads the corresponding database driver program;

[0024] The dynamic data source loader can automatically identify the required data source type and initialize the corresponding database connection according to the runtime requirements of generating reports;

[0025] Furthermore, the dynamic data source loader also includes an intelligent routing algorithm; the intelligent routing algorithm is used to achieve seamless integration of a single data source or multiple data sources, and automatically switch data sources according to data source load, priority or strategy.

[0026] Furthermore, the search conditions include search data fields, screening conditions, and defined report document templates imported by the user;

[0027] The step of parsing the search conditions and generating search results includes:

[0028] When a user performs a search, the RAG framework expects the search results to be data fields and business meaning descriptions;

[0029] When the user needs to execute the download, the RAG framework expects the search results returned to be executable query statements;

[0030] The query statement generation includes generating data fields or statements that can be used for query by using the RAG framework and large model technology to analyze the search content.

[0031] Furthermore, after the user submits a report download request, the data source configuration is loaded and an SQL query is executed;

[0032] The converting the parsed search content into a report file includes: converting the query statement into a data structure, and bringing it into a preset report template to generate a report file;

[0033] The generating of the download link comprises: exporting the generated report file in a format specified by the user, uploading the report file to a publicly accessible directory in the memory, and generating a download link.

[0034] Furthermore, the exporting of the generated report file includes: using a high-efficiency coding and compression algorithm to export the generated report file;

[0035] The storage includes a cloud storage server and a local server;

[0036] The download link is sent to the user end in a streaming manner; when the user end receives the link, the report file can be downloaded.

[0037] Furthermore, the download link is sent to the user end in a streaming manner; when the user end receives the link, the report file can be downloaded.

[0038] A data intelligent acquisition system based on a large model, comprising: a data report search module, a search analysis module, an RGA framework module, a knowledge base management module, a search statistics module, a configuration management module, a report generation module and a report download module, characterized in that:

[0039] The data report search module is used to allow the user to input search conditions, perform searches, identify and feedback available fields;

[0040] The search and analysis module is used to perform search and download operations according to the search conditions input by the user, and to analyze and clean the search results;

[0041] The RAG framework module is used to collect data and build a knowledge repository based on the collected data;

[0042] The knowledge base management module is used to regularly update and maintain the knowledge repository in the RAG framework module;

[0043] The search statistics module is used to record the frequency of user input failures or abnormalities and similar search conditions;

[0044] The configuration management module is used to maintain the data source and database type of the report data;

[0045] The report generation module is used to perform a search from the corresponding data source according to the search results after parsing and cleaning, and complete the query and report download link generation within the safe time window;

[0046] The report download module is used to retrieve the report download link generated in the report generation module after the user submits a report download request, and send it to the user end after compression processing.

[0047] Compared with the prior art, the present invention has the following advantages:

[0048] 1. The present invention proposes a method and device for intelligent data acquisition based on a big model. By combining the big model technology and the RAG framework, the collaborative work of intelligent data report demand identification, information retrieval, query statement generation, configuration management, report generation and download modules is realized.

[0049] 2. The present invention allows users to input requirements through natural language, the system intelligently identifies and generates query statements, uses the configuration management module to obtain the correct data source, the report generation module completes the query and generates reports, and finally the report download module compresses the report data and returns it to the user in a streaming manner, thereby realizing a fast, accurate and personalized report generation and downloading process.

[0050] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 A flow chart of a method for intelligent data acquisition based on a large model of the present invention is shown.

[0053] Figure 2 A schematic diagram of a data intelligent acquisition device based on a large model of the present invention is shown.

[0054] Figure 3 The functional architecture diagram of data search and report downloading implemented based on the present invention is shown. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] like Figure 1 As shown, the present invention proposes a data intelligent acquisition method based on a large model, the steps of which include:

[0057] S1. Deploy the RAG model framework in a privatized environment.

[0058] S1.1. Build a hardware environment that supports high-intensity computing and install necessary basic software.

[0059] S1.2. Obtain the RAG model and the corresponding code library, and build a dedicated knowledge database based on the application scenario.

[0060] S1.3. Containerize applications using Docker and deploy the RGA model framework on a private server or cloud platform.

[0061] Alternatively, deploy on a private server or cloud platform to ensure the availability and security of the service interface; after deployment, implement comprehensive testing and monitoring, and continuously perform performance tuning and model iteration to ensure efficient and stable operation of the system.

[0062] S2. Collect and preprocess the business database table structure information, convert it into a data document acceptable to the large model, and mount it into the RAG knowledge base.

[0063] S2.1. Data collection and preprocessing, including:

[0064] Extract table structure information from the database of the business system;

[0065] The extracted information is preprocessed and converted into a format acceptable to the large model for further processing.

[0066] Optionally, the table structure information includes metadata such as table name, field name, field type, primary key, foreign key, index, etc.

[0067] The acceptable formats for the large model are JSON or CSV.

[0068] S2.2. Construct a domain knowledge graph based on table structure information and relationships;

[0069] Optionally, step S2.2 requires identifying entities (such as table corresponding entities), relationships (such as foreign key associations), and attributes of entities (table fields); using graph databases or knowledge graph storage technology to store these structured knowledge.

[0070] S2.3. Convert the knowledge graph or structured data into an input format acceptable to the large model.

[0071] Optionally, the conversion process is to encode the relational structure into a sequence, a vector or a specific text description.

[0072] S2.4. Query the SQL and data fields of the report data manually provided to users in the historical data, and organize the documents in the form of standard cases.

[0073] S2.5. Mount the data, business information, knowledge graph, historical data query and other large model acceptable data documents output in steps S2.1-S2.4 into the RAG knowledge base.

[0074] S3. Maintain the data source and database type of the report data to provide the correct data source for subsequent query data.

[0075] Optionally, the data source includes: an address and a port, a database name, authentication information, a database driver and a dynamic data source loader;

[0076] The address and port are the network address and port number of the database server;

[0077] The database name is the specific database instance name where the report database is located;

[0078] The authentication information is a username and password that is stored in encrypted form, and a key management system is used to ensure security;

[0079] The database driver automatically identifies the database type (such as MySQL, Oracle, etc.) and loads the corresponding database driver;

[0080] The dynamic data source loader can automatically identify the required data source type and initialize the corresponding database connection according to the runtime requirements of generating reports.

[0081] Optionally, the dynamic data source loader further includes an intelligent routing algorithm, supports seamless integration of a single data source or multiple data sources, and can automatically switch data sources according to data source load, priority or strategy.

[0082] S4. The user submits a search request and search conditions, and the content that meets the search conditions is searched and parsed in the knowledge base.

[0083] S4.1. Receive the search request and search conditions input by the user.

[0084] The search conditions include: search data fields, screening conditions (such as time range, category and data field priority, etc.) and imported defined report document templates.

[0085] When the search finds identifiable response fields, the user selects a response field combination as needed and submits a report download request.

[0086] S4.2. Parse the search conditions entered by the user to form a text description that is easy for the large model to handle, and then pass it into the RAG framework to retrieve the expected results.

[0087] Optionally,

[0088] When a user performs a search, the RAG framework expects the results to be data fields and business meaning descriptions;

[0089] When the user executes the download, the RAG framework expects the returned result to be an executable query statement. After obtaining the query statement, the report generation module is called to execute the report download.

[0090] This step also uses technologies such as JSON Schema to validate the input and ensure that the request is formatted correctly.

[0091] Optionally, the query statement is generated by using the RAG framework and large model technology to generate data fields or statements that can be used for querying the parsed search content.

[0092] S5. After the user submits a report download request, the parsed search content is converted into a report file, a download link is generated, and the download link is sent to the user end.

[0093] S5.1. After the user submits a report download request, the data source configuration is loaded and the SQL query is executed;

[0094] Optionally, for complex queries, paging and caching strategies can be used to optimize performance.

[0095] S5.2. Convert the query statement into a data structure and bring it into a preset report template (such as a table) to generate a report file.

[0096] Optionally, the report template may use libraries such as Apache POI (Excel processing), JasperReports (complex report design), etc. to implement diversified report templates.

[0097] S5.3. Export the generated report file according to the format specified by the user.

[0098] Optionally, this step uses efficient encoding and compression algorithms to ensure that the file size is small and the loading speed is fast.

[0099] S5.4. Upload the generated report file to a cloud storage service or a public access directory on a local server to generate a temporary, secure download link.

[0100] Optionally, the validity period of the download link may be set according to a security policy to ensure data privacy protection.

[0101] S5.5. Send the report download link back to the search user end; after the user end receives the link, the user can click to download, realizing a seamless data acquisition experience.

[0102] like Figure 2 As shown, the present invention provides a data intelligent acquisition device based on a large model, which includes: a data report search module, a search analysis module, a RAG framework module, a knowledge base management module, a search statistics module, a configuration management module, a report generation module and a report download module.

[0103] The data report search module allows the user to input requirements through natural language or import report formats, perform search identification and feedback available fields, so that the user can customize the report according to the available fields.

[0104] The search and analysis module performs search and download operations according to the conditions input by the user, and parses and cleans the search results in preparation for subsequent input into the RAG framework module.

[0105] Optionally, when a user performs a search, data is searched based on the conditions entered by the user, and the results are parsed to organize and classify the queryable data fields; when a user performs a report download, based on the response results generated by the RAG framework module, the search parsing module then calls the report download module to generate a data report.

[0106] The RAG framework module collects the organization's own data, builds a knowledge repository based on the collected data, and calls the knowledge base management module to continuously update the repository to help generative AI technicians provide timely contextual answers.

[0107] The knowledge base management module regularly updates and maintains the knowledge repository in the RAG framework module to ensure that the system can identify the latest data field requirements and specifications, improve the device's understanding of user needs, and ensure the reliability and accuracy of the device.

[0108] The search statistics module records the frequency of user input failures or abnormal situations and similar search conditions, improves the accuracy and stability of report generation, and helps the device better meet user needs.

[0109] The configuration management module maintains the data source and database type of the report data to ensure that the data source can be accurately obtained for query and report generation after the query statement is returned by the search.

[0110] The report generation module performs a search from the corresponding data source according to the query statement and data source user context passed in by the search analysis module, ensuring that the query and report download link generation are completed within a safe time window.

[0111] After the user submits a report download request, the report download module retrieves the report download link generated in the report generation module, compresses it, and then transmits it to the user's browser in a streaming manner, so that the user can download the required report data.

[0112] The above description and the accompanying drawings fully illustrate the embodiments of the present invention so that those skilled in the art can practice them. Other embodiments may include structural and other changes. The embodiments represent only possible variations. Unless explicitly required, individual components and functions are optional, and the order of operations may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The embodiments of the present invention are not limited to the structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A data intelligent acquisition method based on a large model, comprising: Adopt privatized environment to deploy RAG model framework; Collect and pre-process the table structure information in the business system database, convert it into a data document acceptable to the large model, and mount it into the knowledge base in the RAG model framework; Maintain and update the data source and database type of report data in the RAG model framework; According to the search request and search conditions submitted by the user, searching and parsing the content to be searched by the user in the knowledge base; According to the report download request submitted by the user, the parsed search content is converted into a report file.

2. The method for intelligent data acquisition based on a large model according to claim 1, characterized in that: The deployment RAG model framework includes: Building a hardware model that supports high-intensity computing, and installing basic software on the hardware model; Obtain the RAG model and corresponding code base, and build a corresponding exclusive knowledge database based on the application scenario; Containerize applications using Docker and deploy the RAG model framework in a private environment.

3. The method for intelligent data acquisition based on a large model according to claim 1, characterized in that: The collecting and preprocessing of table structure information in the business system database includes: The table structure information includes table name, field name, field type, primary key, foreign key and index metadata; The preprocessing is to construct a knowledge graph based on the table structure information, and then convert the knowledge graph into a data document acceptable to the large model; The conversion process is to encode the relational structure into a sequence, a vector or a specific text description; The RAG model framework knowledge base also contains user historical query data.

4. The method for intelligent data acquisition based on a large model according to claim 1, characterized in that: The data source includes an address and port, a database name, authentication information, a database driver, and a dynamic data source loader; The address and port are the network address and port number of the database server; The database name is the specific database instance name where the report database is located; The authentication information is a username and password that is stored in encrypted form, and a key management system is used to ensure security; The database driver automatically identifies the database type and loads the corresponding database driver program; The dynamic data source loader can automatically identify the required data source type and initialize the corresponding database connection according to the runtime requirements of generating reports.

5. The method for intelligent data acquisition based on a large model according to claim 4 is characterized in that: The dynamic data source loader also includes an intelligent routing algorithm; the intelligent routing algorithm is used to achieve seamless integration of a single data source or multiple data sources, and automatically switch data sources according to data source load, priority or strategy.

6. The method for intelligent data acquisition based on a large model according to claim 1, characterized in that: The search conditions include search data fields, screening conditions, and defined report document templates imported by the user; The step of parsing the search conditions and generating search results includes: When a user performs a search, the RAG framework expects the search results to be data fields and business meaning descriptions; When the user needs to execute the download, the RAG framework expects the search results returned to be executable query statements; The query statement generation includes generating data fields or statements that can be used for query by using the RAG framework and large model technology to analyze the search content.

7. The method for intelligent data acquisition based on a large model according to claim 1, characterized in that: After the user submits a report download request, the data source configuration is loaded and the SQL query is executed; The converting the parsed search content into a report file includes: converting the query statement into a data structure, and bringing it into a preset report template to generate a report file; The generating of the download link comprises: exporting the generated report file in a format specified by the user, uploading the report file to a publicly accessible directory in the memory, and generating a download link.

8. The method for intelligent data acquisition based on a large model according to claim 7 is characterized in that: The exporting of the generated report file includes: using efficient coding and compression algorithms to export the generated report file; The storage includes a cloud storage server and a local server; The download link is sent to the user end in a streaming manner; when the user end receives the link, the report file can be downloaded.

9. The method for intelligent data acquisition based on a large model according to claim 7, characterized in that: The download link is sent to the user end in a streaming manner; when the user end receives the link, the report file can be downloaded.

10. A data intelligent acquisition system based on a large model, comprising: The data report search module, search analysis module, RGA framework module, knowledge base management module, search statistics module, configuration management module, report generation module and report download module are characterized in that: The data report search module is used to allow the user to input search conditions, perform searches, identify and feedback available fields; The search and analysis module is used to perform search and download operations according to the search conditions input by the user, and to analyze and clean the search results; The RAG framework module is used to collect data and build a knowledge repository based on the collected data; The knowledge base management module is used to regularly update and maintain the knowledge repository in the RAG framework module; The search statistics module is used to record the frequency of user input failures or abnormalities and similar search conditions; The configuration management module is used to maintain the data source and database type of the report data; The report generation module is used to perform a search from the corresponding data source according to the search results after parsing and cleaning, and complete the query and report download link generation within the safe time window; The report download module is used to retrieve the report download link generated in the report generation module after the user submits a report download request, and send it to the user end after compression processing.

Images