Intelligent master data query system based on large model

Abstract

The invention discloses an intelligent master data query system based on a large model. The intelligent master data query system comprises a master data acquisition module, a master data storage module, an advanced query module and a query result display module, the main data acquisition module is used for acquiring main data from each data source; the main data storage module is used for storing and managing the main data acquired by the main data acquisition module; the advanced query module is used for understanding a query request of a user through a language model based on a deep learning technology, generating a corresponding query statement and sending the query statement to the main data storage module; the query module is also used for understanding the query result returned by the main data storage module through the language model and generating a corresponding natural language answer as the query result; and the query result display module is used for displaying the query result in an interface mode. According to the invention, the user can acquire the master data more simply and conveniently, the master data acquisition speed is improved, and the master data quality is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of data query, and in particular relates to an intelligent master data query system based on a large model. Background Art

[0002] The master data query system is an important component of the master data management system (MDM), which is responsible for providing access to the core data stored in the master database.

[0003] 1. The main structure of the master data query system is:

[0004] 1. Front-end interface: Users can initiate query requests through the graphical interface or API interface.

[0005] 2. Query processor: Receives requests from the front end and converts them into appropriate database query statements.

[0006] 3. Master database / data warehouse: stores all cleaned and standardized master data.

[0007] 4. Index and cache layer: It is set up to speed up query response time and can pre-build result sets or partial results of commonly used queries.

[0008] 5. Security control module: ensure that only authorized users can access sensitive information and implement fine-grained permission management.

[0009] 6. Logs and audit trails: Record each query activity for subsequent review and to meet regulatory requirements.

[0010] 2. The working principle of the master data query system is:

[0011] 1. Standardized input: No matter what format the user submits the query conditions in, the system will first convert them into a standard form for easy processing.

[0012] 2. Parsing and verification: Check the validity and legality of the input, such as whether it complies with predefined data types, range restrictions, etc.

[0013] 3. Execute the query: Retrieve relevant information from the main database according to the parsed instructions. This step may involve complex SQL statements or higher-level data operation logic.

[0014] 4. Result filtering and sorting: Based on user preferences (such as sorting by date in descending order), the original returned results are further processed.

[0015] 5. Output presentation: Finally, the processed data is presented to the user in an easy-to-understand way, which may be in the form of tables, charts or other visualization methods.

[0016] In summary, an efficient and reliable master data query system requires not only strong back-end technical support, but also good user experience design to truly realize its value. With the development of cloud computing, big models and other related technologies, the master data query system in the future will become more intelligent, flexible and easy to integrate.

[0017] Although the master data query system plays a key role in the enterprise, it also has some problems and shortcomings in practical applications. These problems are usually related to the system structure, technical implementation and complexity of business requirements. In the traditional master data query system, the function provided to the user by the user interface is generally to display the query results in a table after the user selects the query conditions. If the user wants to further analyze and process the query results, the table data needs to be downloaded to the local computer for statistical analysis. This has the disadvantages of unfriendly user experience, affecting the efficiency of user data use, and manual secondary processing of data may result in low data quality. Summary of the invention

[0018] The technical problem to be solved by the present invention is to provide an intelligent master data query system based on a big model in view of the deficiencies in the above-mentioned prior art, and to improve it by introducing big model technology, so that users can obtain master data more simply and conveniently, improve the speed of master data acquisition and ensure the quality of master data.

[0019] To solve the above technical problems, the technical solution adopted by the present invention is: an intelligent master data query system based on a large model, including a master data acquisition module, a master data storage module, an advanced query module and a query result display module;

[0020] The master data collection module is used to collect master data from various data sources;

[0021] The master data storage module is used to store and manage the master data collected by the master data collection module;

[0022] The advanced query module is used to understand the user's query request through a language model based on deep learning technology, generate a corresponding query statement and send it to the main data storage module;

[0023] It is also used to understand the query result returned by the main data storage module through the language model and generate a corresponding natural language answer as the query result;

[0024] The query result display module is used to display the query results in an interface manner.

[0025] The above-mentioned intelligent master data query system based on the big model also includes a general query module, which pre-stores a number of query conditions and is used to generate a query statement according to the query conditions selected by the user and the query keywords input, and send the query statement to the master data storage module;

[0026] It is also used to send the query result returned by the main data storage module to the query result display module for display.

[0027] The above-mentioned intelligent master data query system based on a large model also includes a permission management module for controlling the user's access rights to the system and data operation rights.

[0028] The above-mentioned intelligent master data query system based on a large model also includes a system monitoring and alarm module for real-time monitoring of the operating status and data quality of the master data query system.

[0029] In the above-mentioned intelligent master data query system based on a large model, the master data collection module is also used to clean and filter the collected master data.

[0030] The above-mentioned intelligent master data query system based on the big model, the master data storage module is used to store the cleaned and filtered master data in the distributed data warehouse hive for backup, and synchronize the master data in the distributed data warehouse hive to the data table corresponding to the distributed SQL query engine Trino.

[0031] In the above-mentioned large-model-based intelligent master data query system, the advanced query module uses the LLM model, a language model based on deep learning technology.

[0032] The query result display module of the above-mentioned intelligent master data query system based on the big model supports displaying the query results in the form of tables, charts and / or reports.

[0033] Compared with the prior art, the present invention has the following advantages: the present invention combines language model related technologies, utilizes language models to enrich the functions of the master data query system, and adds an advanced query module based on a large model. When using the advanced query module to query master data, users can use natural language to describe their needs, which can improve user experience. The language model converts user questions into database query instructions, which can accurately obtain the final data that the user wants and return it to the user in the form of natural language, thereby improving the efficiency of user query, understanding and analysis of data, and overcoming the shortcomings of low efficiency and inability to guarantee data accuracy caused by using traditional master data query systems to obtain data and then manually understand and analyze data.

[0034] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a system module architecture diagram of the present invention.

[0036] Figure 2 It is a flow chart of the system working steps of the present invention. DETAILED DESCRIPTION

[0037] like Figure 1 As shown, an intelligent master data query system based on a large model includes a master data acquisition module, a master data storage module, a general query module, an advanced query module, a query result display module, a permission management module, and a system monitoring and alarm module;

[0038] The specific functions and effects of each module are explained one by one below.

[0039] In this embodiment, the master data acquisition module is responsible for collecting master data from various data sources; these data sources mainly include various business system databases, message queues and log files. The master data acquisition module is also responsible for cleaning and filtering the collected master data to ensure that the system can obtain comprehensive and accurate master data, providing a basis for subsequent data query and analysis.

[0040] In this embodiment, the master data storage module is responsible for the storage and management of master data. The cleaned and filtered master data will be stored in the distributed data warehouse hive for backup to ensure data security and reliability. The master data in the distributed data warehouse hive will be synchronized to the data table corresponding to the distributed SQL query engine Trino for subsequent data query and analysis to improve the efficiency of data query. The master data storage module is the database of the master data query system.

[0041] It should be noted that in the main data storage module, Trino, a query engine with better query performance for large amounts of data, is used to replace the MySQL database used in the general traditional main data query system. In the traditional solution, in order to improve the query response speed, it is necessary to establish a cache layer on the basis of MySQL, which increases the burden of system operation and maintenance in the later stage, and there will be obvious performance bottlenecks when dealing with analytical aggregation queries of large amounts of data. In the present invention, Trino replaces MySQL as the main data storage solution, which improves the performance of the system, shortens the response time of returning query data, and provides a more friendly user experience.

[0042] In this embodiment, the general query module provides basic master data query functions. Users can enter query conditions through this module, and the system retrieves and returns corresponding data results in the Trino data table based on these conditions. The general query module supports a variety of query conditions, such as keywords, date ranges, numerical ranges, etc., to meet different query needs of users. The general query module is mainly used in scenarios where users want to directly obtain detailed data without further analysis and processing of the data.

[0043] In this embodiment, the advanced query module corresponds to the ordinary query module, providing a deeper, more efficient, more flexible and convenient main data query function. The advanced query module of the present invention is developed using the large model framework LangChain. The system must first import the necessary LangChain module, then connect to the main data storage module, initialize the OpenAI LLM instance, and then create a SQL Agent from LLM to the database. It can answer questions based on the architecture of the main data storage module and the content of the main data storage module (for example, it will retrieve the description information of a specific table). It also has error correction capabilities. When an error occurs when executing the generated query, it can capture the error and then correctly regenerate and execute a new query. LangChain uses the create_sql_agent function to initialize the agent. The SQL agent created by this function contains SQLDatabaseToolkit. This toolbox contains three tools: creating and executing queries, checking query syntax, and retrieving data table descriptions. With the assistance of these tools, SQL Agent can drive LLM to complete SQL query tasks. First, after thinking, it will determine that the first action is to use the tool sql_db_list_tables, then observe the table returned by the tool, and then after thinking, determine that the next action is sql_db_schema, that is, to create SQL statements, and move forward layer by layer until the answer is obtained. Finally, LLM will convert the answer into natural language and return it to the user.

[0044] In this embodiment, the query result display module is responsible for presenting the query results retrieved by the system in a user-friendly manner. It can select different display methods, such as tables, charts, reports, etc., according to the user's preferences and needs. At the same time, the module also supports operations such as exporting and printing the query results, so that users can use the query results for other purposes.

[0045] In this embodiment, the permission management module is used to control the user's access permissions and data operation permissions to the system. Through the permission management module, the system administrator can assign different roles and permissions to users to ensure that only authorized users can access and operate sensitive data. In addition, the permission management module also supports recording and auditing the operations of users to improve the security and traceability of the system.

[0046] In this embodiment, the system monitoring and alerting module is responsible for real-time monitoring of the running status and data quality of the master data query system. By detecting the performance metrics of the system (such as response time, throughput, etc.) and data quality metrics (such as data integrity, accuracy, etc.), and sending alerts in a timely manner when anomalies occur. Through the system monitoring and alerting module, the system administrator can discover and solve problems in a timely manner to ensure the stability and reliability of the system.

[0047] As Figure 2 shown, the working steps of the system of the present invention are as follows:

[0048] Step1. The user enters the master data query system interface, adds filtering conditions, and clicks the query button to initiate a query request.

[0049] Step2. If the user clicks the normal query button, the system receives the user's request, calls the backend interface to query the table data in the master data storage module, and returns the query result.

[0050] Step3. If the user clicks the advanced query button and asks a question in the pop-up dialog box, such as "How many customers are registered in Shanghai?" After the user clicks the submit button, the system receives the user's request and hands the request to the LLM.

[0051] Step4. LLM understands and translates: The LLM will first parse this question to understand the intention and required information behind it. Then, the model will generate the corresponding SQL query statement according to the parsed content, such as "SELECT SUM(sales) FROM customer_data WHERE regist_place = 'Shanghai';".

[0052] Step5. Execute the SQL query: The generated SQL query statement will be sent to the master data storage module for execution. The master data storage module processes this query and returns the required data result.

[0053] Step6. LLM receives and interprets the result: When the master data storage module returns the query result, the LLM will receive this data. Then, the LLM will start to parse this data and convert it into an answer format that is easier for humans to understand.

[0054] Step 7, provide answers: Finally, LLM converts the results into natural language answers and returns them to the user. For example, "There are 88 customers whose company is registered in Shanghai."

[0055] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. An intelligent master data query system based on a large model, characterized in that: It includes main data collection module, main data storage module, advanced query module and query result display module; The master data collection module is used to collect master data from various data sources; The master data storage module is used to store and manage the master data collected by the master data collection module; The advanced query module is used to understand the user's query request through a language model based on deep learning technology, generate a corresponding query statement and send it to the main data storage module; It is also used to understand the query result returned by the main data storage module through the language model and generate a corresponding natural language answer as the query result; The query result display module is used to display the query results in an interface manner.

2. The intelligent master data query system based on a large model according to claim 1, characterized in that: It also includes a general query module, which pre-stores a number of query conditions and is used to generate a query statement according to the query conditions selected by the user and the query keywords input, and send the query statement to the main data storage module; It is also used to send the query result returned by the main data storage module to the query result display module for display.

3. The intelligent master data query system based on a large model according to claim 1, characterized in that: It also includes a rights management module for controlling user access rights to the system and data operation rights.

4. The intelligent master data query system based on a large model according to claim 1, characterized in that: It also includes a system monitoring and alarm module for real-time monitoring of the operating status and data quality of the master data query system.

5. The intelligent master data query system based on a large model according to claim 1 is characterized in that: The master data collection module is also used to clean and filter the collected master data.

6. The intelligent master data query system based on a large model according to claim 5, characterized in that: The master data storage module is used to store the cleaned and filtered master data in the distributed data warehouse hive for backup, and synchronize the master data in the distributed data warehouse hive to the data table corresponding to the distributed SQL query engine Trino.

7. The intelligent master data query system based on a large model according to claim 1 is characterized in that: The advanced query module uses an LLM model as the language model based on deep learning technology.

8. The intelligent master data query system based on a large model according to claim 1, characterized in that: The query result display module supports displaying the query results in the form of tables, charts and / or reports.

Images