Intelligent model generation and interaction system

Abstract

This disclosure relates to an intelligent model generation and interaction system, which runs on a server and includes: a data source management module, which includes a data dictionary, a knowledge domain dictionary, and a data source module; a data processing and analysis module, which includes a data processing channel, a storage unit, and a virtual data access layer; a machine learning development module, which includes a data acquisition and management module, a model development module, a model evaluation module, a model operation and maintenance module, and a model analysis and retraining module; and a retrieval enhancement and generation module, which includes a prompting engineering module, a vector module, a retrieval module, a large language model management module, and a retrieval and generation feedback module. This intelligent model generation and interaction system can communicate and connect with multiple large language models, generating models and query results respectively based on the data to be analyzed provided by the user and the user query.

Description

[Technical Field]

[0001] This disclosure relates to an intelligent model generation and interaction system. Specifically, this disclosure, through a pre-established data processing and analysis module, can accept user queries in natural language and data to be analyzed, thereby generating data analysis and model generation to achieve human-computer interaction. [Previous Technology]

[0002] In the field of machine learning, training involves inputting a large dataset or data set into a model composed of specific functions. By adjusting parameters, the model learns from the data, resulting in a mathematical model that fits and describes the characteristics of the dataset. Given the rapid development of this field, although many open-source function libraries are available, procedural languages ​​are still required to create the relevant functions to accurately execute model training, depending on the different types of training sets or data sources. Furthermore, the degree of feature standardization and normalization of the data input into the model is a crucial factor affecting the training effect, posing a significant challenge for users unfamiliar with complex algorithms or programming languages.

[0003] Furthermore, as a large-scale deep learning model that utilizes massive amounts of data for pre-training, the large language model (LLM) is applicable to various natural language processing tasks, including text generation, machine translation, text summarization, question answering, and building chatbots that can converse with real people. However, since LLM also requires pre-training with massive amounts of data, although it demonstrates excellent capabilities in retrieving and generating human-like text, it is also prone to inheriting and amplifying errors and biases present in the training data, leading to systematic biases.

[0004] In view of this, there is an urgent need in the field for an integrated system that integrates and manages various data sources, laying the foundation for subsequent model generation and interaction, so that users can perform data analysis and model generation in the form of natural language. [Summary of the Invention]

[0005] The present invention is intended to provide a simplified summary of this disclosure so that the reader can have a basic understanding of it. This present invention is not a complete overview of this disclosure, and it is not intended to identify important / critical elements of the embodiments of the invention or to define the scope of the invention.

[0006] This disclosure relates to an intelligent model generation and interaction system, which runs on a server and communicates with a plurality of large language models (LLMs). It generates a model and a query result based on data to be analyzed provided by a user and a user query, wherein the user query is in natural language. The intelligent model generation and interaction system includes a data source management module, a data processing and analysis module, a machine learning development module, and a retrieval-augmented generation (RAG) module. The RAG module is configured to communicate with the plurality of large language models and can enhance the generated response through retrieved information.

[0007] The data source management module included in the intelligent model generation and interaction system of this disclosure includes a data dictionary for storing multiple metadata; a knowledge domain dictionary for storing multiple technical term definitions of a knowledge domain; and a data source module for accepting an external data source of the knowledge domain, wherein the external data source includes a structured data and an unstructured data.

[0008] The data processing and analysis module included in the intelligent model generation and interaction system disclosed herein is communicatively connected to the data source management module. It includes a data processing channel, which is communicatively connected to the data source module, for processing the structured data and the unstructured data, thereby converting them into standardized data; a storage unit, for storing a common data model (CDM), wherein the common data model is constructed by the standardized data, the plurality of metadata and the plurality of technical terminology definitions; and a virtual data access layer, which is communicatively connected to the storage unit.

[0009] The intelligent model generation and interaction system disclosed herein includes a machine learning development module and a data processing and analysis module connected in communication. The system comprises a data acquisition and management module for acquiring and preprocessing the user's data to be analyzed to form processed data, and obtaining the Common Data Model (CDM) through the virtual data access layer; a model development module, connected in communication with the data acquisition and management module, for training and generating the model based on the processed data and the Common Data Model; a model evaluation module, connected in communication with the model development module, for evaluating the performance of the model; a model operation and maintenance module, connected in communication with the model evaluation module, for managing, publishing, and maintaining the operation of the model; and a model analysis and retraining module, connected in communication with the model operation and maintenance module, for detecting the performance of the model and generating an updated model.

[0010] The retrieval-augmented generation (RAG) module included in the intelligent model generation and interaction system disclosed herein is designed to communicate with multiple large language models and the data processing and analysis module. The RAG module specifically includes: a prompt engineering module for constructing a prompt based on the user query; a vector module, communicating with the prompt engineering module, for converting the user query into multiple vectors, converting the external data source into multiple reference vectors, and obtaining the similarity between each of the multiple vectors and one of the multiple reference vectors; a retrieval module, communicating with the vector module, for integrating the external data source based on the multiple reference vectors to generate an integrated external data source, and for retrieving from the integrated external data source based on the similarity to generate a retrieval result, wherein the retrieval result includes a text fragment matching the prompt; a large language model (LLM) management module, communicating with the retrieval module, for selecting a selected large language model based on the prompt, the user query, and the text fragment to generate a query result; and a retrieval and feedback generation module, communicating with the retrieval module and the large language model management module, for evaluating the retrieval result, recording the user query process, and optimizing the query result accordingly.

[0011] According to the embodiments of the present disclosure, in the intelligent model generation and interaction system of the present disclosure, the machine learning development module further includes a data version control module, which is connected to the data acquisition and management module to mark different versions of the processed data and record any changes to the plurality of metadata.

[0012] According to the embodiments of the present disclosure, in the intelligent model generation and interaction system of the present disclosure, the model development module includes a feature engineering and algorithm selection module, which is used to perform feature engineering on the processed data and the common data model to extract a plurality of features, and select an algorithm based on the plurality of features; and an experiment management and optimization module, which is communicatively connected to the feature engineering and algorithm selection module, for training and optimizing the processed data and the common data model based on the algorithm, thereby generating the model.

[0013] According to the embodiments of the present disclosure, in the intelligent model generation and interaction system of the present disclosure, the model evaluation module includes an evaluation index module for evaluating a plurality of indicators of the model to generate a plurality of model evaluation results; and a model interpretation module, which is communicatively connected to the evaluation index module for generating a model interpretation result based on the plurality of model evaluation results, wherein the performance is determined based on the model interpretation result.

[0014] According to the embodiments of the present disclosure, in the intelligent model generation and interaction system of the present disclosure, the model operation and maintenance module includes a model publishing and deployment module, which is used to package and publish the model whose performance has been evaluated by the model evaluation module to a runtime environment; and a model life cycle management module, which is communicatively connected to the model publishing and deployment module, for monitoring the performance of the model in the runtime environment.

[0015] According to the embodiments of the present disclosure, in the intelligent model generation and interaction system of the present disclosure, the model analysis and retraining module includes a model output analysis module for analyzing and detecting the model to generate an abnormal result; and a model retraining module, which is communicatively connected to the model output analysis module to generate a retraining instruction to the model development module based on the abnormal result, thereby retraining the model and generating an updated model.

[0016] According to the embodiments of the present disclosure, in the intelligent model generation and interaction system of the present disclosure, the prompt engineering module includes a prompt design module for designing a prompt template based on the user query; and a prompt extension module, which is communicatively connected to the prompt design module for generating an extended prompt based on the user query and the prompt template, thereby constructing the prompt.

[0017] According to the embodiments of the present disclosure, in the intelligent model generation and interaction system of the present disclosure, the vector module includes a text vectorization module for converting the user query into the plurality of vectors and the external data source into the plurality of reference vectors; a vector database, which is communicatively connected to the text vectorization module for storing the plurality of reference vectors; and a vector similarity module, which is communicatively connected to the text vectorization module and the vector database for calculating or obtaining the similarity between each of the plurality of vectors and each of the plurality of reference vectors.

[0018] According to the embodiments of the present disclosure, in the intelligent model generation and interaction system of the present disclosure, the retrieval module includes a knowledge source integration module, which is used to standardize the external data source according to the plurality of reference vectors and integrate the multiple external data sources to generate the integrated external data source; and an enhanced retrieval module, which is communicatively connected to the knowledge source integration module, for retrieving the text fragment that matches the prompt from the integrated external data source according to the similarity, and extracting a contextual message of the text fragment.

[0019] According to the embodiments of the present disclosure, in the intelligent model generation and interaction system of the present disclosure, the large language model management module includes a model switching module for selecting the aforementioned selected large language model based on the prompt, the user query and the text fragment; and a model adaptation module, which is communicatively connected to the model switching module for fine-tuning and adapting the selected large language model to produce the query result.

[0020] According to the embodiments of the present disclosure, in the intelligent model generation and interaction system of the present disclosure, the retrieval and generation feedback module includes a process memory module for recording and managing the process of the user's query and adding context to the user's query based on the process; a retrieval and generation quality assessment module for evaluating the retrieval results and the query results to obtain a quality assessment result; and an output feedback module, which is communicatively connected to the retrieval and generation quality assessment module, for analyzing the query results and generating an optimization instruction to the prompt engineering module, the retrieval module and / or the large language model management module based on feedback from the user, thereby optimizing the query results.

[0021] According to a preferred or alternative embodiment of the present disclosure, the retrieval enhancement generation module in the intelligent model generation and interaction system of the present disclosure further includes a token calculation module for calculating the number of tokens interacting with the large language model.

[0022] After reading the following embodiments, those skilled in the art will be able to easily understand the basic spirit and other inventive objectives of the present invention, as well as the technical means and implementation methods adopted by the present invention.

Implementation Method

[0030] To make the description of this disclosure more detailed and complete, illustrative descriptions of embodiments and specific examples of the present invention are provided below; however, this is not the only form of implementing or utilizing the specific examples of the present invention. The embodiments cover features of multiple specific examples and methods and procedures for constructing and operating these specific examples, as well as their order. However, other specific examples may also be used to achieve the same or equivalent functions and procedural sequences.

[0031] I Definition

[0032] For convenience, specific proper nouns used in this specification, embodiments, and appended claims are concentrated herein. Unless otherwise defined in this specification, the scientific and technical terms used herein have the same meaning as understood and commonly used by one of ordinary skill in the art to which this invention pertains. Furthermore, unless conflicting with the context, the singular nouns used herein include their plural forms, and vice versa. Specifically, in this specification and the claims, the singular form "a" (a and an) includes the plural reference value, unless otherwise indicated by the context. In addition, in this specification and the claims, the expressions "at least one" and "one or more" have the same meaning, both representing a total of one, two, three, or more.

[0033] The term "data" or "information" is used interchangeably in this disclosure. Both terms refer to numerical characteristics or information of a plurality of objective facts, whether observed, generated, collected, or stored somewhere, whether processed or unprocessed, including qualitative or quantitative variables relating to a particular object (not limited to people, things, or objects). Unless otherwise stated, "data" or "information" have the same meaning in this disclosure.

[0034] The "data to be analyzed" used in this disclosure refers to a set of data owned by a user who intends to use the intelligent model generation and interaction system of this disclosure for analysis and processing to seek correlations or obtain integrated information. The "data to be analyzed" in this disclosure includes both structured and unstructured data, as well as data in various forms or formats, such as voice, images, text, pictures, or programming languages. According to the specific implementation of this disclosure, when the data to be analyzed is unstructured data (e.g., natural language), it can be converted into structured data using conventional technical means in the art to facilitate its use by the intelligent model generation and interaction system of this disclosure.

[0035] This disclosure uses the terms "data source" or "knowledge source" interchangeably, referring to a data source, specifically including a method for establishing connections with other systems, files, and / or databases (databases) to extract data. Generally, a data source stores all information for establishing database connections; by providing the correct data source name, the corresponding database can be found and connected. "External data source" refers to data from different specific sources and different knowledge domains stored outside the intelligent model generation and interaction system of this disclosure. That is, data stored in external databases (including data stored on hardware, in the cloud, or on the blockchain) can be introduced into the intelligent model generation and interaction system of this disclosure through interconnection technologies known in the art (such as application interfaces) for integration and use.

[0036] The term "model" used in this disclosure refers to a statistically significant model built based on the data to be analyzed or external data, intended to establish an objective basis for a computer system to execute operations on that data. According to the implementation method of this disclosure, the model is trained and built through machine learning algorithms and / or machine learning modules. Models applicable to this disclosure include, but are not limited to, deep learning models, predictive models, and classification models.

[0037] In this disclosure, a "server" is a computer system used to provide services, resources, or functions to other computers or devices, and to run specific software through computing power to provide services. Unless otherwise specified, the term "server" as used in this disclosure covers both ground-based and cloud-based servers. Typically, a server has at least some form of storage unit and communication unit. The storage unit includes electrically and non-electrically dependent, removable and non-removable media, which can be used by appropriate methods or technologies to store desired information (such as computer-readable instructions, data structures, application modules, and other data). Storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), electrically-erasable programmable read-only memory (EEPROM), flash memory, or other memory technologies, or any other media capable of storing required information and accessible to the processor. Generally, a communication unit can materialize computer-readable instructions, data, structures, application modules, and other data into various data signals, and can transmit them through any communication medium. By way of example and not limitation, communication media include wired media (such as wired networks or direct wired connections) and wireless media (such as radio waves, e.g., Wi-Fi or Bluetooth). In one embodiment of this disclosure, the intelligent model generation and interaction system operates within a server and communicates with at least one external client interface to analyze data and generate models and / or query results by having a user from that client interface query and upload data to be analyzed.

[0038] The term "performance" used in this disclosure is used broadly to refer to the effectiveness and efficiency of the models generated by the intelligent model generation and interaction system of this disclosure, and is presented in several quantitative indicators. Since the goal of machine learning models is to make accurate predictions on new data, quantitative indicators can be used to evaluate the effectiveness of learning and to specifically quantify the accuracy, stability, and generalization ability of the model in different contexts, ensuring the reliability and effectiveness of the model in practical applications. Specifically, model effectiveness includes indicators that measure the model's prediction accuracy and prediction quality, including but not limited to accuracy, precision, recall, area under the receiver operating characteristic curve (ROC-AUC), confusion matrix, and loss function. Furthermore, model efficiency includes indicators that measure the model's resource usage and running speed, such as training time, inference time, and memory usage.

[0039] II Detailed Implementation

[0040] In order to lower the barrier for general non-programmers to use model generation systems and data analysis, this disclosure integrates pre-set and defined data dictionaries, domain dictionaries, data sources and automated data processing pipelines to smoothly and quickly eliminate the differences between unstructured data (e.g., natural language) and structured data, thereby realizing natural language understanding, text retrieval and model generation.

[0041] This disclosure aims to provide an intelligent model generation and interaction system that runs on a server and communicates with multiple large language models (LLMs) to generate a model and a query result based on data to be analyzed provided by a user and a user query. According to the embodiments of this disclosure, the user query can include conversational natural language or natural language data in text format. In a specific embodiment, when using the intelligent model generation and interaction system of this disclosure, the user can input or upload data to be analyzed to the system through a user query in natural language format and data provided by the user, thereby enabling the system to generate a model and / or query results.

[0042] Schematic diagrams of the intelligent model generation and interaction system 1 according to one embodiment of the present disclosure are shown in Figures 1A and 1C. Specifically, Figure 1A shows an overall schematic diagram of the intelligent model generation and interaction system 1 of the present disclosure, Figure 1B shows a schematic diagram of the intelligent model generation and interaction system 1 of the present disclosure generating a model through the interaction of the machine learning development module 15 and the data processing and analysis module 13, and Figure 1C shows a schematic diagram of the intelligent model generation and interaction system 1 of the present disclosure generating query results through the interaction of the retrieval enhancement generation module 17 and the data processing and analysis module 13.

[0043] First, please refer to Figure 1A. The intelligent model generation and interaction system 1 operates within a server and includes a data source management module 11, a data processing and analysis module 13, a machine learning development module 15, and a retrieval enhancement generation module 17. The data processing and analysis module 13 is communicatively connected to the data source management module 11; the machine learning development module 15 is communicatively connected to the data processing and analysis module 13; and the retrieval enhancement generation module 17 is communicatively connected to the aforementioned plurality of large language models. Note that only three large language models L1, L2, and L3 are illustrated in the figure as examples, but the actual implementation is not limited to these; multiple large language models can be connected using conventional serialization methods in the art.

[0044] The data source management module 11 of this disclosure is used to integrate and manage various data sources required by the intelligent model generation and interaction system 1 of this disclosure. According to one embodiment, the data management module 11 includes a data dictionary 112, a knowledge domain dictionary 114, and a data source module 116. The data dictionary 112 defines and stores a plurality of metadata. Specifically, each metadata contains the structure of each data item, describing data about a specific set of data. In this disclosure, the data dictionary 112 is used to define and record detailed information about the data entities, attributes, and relationships of the intelligent model generation and interaction system 1. The knowledge domain dictionary 114 specifically stores a dictionary of a plurality of professional terms and / or concepts for a specific knowledge domain or industry, and captures and defines the proprietary terms, concepts, and knowledge of that domain. On the other hand, the data source module 116 of this disclosure is used to aggregate at least one external data source, and the at least one external data source includes structured data and unstructured data. Generally, external data, which includes a collection of numerous knowledge sources, is gathered into the intelligent model generation and interaction system 1 of this disclosure through techniques or means known in the art. For example, these external data sources are obtained and integrated by connecting to a specific database, by connecting through an application programming interface (API), or by extracting open data.

[0045] According to the embodiments of this disclosure, the technical fields to which the intelligent model generation and interaction system 1 of this disclosure can be applied include, but are not limited to, the medical field and the financial field. Users can collect and compile knowledge sources of specific related technical fields according to actual needs, and establish a data dictionary 112 and a knowledge domain dictionary 114 in advance. According to a specific embodiment of this disclosure, when the specific knowledge domain is a medical-related field, the external data source includes medical and health care data, which records specific values ​​such as an individual's height, weight, and age. At this time, the data dictionary 112 defines the fields recorded in the medical and health care data that have "height," "weight," and "age," and stores the fields in the data dictionary 112, rather than storing the specific values ​​of these fields; the knowledge domain dictionary 114 includes definitions of multiple professional terms such as medical terms, disease names, symptom descriptions, and examination items.

[0046] Referring again to Figure 1A, according to the embodiments disclosed herein, the data processing and analysis module 13 is communicatively connected to the data source management module 11 and includes a data processing channel 132, a storage unit 134, and a virtual data access layer 136. The data processing channel 132 is communicatively connected to the data source module 116, processing the received structured and unstructured data to convert it into standardized data. Specifically, the data processing channel 132 is used to automatically execute an end-to-end process from the original source of data to its processed and ready state. Conventional tools or techniques in the art can be used to extract, load, and transform the received data, thereby processing the original state of the data (including structured and unstructured data) through various data cleaning, standardization, or merging processes to generate standardized data for subsequent use by the intelligent model generation and interaction system 1 of this invention.

[0047] When external data sources with the same knowledge domain are not obtained from a single data source, a common data model A specific to the external data source can be constructed using the standardized data generated via the aforementioned data processing channel 132, and based on the multiple metadata of the same knowledge domain stored in the aforementioned data dictionary 112 and the multiple professional term definitions stored in the knowledge domain dictionary 114. The common data model A can be stored in the storage unit 134 for subsequent use.

[0048] The virtual data access layer 136 is connected to the storage unit 134 so that other modules (such as the machine learning development module 15 and the retrieval enhancement generation module 17) can access the common data model A through the virtual data access layer 136 to obtain the basis for data analysis.

[0049] Please refer to Figure 1B, which illustrates the interaction between the machine learning development module 15 and the data processing and analysis module 13 to generate a model. The machine learning development module 15 and the data processing and analysis module 13 are communicatively connected, specifically including a data acquisition and management module 151, a model development module 152, a model evaluation module 153, a model operation and maintenance module 154, and a model analysis and retraining module 155.

[0050] According to the embodiments disclosed herein, when the intelligent model generation and interaction system 1 receives data to be analyzed provided by a user, the data acquisition and management module 151 acquires and preprocesses the user's data to form processed data. Simultaneously, the data acquisition and management module 151 communicates with the virtual data access layer 136 of the data processing and analysis module 13, and obtains the common data model A through the virtual data access layer 136. Next, the model development module 152, communicating with the data acquisition and management module 151, trains and generates a model corresponding to the data to be analyzed based on the aforementioned processed data and the common data model A obtained from the data processing and analysis module 13.

[0051] According to the embodiments of this disclosure, after the model is generated, the model evaluation module 153, which is communicatively connected to the model development module 152, is configured to evaluate the performance of the model. According to specific embodiments, model performance includes model effects, such as accuracy, precision, recall, area under the receiver operating characteristic curve (ROC-AUC), confusion matrix, or loss function; and / or model efficiency, including but not limited to training time, inference time, and memory usage.

[0052] Next, the model operation and maintenance module 154 communicates with the model evaluation module 153 and the model analysis and retraining module 155 to manage the model, publish the model to a specific operating environment (e.g., for users to download), and maintain the operation of the model. Through the model analysis and retraining module 155, the performance of the generated model can be detected and an updated model can be generated.

[0053] Next, please refer to Figure 1C, which is a schematic diagram illustrating how the retrieval enhancement generation module 17 generates query results through interaction with the data processing and analysis module 13. According to the embodiments of this disclosure, the retrieval enhancement generation module 17 is communicatively connected to both the large language model (L1 to L3) and the data processing and analysis module 13. Specifically, the retrieval enhancement generation module 17 includes a prompting engineering module 171, a vector module 172, a retrieval module 173, a large language model (LLM) management module 174, and a retrieval and generation feedback module 175. Among them, the vector module 172 is communicatively connected to the prompting engineering module 171 and the retrieval module 173, the large language model management module 174 is communicatively connected to the retrieval module 173, and the retrieval and generation feedback module 175 is communicatively connected to both the large language model management module 174 and the retrieval module 173.

[0054] According to an embodiment of this disclosure, when the intelligent model generation and interaction system 1 receives a user query from a user, the prompt engineering module 171 first constructs a prompt based on the user query. According to a specific embodiment of this disclosure, a prompt refers to the text content to be input into a large language model, used to guide the large language model to produce query results corresponding to the user query. The quality of the prompt is crucial to the quality of the output. After the prompt engineering module 171 constructs the prompt, the vector module 172 converts the user query into a plurality of vectors and the external data source into a plurality of reference vectors, thereby obtaining a similarity between each vector and these reference vectors. According to a preferred or alternative embodiment of this disclosure, a common data model A established based on the external data source is obtained through the virtual data access layer 136. In the case of multiple external data sources, each of these external data sources generates its corresponding reference vector through the transformation by the vector module 172. At this point, the retrieval module 173 integrates these different external data sources based on the aforementioned reference vectors, thereby generating an integrated external data source to facilitate subsequent retrieval and large-scale language text generation. In addition, the retrieval module 173 also retrieves data from this integrated external data source based on the aforementioned similarity, to generate a retrieval result. According to a specific embodiment of this disclosure, the retrieval result includes text fragments matching the prompts constructed by the aforementioned prompt engineering module 171. The large-scale language model management module 174 and at least one large-scale language model L can select a suitable large-scale language model L based on the aforementioned prompts, user queries, and the aforementioned text fragments, thereby generating a query result. According to a specific embodiment of this disclosure, the query result essentially includes the aforementioned text fragments and the contextual information of those text fragments, forming a complete answer, that is, the result expected by the user through the user query. According to the specific implementation of this disclosure, in order to better improve the user experience of the intelligent model generation and interaction system 1, the retrieval and generation feedback module 175 is communicatively connected with the large language model management module 174 and the retrieval module 173 to evaluate the aforementioned retrieval results, record the process of the user query received by the intelligent model generation and interaction system 1, and optimize the query results accordingly.

[0055] Please see Figure 2. Figure 2 is a schematic diagram illustrating the specific operation of the machine learning development module 15 according to one embodiment of this disclosure.

[0056] According to a preferred or additional embodiment, the machine learning development module 15 disclosed herein further includes a data version control module 156, which is communicatively connected to the data acquisition and management module 151, for marking different versions of the aforementioned processed data and recording any changes to the plurality of metadata.

[0057] Referring also to Figure 2, the model development module 152 disclosed herein specifically includes a feature engineering and algorithm selection module 1521 and an experiment management and optimization module 1523, which are interconnected. According to the preferred or specific implementation of this disclosure, after the data acquisition and management module 151 obtains the processed data and common data model A, the feature engineering and algorithm selection module 1521 first performs feature engineering on the processed data and common data model A to extract a plurality of features, and then selects a suitable algorithm based on the plurality of features. Next, the experiment management and optimization module 1523 trains and optimizes the processed data and common data model A based on the algorithm, thereby generating a model corresponding to the data to be analyzed.

[0058] Referring again to Figure 2, the model evaluation module 153 disclosed herein is used to evaluate the performance of the model produced by the aforementioned model development module 152. The model evaluation module 153 includes an evaluation metric module 1531 and a model interpretation module 1533 that are interconnected. The evaluation metric module 1531 evaluates a plurality of metrics of the model to generate a plurality of model evaluation results. According to the specific implementation of this disclosure, the model evaluation results include various commonly used metrics for evaluating the performance of machine learning models, such as commonly used metrics for evaluating binary classification learning tasks, such as accuracy, precision, recall, and F1 score, or metrics for evaluating regression learning tasks, such as root mean square error (RMSE) or mean absolute percentage error (MAPE). After obtaining a plurality of model evaluation results, the model interpretation module 1533 can interpret the model to generate a model interpretation result. Specifically, through various conventional interpretation techniques, such as Shapley values, Local Interpretable Model-Agnostic Explanations (LIME), or Integrated Gradients (IG), the model interpretation module can present the marginal impact of features extracted through feature engineering on model predictions, identifying key features of interest to the model. Depending on optional or alternative implementations, the model interpretation module can also provide visualization tools, such as saliency maps, decision trees, or feature importance plots, to intuitively present the model's decision path for system users or developers to understand. The model interpretation results produced by the model interpretation module 1533 can be used to evaluate the performance of the generated model, thereby providing a basis for system developers or users to subsequently adjust the model or retrain it.

[0059] The model operation and maintenance module 154 disclosed herein includes a model publishing and deployment module 1541 and a model lifecycle and lifecycle management module 1543, which are interconnected. The model publishing and deployment module 1541 is designed to package the previously evaluated model using containerization technology and deploy it to an appropriate endpoint for execution in its runtime environment (or execution environment). The model lifecycle and lifecycle management module 1543 monitors the performance of the model in this runtime environment. According to specific implementations, the intelligent model generation and interaction system 1 disclosed herein can package the generated model using containerization technology and deploy it to an appropriate runtime environment for execution in online or local systems; in addition, the model can also be uploaded to the cloud for remote execution and management as needed.

[0060] To ensure the output quality and reliability of the models generated by the intelligent model generation and interaction system 1 of this invention, the model analysis and retraining module 155 disclosed herein includes a model output analysis module 1551 and a model retraining module 1553, which analyze the model and retrain it based on the analysis results. Specifically, the model output analysis module 1551 detects and analyzes whether the aforementioned model has any bias or anomaly, especially whether there are outliers, biases, or drifts in the model performance (e.g., quantitative indicators of model effect and model efficiency) caused by problems such as data drift, negative feedback loops, and model inaccuracy, and generates abnormal results accordingly. The model retraining module 1543 then generates an instruction to the model development module 152 based on the aforementioned abnormal results, specifically a retraining instruction. The model development module 152 accepts this retraining instruction, restarts the model training process, and finally generates an updated model. According to embodiments of this disclosure, the retraining instruction includes instructions that can adjust the training process, such as reselecting the algorithm, adjusting hyperparameters, or supplementing with additional training data, to maximize the performance of the updated model (as described above regarding model effectiveness and efficiency). According to some additional or alternative embodiments of this disclosure, the model retraining module 1543 not only receives abnormal results from the model output analysis module 1541, but also collects new data samples from the online environment and combines them with the aforementioned analysis results to correct model biases and improve adaptability.

[0061] According to a specific embodiment of this disclosure, the retrieval enhancement generation module 17 of the intelligent model generation and interaction system 1 of the present invention receives user queries from users and generates query results. Please see Figure 3, which is a schematic diagram illustrating the specific operation of the retrieval enhancement generation module 17 according to an embodiment of this disclosure. In an additional or preferred embodiment, the prompt engineering module 171 of the retrieval enhancement generation module 17 includes a prompt design module 1711 and a prompt expansion module 1713 that are communicatively connected to each other. In this embodiment, when the intelligent model generation and interaction system 1 receives a user query from a user, the prompt design module 1711 designs a prompt template based on the user query to facilitate subsequent interaction with large language models. According to an additional or optional embodiment, the prompt design module 1711 can also be used to analyze prompts, evaluate and diagnose the quality of prompts, and improve the design of prompt templates. Then, the prompt expansion module 1713 is used to generate an expanded prompt based on the user query and the prompt template, thereby constructing a prompt that can be used for interaction with large language models. Specifically, the prompt expansion module 1713 can identify and, based on specific context and semantics, retrieve relevant background information and context from a specific database, integrate this supplementary information into the content of the original prompt template, and finally construct a more comprehensive and richer contextual prompt, thereby improving the accuracy of subsequent large-scale language model interactions.

[0062] Referring again to Figure 3, according to an embodiment of this disclosure, the vector module 172 includes a text vectorization module 1721, a vector similarity module 1723, and a vector database V. The vector database V is communicatively connected to both the text vectorization module 1721 and the vector similarity module 1723, and the vector similarity module 1723 is also communicatively connected to the text vectorization module 1721. According to one embodiment of this disclosure, the text vectorization module 1721 converts external data sources into a plurality of reference vectors, which are pre-stored in the vector database V. When a user query is received, the user query is converted into a plurality of vectors. The vector similarity module 1723 is used to calculate and compare the similarity between the plurality of vectors and the plurality of reference vectors. Vector similarity can be obtained using conventional tools or calculation methods in the field. The calculation methods applicable to the vector similarity module of this disclosure include, but are not limited to, cosine similarity, Euclidean distance, Jaccard similarity, Manhattan distance, and Pearson correlation coefficient.

[0063] The content retrieval module 173 disclosed herein includes a knowledge source integration module 1731 and an enhanced retrieval module 1733. The knowledge source integration module 1731 standardizes external data sources based on a plurality of reference vectors stored in the vector database V. Specifically, external data sources include various structured data (such as knowledge graphs and relational databases) and unstructured data (such as text). The knowledge source integration module 1731 can integrate more than one external data source, but these heterogeneous data sources must be preprocessed, cleaned, and standardized to achieve seamless integration. Therefore, through integrating multiple external data sources, the knowledge source integration module 1731 generates an integrated external data source. Then, the enhanced retrieval module 1733 communicates with the knowledge source integration module 1731 to retrieve text fragments from the integrated external data source that match the prompts constructed by the prompt engineering module 171 based on the similarity calculated from the vector similarity module 1723, and also extracts contextual information from the text fragments to form retrieval results.

[0064] According to the embodiments of this disclosure, the large language model management module 174 includes a model switching module 1741 and a model adaptation module 1743, which are communicatively connected to each other. The model switching module 1741 is used to select a suitable large language model based on prompts, user queries, and text fragments. As mentioned above, the intelligent model generation and interaction system 1 of this disclosure is connected to multiple large language models L, so the model switching module 1741 is mainly used to switch between multiple large language models (e.g., L1 to L3 shown in Figure 3) and select the most suitable large language model according to predefined rules or strategies, such as determining the most suitable large language model based on task type, query topic, and data volume. In addition, the model adaptation module 1743 is used to fine-tune the large language model according to specific rules or strategies to facilitate the large language model's adaptation to the performance in certain specific scenarios, such as fine-tuning for user queries in a specific knowledge domain. After fine-tuning and adaptation by the model adaptation module 1743, query results that meet the user query are produced.

[0065] To ensure that the query results generated by the intelligent model generation and interaction system 1 of this disclosure more accurately meet the user's needs, the content retrieval enhancement generation module 17 of this disclosure can also optimize the query results according to the user's query process. According to one embodiment of this disclosure, the retrieval and generation feedback module 175 includes a process memory module 1751, a retrieval and generation quality evaluation module 1753, and an output feedback module 1755. The output feedback module 1755 is communicatively connected to both the process memory module 1751 and the retrieval and generation quality evaluation module 1753. When the intelligent model generation and interaction system 1 receives a user query, the process memory module 1751 records and manages all user query processes, and adds context to the user query based on these processes, thereby improving the coherence and consistency of the large language model's response. The retrieval and generation quality assessment module 1753 evaluates the retrieval results (specifically including text fragments matching the prompts) obtained by the retrieval module 173 and the query results obtained by the large language model management module 174, resulting in a quality assessment result. Specifically, the retrieval and generation quality assessment module 1753 can design evaluation criteria and datasets based on user-preset scenarios. According to a specific implementation, the quality assessment result includes a score based on relevant indicators (such as relevance, fluency, and consistency). This function aims to identify potential errors or deficiencies, ensuring the reliability and accuracy of the content generated during the retrieval enhancement process. Finally, the query results are optimized through the output feedback module 1755. Specifically, the output feedback module 1755 analyzes the aforementioned quality assessment results and can also accept feedback from the user. Based on the quality assessment results and user feedback or opinions, the output feedback module 1755 generates optimization instructions to the prompt engineering module 171, the retrieval module 173, and / or the large language model management module 174, thereby enabling these modules to readjust or adjust their relevant functions and optimize the query results.

[0066] In an alternative or additional implementation, the content retrieval enhancement generation module 17 of this disclosure will generate a set of workflows, which will be packaged in a containerized manner and deployed to appropriate endpoints for execution in their runtime environment (or execution environment).

[0067] Unless necessary or additional, the content retrieval enhancement generation module 17 may also include a token calculation module 176 for calculating the number of tokens that interact with each large language model.

[0068] In summary, this disclosure provides an intelligent model generation and interaction system. By managing data sources and integrating predefined data dictionaries, domain dictionaries, and automated data processing pipelines, it constructs a system that can generate corresponding query results based on user queries in natural language (conversational or text-based). It can also generate corresponding machine learning models for the data to be analyzed (including structured and unstructured data) without the need for complex algorithms and programming languages, thereby lowering the barrier for non-programming language proficiency users to use large-scale natural language models and machine learning models. Therefore, this intelligent model generation and interaction system can achieve "intelligent" generation and analysis results for specific knowledge domains using only natural language, providing users with a more accurate, faster, and simpler data analysis experience. Furthermore, this intelligent model generation and interaction system can continuously improve its performance through adjustments, feedback, and retraining, fully realizing human-computer interaction and significantly enhancing the user experience.

[0069] It should be understood that the foregoing description of the embodiments is merely given by way of example, and various modifications can be made by those skilled in the art. The above specification, embodiments, and experimental results provide a complete description of the structure and use of exemplary embodiments of the present invention. Although various specific embodiments of the present invention are disclosed in the above embodiments, they are not intended to limit the present invention. Those skilled in the art can make various modifications and alterations without departing from the principles and spirit of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims. [Simplified Explanation of the Diagram]

[0023] To make the above and other objects, features, advantages and embodiments of the present invention more apparent and understandable, the accompanying drawings are described below:

[0024] Figure 1A is a schematic diagram of the intelligent model generation and interaction system 1 illustrated according to an embodiment of the present disclosure;

[0025] Figure 1B is a schematic diagram illustrating, according to an embodiment of the present disclosure, that the intelligent model generation and interaction system 1 generates a model through the interaction of the machine learning development module 15 and the data processing and analysis module 13.

[0026] Figure 1C is a schematic diagram illustrating, according to an embodiment of the present disclosure, how the intelligent model generation and interaction system 1 generates query results through the interaction of the retrieval enhancement generation module 17 and the data processing and analysis module 13;

[0027] Figure 2 is a schematic diagram illustrating the specific operation of the machine learning development module 15 according to an embodiment of this disclosure; and

[0028] Figure 3 is a schematic diagram illustrating the specific operation of the search enhancement generation module 17 according to an embodiment of the present disclosure.

[0029] In accordance with conventional operating practices, the various features and elements in the figures are not drawn to scale. The way they are drawn is to present the specific features and elements related to the present invention in the best possible way. In addition, similar elements / parts are referred to by the same or similar element symbols in different figures.

Claims

1. An intelligent model generation and interaction system, running on a server and communicating with multiple large language models (LLMs), is used to generate a model and a query result based on data to be analyzed provided by a user and a user query, wherein the user query is in natural language. The intelligent model generation and interaction system comprises: a data source management module, including a data dictionary for storing multiple metadata; a knowledge domain dictionary for storing multiple definitions of technical terms in a knowledge domain; and a data source module for accepting an external data source from the knowledge domain, wherein the external data source includes structured data and unstructured data; a data processing and analysis module, communicating with the data source management module, including a data processing channel communicating with the data source module, for processing the structured data and the unstructured data, thereby converting them into standardized data; and a storage unit for storing a common data model. The common data model (CDM) is constructed using the standardized data, the plurality of metadata, and the plurality of technical terminology definitions. The system includes: a virtual data access layer communicating with the storage unit; a machine learning development module communicating with the data processing and analysis module, comprising a data acquisition and management module for acquiring and preprocessing the user's data to be analyzed to form processed data, and obtaining the common data model through the virtual data access layer; a model development module communicating with the data acquisition and management module for training and generating the model based on the processed data and the common data model; a model evaluation module communicating with the model development module for evaluating the performance of the model; a model operation and maintenance module communicating with the model evaluation module for managing, publishing, and maintaining the operation of the model; a model analysis and retraining module communicating with the model operation and maintenance module for detecting the performance of the model and generating an updated model; and a retrieval-augmented generation system. A generation (RAG) module, communicating with the plurality of large language models and the data processing and analysis module, includes a prompt engineering module for constructing a prompt based on the user query; a vector module, communicating with the prompt engineering module, for converting the user query into a plurality of vectors, converting the external data source into a plurality of reference vectors, and obtaining the similarity between each of the plurality of vectors and one of the plurality of reference vectors; a retrieval module, communicating with the vector module, for integrating the external data source based on the plurality of reference vectors to generate an integrated external data source, and for retrieving from the integrated external data source based on the similarity to generate a retrieval result, wherein the retrieval result includes a text fragment matching the prompt; and a large language model (LLM) management module, communicating with the retrieval module, for selecting a selected large language model based on the prompt, the user query, and the text fragment to produce a query result. And a retrieval and feedback generation module, which communicates with the large language model management module and the retrieval module, to evaluate the retrieval results and record the user's query process and optimize the query results accordingly.

2. The intelligent model generation and interaction system as described in claim 1, wherein the retrieval enhancement generation module further includes a token calculation module for calculating the number of tokens interacting with the large language models.

3. The intelligent model generation and interaction system as described in claim 1, wherein the machine learning development module further includes a data version control module, which is communicatively connected to the data acquisition and management module, for marking different versions of the processed data and recording any changes to the plurality of metadata.

4. The intelligent model generation and interaction system as described in claim 1, wherein the model development module includes a feature engineering and algorithm selection module for performing feature engineering on the processed data and the common data model to extract a plurality of features, and selecting an algorithm based on the plurality of features; and an experiment management and optimization module, communicatively connected to the feature engineering and algorithm selection module, for training and optimizing the processed data and the common data model based on the algorithm, thereby generating the model.

5. The intelligent model generation and interaction system as described in claim 1, wherein the model evaluation module includes an evaluation index module for evaluating a plurality of indicators of the model to generate a plurality of model evaluation results; and a model interpretation module, communicatively connected to the evaluation index module, for generating a model interpretation result based on the plurality of model evaluation results, wherein the performance is determined based on the model interpretation result.

6. The intelligent model generation and interaction system as described in claim 1, wherein the model operation and maintenance module includes a model publishing and deployment module for packaging and publishing the model whose performance has been evaluated by the model evaluation module to an operating environment; and a model lifecycle and periodic management module, which is communicatively connected to the model publishing and deployment module for monitoring the performance of the model in the operating environment.

7. The intelligent model generation and interaction system as described in claim 1, wherein the model analysis and retraining module includes a model output analysis module for analyzing and detecting the model to generate an abnormal result; and a model retraining module, communicatively connected to the model output analysis module, for generating a retraining instruction to the model development module based on the abnormal result, thereby retraining the model and generating an updated model.

8. The intelligent model generation and interaction system as described in claim 1, wherein the prompt engineering module includes a prompt design module for designing a prompt template based on the user query; and a prompt extension module, communicatively connected to the prompt design module, for generating an extended prompt based on the user query and the prompt template, thereby constructing the prompt.

9. The intelligent model generation and interaction system as described in claim 1, wherein the vector module includes a text vectorization module for converting the user query into the plurality of vectors and the external data source into the plurality of reference vectors; a vector database, communicatively connected to the text vectorization module, for storing the plurality of reference vectors; and a vector similarity module, communicatively connected to the text vectorization module and the vector database, for calculating or obtaining the similarity between each of the plurality of vectors and each of the plurality of reference vectors.

10. The intelligent model generation and interaction system as described in claim 1, wherein the retrieval module includes a knowledge source integration module for standardizing the external data source based on the plurality of reference vectors and integrating the multiple external data sources to generate the integrated external data source; and an enhanced retrieval module, communicatively connected to the knowledge source integration module, for retrieving the text fragment matching the prompt from the integrated external data source based on the similarity, and extracting a contextual message of the text fragment.

11. The intelligent model generation and interaction system as described in claim 1, wherein the large language model management module includes a model switching module for selecting the selected large language model based on the prompt, the user query, and the text fragment; and a model adaptation module, communicatively connected to the model switching module, for fine-tuning and adapting the selected large language model to produce the query result.

12. The intelligent model generation and interaction system as described in claim 1, wherein the retrieval and generation feedback module includes a process memory module for recording and managing the process of the user's query and adding context to the user's query based on the process; a retrieval and generation quality assessment module for evaluating the retrieval results and the query results to obtain a quality assessment result; and an output feedback module communicatively connected to the process memory module and the retrieval and generation quality assessment module for analyzing the quality assessment result and generating an optimization instruction to the prompt engineering module, the retrieval module, and / or the large language model management module based on feedback from the user, thereby optimizing the query results.

Images