A large model combination system across knowledge domains

Through a large-scale model combination system across the knowledge field, using multimodal large models of agents and chain structures, the existing large-scale models have solved the problem of high cost and high power consumption in cross-domain inference, and achieved low cost and low cost Power consumption, high accuracy cross-domain reasoning, supports flexible expansion and infinite processing in the knowledge field.

CN118863098BActive Publication Date: 2024-11-22HUAZHONG UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411332916.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-24
Publication Date
2024-11-22
Estimated Expiration
2044-09-24

AI Technical Summary

Technical Problem

Existing large models are costly, have high power consumption and poor applicability when reasoning across knowledge fields, making them difficult to deal with tasks in different fields, and their tasks in the same field are not as accurate as a single large model.

Method used

Adopt a large-model combination system across the knowledge field, including defining agents, selecting agents and inspecting agents, and flexibly scheduling and selecting large-modal models suitable for the field for reasoning by managing multimodal large-scale models with field keywords and chain structures. , realize cross-domain reasoning with low cost and low power consumption.

Benefits of technology

It improves the applicability and accuracy of the large model, reduces inference costs and power consumption, supports flexible expansion and unlimited knowledge field processing, and avoids the surge in power consumption and forgetting problems caused by the increase in knowledge field.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118863098B_ABST
    Figure CN118863098B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of artificial intelligence related technology, and discloses a large model combination system across knowledge domains, including defining an agent, selecting an agent, a main large model, at least two secondary large models, and a checking agent. The definition agent manages the domain keywords of all current large models and calls the main large model to check whether there is a domain that is adapted to the problem and transmits the review result to the selection agent; the selection agent selects a large model according to the review result: when there is no adapted domain, the main large model is called to process the user input question and generate an answer; when there is an adapted domain, the large model of the adapted domain is called to process the user's question and generate an answer; the checking agent checks the correctness of the answer generated by the large model and when the answer does not meet the requirements, the corresponding large model is called to regenerate the result. Based on the above system, cross-knowledge domain reasoning tasks can be completed at low cost, low power consumption, and high quality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence related technologies, and more specifically, relates to a large model combination system across knowledge fields. Background Art

[0002] Represented by large models such as the ChatGPT series, LLaMa series, and Gemini series, large language model technology, especially multimodal large model technology, has made tremendous breakthroughs and progress in the field of artificial intelligence, and has been amazing in understanding and generating complex texts, understanding images and audio content, code writing, text graphics, text videos, and more and more aspects. The super capabilities of large models have further demonstrated their huge potential in many fields such as education, medical care, autonomous driving, and intelligent manufacturing.

[0003] At present, when large models are used for reasoning applications, the systems mostly use a single pre-trained or fine-tuned large language model or multimodal large language model as the base model, and develop large model applications on this base model. Such systems have poor applicability, that is, a single large model can have good reasoning capabilities in its corresponding field, but it is difficult to handle tasks in other fields. If the applicability of large models in different fields is to be improved, the number of parameters of the base large model must be at the level of hundreds of billions or even larger, and the computing power required for reasoning is huge, resulting in huge reasoning costs and power consumption, low energy efficiency, and poor applicability. When processing tasks in the same field, its performance accuracy is not as good as that of a single large model in that field.

[0004] In summary, there is an urgent need for a large model combination system that can complete cross-knowledge domain reasoning tasks at low cost, low power consumption and high quality. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention provides a large model combination system across knowledge domains, which aims to complete cross-knowledge domain reasoning tasks with low cost, low power consumption and high quality.

[0006] To achieve the above object, according to one aspect of the present invention, a cross-knowledge domain large model combination system is provided, which includes a large model combination, wherein the large model combination includes a definition agent, a selection agent, a main large model, at least two secondary large models, and a check agent corresponding to each large model;

[0007] The definition agent is used to manage the domain keywords of all current large models and input the domain keywords into the main large model when the user inputs a question, call the main large model to check whether there is a domain that is suitable for the question, obtain the review result of the main large model and transmit it to the selection agent; wherein different large models correspond to different domain keywords;

[0008] The selection agent is used to receive the review result and select a large model according to the review result: when the review result is that there is no adapted field, the main large model is called to process the user's input question and generate an answer; when the review result is that there is an adapted field, the large model of the adapted field is called to process the user's question and generate an answer;

[0009] The checking agent is used to call the corresponding large model to check the correctness of the answer generated by the large model itself and to call the corresponding large model to regenerate the result when the answer does not meet the requirements.

[0010] Preferably, the definition agent, selection agent and inspection agent all have corresponding definition modules, planning modules and action modules;

[0011] In the definition agent: the definition module is used to define the role of the definition agent and store the domain keywords of all large models, the planning module is used to plan the execution steps of the definition agent, and the action module is used to execute the execution steps in the corresponding planning module;

[0012] In the selection agent: the definition module is used to define the role of the selection agent, the planning module is used to plan the execution steps of the selection agent, and the action module is used to execute the execution steps in the corresponding planning module;

[0013] In the inspection agent: the definition module is used to define the role of the inspection agent, the planning module is used to plan the execution steps of the inspection agent, and the action module is used to execute the execution steps in the corresponding planning module.

[0014] Preferably, the planning modules in the definition agent and the selection agent both include fixed task submodules, which are provided with fixed execution steps, and the corresponding action modules are used to execute the fixed execution steps in the corresponding planning modules to realize the functions of the corresponding agents.

[0015] Preferably, in the inspection agent, the planning module comprises:

[0016] A fixed task submodule is provided with fixed execution steps; the corresponding action module is used to execute the fixed execution steps in the corresponding execution planning module to realize the function of the inspection agent;

[0017] and / or

[0018] The non-fixed task submodule is used to call the corresponding large model to generate execution steps; the corresponding action module is used to cooperate with the non-fixed task submodule to execute the steps of calling the large model and executing the execution steps generated by the large model.

[0019] Preferably, calling the corresponding large model to generate execution steps includes:

[0020] Send the target task to the corresponding large model and prompt the large model to decompose the task;

[0021] Get the small tasks generated by decomposing the large model, send the small tasks to the corresponding large model again and prompt the large model to generate the corresponding execution steps;

[0022] Get the execution steps for large model generation.

[0023] Preferably, the inspection agent is also used to manage domain keywords corresponding to the large model;

[0024] Before the selection agent calls the large model of the adapted field to process the user's question and generate an answer, it also includes:

[0025] The selection agent matches information with each of the inspection agents to determine a large model corresponding to the adaptation field.

[0026] Preferably, the primary large model and each of the secondary large models adopt a chain structure, and different functional components are mounted through the chain structure.

[0027] Preferably, the primary large model is selected from a general multimodal large model MLLM; the secondary large model is selected from a retrained or fine-tuned multimodal large model MLLM or a large language model LLM.

[0028] Preferably, when the secondary big model is added or deleted, the domain keywords corresponding to the secondary big model are added or deleted in the definition agent.

[0029] Preferably, it comprises a plurality of cascaded large model combinations, wherein the main large model of the large model combination of the latter level is the secondary large model of the large model combination of the former level; and the field of the large model combination of the latter level is a subdivided field of the field of the secondary large model of the former level.

[0030] In general, compared with the prior art, the above technical solution conceived by the present invention has the following beneficial effects:

[0031] 1. This cross-knowledge domain large model combination system combines intelligent agents and multiple large models. The intelligent agent can flexibly mobilize the large model to perform different tasks, and can dispatch the large model of the corresponding field to handle user questions according to different fields. Compared with a single large model system, the large model combination system of this application is more applicable and can answer questions in different professional knowledge fields. Compared with a general large model trained based on a single base large model, the large model combination system of this application has higher accuracy in answering questions;

[0032] 2. The cross-knowledge domain large model combination system will eventually select a large model for a specific domain for reasoning. Compared with the general large model trained based on a single base large model, the reasoning power consumption required by this application is lower, and its power consumption is always maintained within a suitable range, and will not cause a surge in power consumption due to the increasing number of added professional knowledge domains;

[0033] 3. This large model combination system across knowledge domains supports the flexible addition or deletion of any number of large models without affecting other parts. At the same time, it realizes a method that can expand the knowledge domain processed without limit, and does not cause forgetting due to too many knowledge domains processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a system-of-systems diagram of a large model combination system across knowledge domains in one embodiment of the present application.

[0035] Figure 2 It is a chain structure diagram of a large model in one embodiment of the present application.

[0036] Figure 3 It is a schematic diagram of the structure of an intelligent agent in one embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0038] like Figure 1 Shown is a system diagram of a large model combination system across knowledge domains in one embodiment of the present application.

[0039] refer to Figure 1As shown, the cross-knowledge domain big model combination system mentioned in the present application includes a big model combination, and the big model combination includes a definition agent, a selection agent, a main big model, at least two secondary big models, and an inspection agent corresponding to each big model.

[0040] Among them, the intelligent agent is a computer program or entity developed based on the big model with autonomy, responsiveness and learning capabilities.

[0041] Define an agent to manage the domain keywords of all current large models and input the domain keywords and prompt words to determine whether they are compatible with known domains into the main large model when the user enters a question, call the main large model to check whether there is a domain that is compatible with the question, obtain the review result of the main large model and transmit it to the selection agent; different large models correspond to different domain keywords.

[0042] Specifically, different large models deal with problems in different fields, so each model has corresponding domain keywords and is stored in the definition agent. For example, the system has large models that focus on knowledge fields such as "education", "finance", and "law", so the domain keywords of "education", "finance", and "law" can be stored in the definition agent.

[0043] Specifically, the system can flexibly add or delete secondary large models, define the intelligent agent to manage the domain keywords of all current large models, and when the system adds or deletes secondary large models, it defines the domain keywords of the corresponding secondary large models in the intelligent agent to add or delete, ensuring that the domain keywords currently stored in the defined intelligent agent correspond one-to-one with the current large models of the system.

[0044] It can be understood that the system has an input terminal for receiving user input, including user questions and user prompt words. For example, the user question is "X 2 +Y 2 = 1", and the user prompt is "Please solve this equation for all integer solutions".

[0045] When the system receives the user's input, it will input the user's question into the top-level main model, and will package all the user's input content and send it to the selection agent for temporary storage. At the same time, the definition agent will input the currently stored domain keywords and the prompt words to determine whether it is compatible with the known domain into the corresponding main model. The definition agent calls the main model to check whether there is a domain that is compatible with the question and obtain the review results.

[0046] There are two types of review results: the first is that the current system does not have any areas that are suitable for it; the second is that the current system has areas that are suitable for it and provides feedback on the specific areas.

[0047] After obtaining the review results, the definition agent transmits them to the selection agent. At this time, the selection agent has both the content input by the user and the review results for domain adaptation.

[0048] In short, the definition agent is an agent used to review whether the knowledge domain of the question entered by the user is within the knowledge domain that the system focuses on. The agent calls the large model to determine what knowledge domain the question entered by the user belongs to, obtains the review result, and passes the review result to the selection agent to perform the next step.

[0049] The selection agent is used to receive the review results and select the big model based on the review results: when the review result is that there is no adapted field, the temporarily stored user input questions and prompt words are sent to the main big model, and then the main big model is called to process and generate answers; when the review result is that there is an adapted field, the big model of the adapted field is called to process the user's questions and generate answers.

[0050] First, the selection agent will select the big model to handle the user's question based on the review results. The selection strategy is: when the review result is that there is no adapted field, call the main big model to handle the user's input question and generate an answer; when the review result is that there is an adapted field, call the big model of the adapted field to handle the user's question and generate an answer. Specifically, the inspection agent can manage the field keywords corresponding to the big model. When the review result is that there is an adapted field, the selection agent can match information with the inspection agents of each big model, determine the big model corresponding to the adapted field, and then transmit the user's input content to the big model. After receiving the user's question and prompt word, the big model processes it and generates an answer.

[0051] That is to say, the selection agent is an agent that has a vector database that temporarily stores user questions and prompt words, and performs specific operations according to the review results of the definition agent. If the review result of the definition agent is that there is a sub-model that focuses on this knowledge field in the sub-model, the keyword of the corresponding knowledge field is passed to the corresponding selection agent, and the selection agent interacts with the inspection agent corresponding to each sub-model to determine the corresponding sub-model and send the temporarily stored questions and prompt words to the corresponding sub-model; if the review result of the definition agent is that there is no sub-model that focuses on this knowledge field in the sub-model, the selection agent returns the temporarily stored questions to the main model for processing.

[0052] The checking agent is used to call the corresponding large model to check the correctness of the answer generated by the large model itself and to call the corresponding large model to regenerate the result when the answer does not meet the requirements.

[0053] Specifically, the checking agent is an agent used to implement the correctness check, quality assessment and backtracking of the answers of the primary and secondary large models. Preferably, the checking agent contains the knowledge domain keywords of the corresponding secondary large model, and interacts with the selection agent. After determining which secondary large model it is, the selection agent inputs all temporarily stored questions and prompt words into the secondary large model corresponding to the checking agent as input. After the large model outputs the generated answer, the checking agent takes the answer generated by the large model as input again, and inputs it into the corresponding large model together with the prompt words for correctness check, quality assessment and the prompt words of the assessment standard, and outputs the quality and correctness assessment of the generated content. If the set standard is met, the final generated result is output. If the set standard is not met, the checking agent requires the large model to be regenerated until it meets the requirements, and outputs the final generated result.

[0054] Based on the above large model combination system across knowledge domains, after inputting user questions and prompt words, the system is reviewed by the definition agent to see whether it is suitable for the knowledge domain asked by the user. At the same time, the user questions and prompt words are handed over to the selection agent for temporary storage. If they are included, the questions and prompt words are passed from the selection agent to the large model focusing on the corresponding knowledge domain for reasoning. The checking agent completes the correctness check, quality assessment and backtracking of the answer. After a cycle, a high-quality answer is given. If they are not included, the user's questions and prompt words are returned from the selection agent to the main large model for reasoning. The checking agent also completes the verification and backtracking of the answer, and finally the answer is output.

[0055] This is explained with a specific example.

[0056] When the user inputs a question and a prompt word, the input part of the system acts as a selector. On the one hand, it temporarily stores the question and prompt word entered by the user in the selection agent, and on the other hand, it inputs the user's question into the main large model. At the same time, the definition agent inputs the keywords of the knowledge fields that the system can currently handle and the prompt words for reviewing the relationship between the knowledge fields into the main large model for reasoning, and obtains a judgment on whether the user's question belongs to the knowledge fields already included in the system. For example, the system already contains large models focusing on these knowledge fields, such as "education", "finance", and "law". Keywords such as "education", "finance", and "law" are used as prompt words and input into the main large model together with the question entered by the user for comparison and judgment of the inference results, and two comparison and judgment results are obtained, namely, whether it matches a knowledge field currently included in the system or is not related to the included knowledge field. The judgment result is returned to the definition agent, and the definition agent passes the comparison and judgment result to the selection agent, and the selection agent takes the next step according to the comparison and judgment result.

[0057] When the comparison result transmitted is not relevant to the included knowledge domain, the selection agent will re-transmit the temporarily stored user input questions and prompt words to the main large model MLLM to complete the reasoning and generate the output result. The inspection agent corresponding to the main large model will re-input the generated content and the prompt words for correctness inspection and quality inspection into the main large model for evaluation. If the evaluation fails, the large model is required to regenerate the answer until the evaluation is qualified, and the final result output is given to complete the reasoning task.

[0058] When the comparison result transmitted matches a certain knowledge domain, the selection agent interacts with the inspection agents corresponding to all secondary large models, compares the keywords of the transmitted knowledge domain with the keywords of the knowledge domain of the inspection agent in each secondary large model, determines which secondary large model the inspection agent is, and then selects the agent to send the temporarily stored user input questions and prompt words to the determined secondary large model, performs reasoning, and obtains the generated content. The inspection agent verifies the correctness and quality of the generated content. If the evaluation is unqualified, the secondary large model is called to regenerate and evaluate again until the evaluation result is qualified, and the final output result is given to complete this reasoning task. According to this process, after each user's question and prompt word are input, the system automatically completes all the above steps and finally gives the output result.

[0059] Specifically, the main-level big model can be selected from a general multimodal big model MLLM, that is, a big language model that can understand and reason about data types in a variety of different formats, such as text, images, videos, audio, time series, etc., and is used to reason whether the question input by the user matches the focused knowledge domain already included in the system, and when no known domain matches, perform the reasoning task to generate output results. There is no restriction on the selection of the big model, which can be an open source or closed source multimodal big model, which can include but is not limited to pre-trained or fine-tuned open source or closed source multimodal big models based on Transformer, infini-transformer or similar systems, and can be LLAMA series, ChatGLM series, OpenChat series, ChatGPT series, Cluade series, Mistral series, etc.

[0060] Specifically, the secondary big model is a multimodal big model MLLM or a large language model LLM (Large language model) that has been retrained or fine-tuned for a certain knowledge domain, and is used to perform reasoning tasks in a fixed professional knowledge domain and generate the final output results. Retraining refers to a big model with new parameters obtained after all the weights in the big model are updated with the data of this knowledge domain, and fine-tuning refers to using the data set of this knowledge domain to adjust all or part of the weights of the pre-trained big model to achieve better reasoning results in this knowledge domain, thereby turning the pre-trained general big model into a professional big model that handles a certain professional knowledge domain through retraining and fine-tuning.

[0061] It can be understood that the knowledge domain is the knowledge category about a certain field or a certain topic, or the knowledge within a specific field. It can be divided into knowledge fields such as education, medical care, law, transportation, etc. from a macro perspective, or it can be knowledge within a specific scope classified under a certain concept. For example, within the concept of education, it can be subdivided into elementary school knowledge fields, junior high school knowledge fields, high school knowledge fields, or divided into liberal arts knowledge fields and science knowledge fields. Therefore, the specific definition of the knowledge domain is based on the specific scenarios in which the system is applied. According to the application scenarios, the different knowledge domains used in the actual construction are defined, and the data sets of these knowledge domains are used to fine-tune or retrain the large models corresponding to each knowledge domain.

[0062] Furthermore, the system includes a combination of multiple cascaded large models, and the main large model of the latter large model combination is the secondary large model of the former large model combination; the field of the latter large model combination is a subdivided field of the field of the former secondary large model. In other words, this cross-knowledge domain large model combination system can be iterated, that is, when the task requires further subdivision of the knowledge field, such as when the knowledge field focused on by the secondary large model is "education", and when the knowledge field needs to be further classified in the later practical application, such as the need to further subdivide "education" into two more detailed knowledge fields such as "liberal arts" and "science", the original secondary large model can be replaced with the main large model, and further constructed as follows. Figure 1 The system shown in FIG. 1 includes a further sub-sub-sub-large model and a Figure 1 The corresponding intelligent agent shown completes the construction of the structure with an iterative paradigm.

[0063] In one embodiment, the main large model and each secondary large model adopt a chain structure, and different functional components are mounted through the chain structure. Figure 2The chain structure diagram of the large model is shown. In actual application scenarios, the capabilities of the secondary large model need to be further enhanced to handle more complex practical problems. The system supports the development of the secondary large model using a chain structure to mount a variety of large model development components, such as speech-to-text (STT) components, text-to-speech (TTS) components and other functional components for different purposes before the input and after the output of the secondary large model. It also supports mounting professional models and tools such as embedding models, external vector databases, and databases that manage the memory of historical answers of large models. The chain development framework is the encapsulation and free combination of components.

[0064] In addition to STT (speech to text) and TTS (text to speech), you can add more different components and your own customized functional components to continuously enhance the capabilities of the secondary large model. When calling, you can freely call any number of these components and freely combine them with the secondary large model, thereby realizing the development and implementation of multiple functions based on the large model.

[0065] like Figure 3 The figure shows a schematic diagram of the structure of an agent in an embodiment of the present application. In one embodiment, the definition agent, the selection agent and the inspection agent all have corresponding definition modules, planning modules and action modules.

[0066] In defining an agent: the definition module is used to define the role of the defined agent and store the domain keywords of all large models, the planning module is used to plan the execution steps of the defined agent, and the action module is used to execute the execution steps in the corresponding planning module. Furthermore, the planning module of the defined agent includes a fixed task submodule, which is provided with fixed execution steps, and the corresponding action module is used to execute the fixed execution steps in the corresponding planning module to realize the functions of the corresponding agent.

[0067] Specifically, the definition module of the definition agent has two parts: role and function. The definition of role is "the knowledge domain of judging the problem", and the definition of function is the knowledge domain included, such as "education, law, medical, finance, computer" and other knowledge domain keywords. The planning module executes a fixed program. When the user inputs a question and a prompt word, the system input end acts as a selector to send the user's question to the main-level model. At the same time, the definition agent will review the relationship between the knowledge domain to which the question belongs and the knowledge domain currently included in the system and the knowledge domain included in the function together with the prompt word to the main-level model. After the main-level model completes the reasoning, it returns the review result to the definition agent. The definition agent receives and passes the result to the selection agent to complete the fixed task and realize the pre-programmed function. Its action module executes the pre-programmed function of the planning module, excluding the evaluation function, that is, calling the main-level model to generate the review result, and returning it to the definition agent to realize the construction and implementation of the entire definition agent.

[0068] When adding or deleting some secondary large models that focus on fixed knowledge areas, you only need to add and delete the keywords of the corresponding knowledge areas in the functional part of the code. For example, if you delete "computer", the function becomes function = ["education", "law", "medical", "finance"]. In this way, when the main large model is reasoning, it can clearly know which other knowledge areas it contains and complete the correct reasoning results. In other words, the system itself knows which secondary large models that focus on fixed knowledge areas are currently contained in the system.

[0069] In the selection agent: the definition module is used to define the role of the selection agent, the planning module is used to plan the execution steps of the selection agent, and the action module is used to execute the execution steps in the corresponding planning module. Further, the planning module of the selection agent includes a fixed task submodule, which is provided with fixed execution steps, and the corresponding action module is used to execute the fixed execution steps in the corresponding planning module to realize the function of the corresponding agent.

[0070] Specifically, the selection agent is a vector database for temporarily storing user input questions and prompt words and an agent that selects whether to send the temporarily stored data to the main large model or the corresponding secondary large model. Its agent part also includes a definition module, a planning module and an action module. Its definition module includes a role, and the role is defined as "selecting and sending data according to the received judgment result", and it is not necessary to include a function to explicitly limit the scope of the agent's capabilities; its planning module also executes a pre-set program, i.e., a fixed task, and selects whether to send data to the main large model or the secondary large model according to the incoming judgment result. Its action module performs the following: when the incoming judgment result is not related to the included knowledge domain, the selection agent executes to send the temporarily stored data to the main large model, and when the incoming judgment result matches a certain knowledge domain, the selection agent executes to interact with the inspection agents of all the secondary large models, compares the keywords of the knowledge domain, finds the inspection agent that focuses on this knowledge domain, and then sends the temporarily stored data to the secondary large model of this knowledge domain, and the action ends. The selection agent action module does not include an evaluation function.

[0071] In the inspection agent: the definition module is used to define the role of the inspection agent, the planning module is used to plan the execution steps of the inspection agent, and the action module is used to execute the execution steps in the corresponding planning module. Further, the planning module of the inspection agent includes: a fixed task submodule, which is provided with fixed execution steps; the corresponding action module is used to execute the fixed execution steps in the planning module to realize the functions of the inspection agent; and / or, a non-fixed task submodule, which is used to call the corresponding large model to generate execution steps; the corresponding action module is used to cooperate with the non-fixed task submodule to execute the steps of calling the large model and the execution steps generated by the large model. That is, the planning module of the inspection agent can only include fixed task submodules, or only include non-fixed task submodules, or include fixed task submodules and non-fixed task submodules at the same time. Because the secondary large model itself has been fine-tuned or retrained in a certain knowledge field, when performing task decomposition and generating corresponding execution steps, it will perform task decomposition and corresponding step generation that are more suitable for the tasks in this knowledge field according to its own capabilities, and its effect is sometimes more specific, comprehensive, and close to the tasks in this field than the preset prompt words.

[0072] Specifically, the checking agent is an agent used for checking the correctness of the answers of the big model, evaluating the quality and backtracking. Its agent part also includes a definition module, a planning module and an action module. The definition module contains roles but not functions. The definition of the role is "checking and evaluating the quality of the answer". The planning module can execute a fixed pre-written program, that is, the content generated by the big model and the prompt words for evaluating the quality of the generated content, such as "judge whether this content is correct, how is the quality, whether it is good, and whether it needs to be improved" are sent to the main big model or the secondary big model, so that the big model can judge whether it is correct and whether the content meets the quality requirements; it can also execute non-fixed tasks, such as sending the definition of the role as the task goal to the big model, and sending the prompt word "decompose the target task into small tasks one by one". The big model generates the decomposed small tasks according to the task goal and the prompt word. The planning module then sends these small tasks and the prompt word "generate appropriate execution steps according to these tasks" to the big model to obtain the generated execution steps. The action module includes an evaluation sub-module. Taking the execution of a fixed task as an example, that is, executing pre-programmed content, the evaluation sub-module inputs the content generated by the large model and the prompt words "Judge whether this content is correct, what is the quality, whether it is good, and whether it needs improvement" into the large model to generate an evaluation result. If the evaluation result meets the standard, the output is the final result. If the evaluation result does not meet the standard, it will continue to return to generate until the result meets the requirements, and the final output is the final result.

[0073] In general, the above scheme can achieve the following beneficial effects:

[0074] 1. The system mounts secondary large models that have been retrained or fine-tuned in fixed knowledge domains, so that when these secondary large models infer answers within their fixed knowledge domains, the quality of the answers they generate is higher than that of general large models that have not been fine-tuned or trained in this domain. At the same time, the system develops inspection agents for the quality of the corresponding large model answers and the right and wrong judgment to repeatedly optimize the answers generated by the large model, thereby achieving high-quality answers to questions in different professional knowledge fields.

[0075] 2. The system has no restrictions on the selection of specific big models, whether they are open source big models, or whether they are commercially available big models, so the system has strong versatility.

[0076] 3. During each inference operation, the power consumption of the system mainly comes from the inference of the main large model and one of the secondary large models, or the power consumption of the inference of the two main large models. Therefore, the overall inference power consumption of the system is always maintained within an appropriate range, and the power consumption will not surge due to the increasing number of large models in professional knowledge fields. Therefore, the system has strong practical application capabilities and can be deployed in some practical scenarios.

[0077] 4. The system has super scalability and a certain degree of autonomy. It can expand the methods of the knowledge domains it processes without limit, and will not forget because of too many knowledge domains it processes. Because every time the system adds or deletes the secondary large models of certain knowledge domains, because the secondary large models of other knowledge domains do not transmit data and instruction information to each other during reasoning, deleting or adding different secondary large models will not affect the secondary large models of other knowledge domains. Moreover, by only modifying the functional part of the definition of the intelligent agent, the system can realize that the entire system can clearly know what professional knowledge domains it contains, and automatically make judgments and complete the autonomous reasoning operation and result output of the entire system according to the specific actual situation. Since the system realizes high-quality reasoning of cross-knowledge domain and multi-knowledge domain problems by adding secondary large models of different knowledge domains, and does not change the parameter weights of the original knowledge domain secondary large models, it will not cause knowledge forgetting and reduce the accuracy of reasoning as the number of knowledge domains continues to increase.

[0078] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. It should be noted that "in one embodiment", "for example", "for example", etc. of the present invention are intended to illustrate the present invention, rather than to limit the present invention.

[0079] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent application. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A large model combination system across knowledge domains, characterized in that: The method comprises a large model combination, wherein the large model combination comprises a definition agent, a selection agent, a main large model, at least two secondary large models, and a check agent corresponding to each large model; The primary large model is a general multimodal large model MLLM, and the secondary large model is a multimodal large model MLLM or a large language model LLM that has been retrained or fine-tuned for a certain knowledge domain; The definition agent is used to manage the domain keywords of all current large models and input the domain keywords into the main large model when the user inputs a question, call the main large model to check whether there is a domain that is suitable for the question, obtain the review result of the main large model and transmit it to the selection agent; wherein different large models handle problems in different fields, and different large models correspond to different domain keywords. When the secondary large model is added or deleted, the domain keyword corresponding to the secondary large model is added or deleted in the definition agent; The selection agent is used to receive the review result and select a large model according to the review result: when the review result is that there is no adapted field, the main large model is called to process the user's input question and generate an answer; when the review result is that there is an adapted field, the large model of the adapted field is called to process the user's question and generate an answer; The checking agent is used to call the corresponding large model to check the correctness of the answer generated by the large model itself and call the corresponding large model to regenerate the result when the answer does not meet the requirements; The inspection agent is also used to manage domain keywords corresponding to the large model. Before the selection agent calls the large model of the adapted domain to process user questions and generate answers, it also includes: the selection agent matches information with each of the inspection agents to determine the large model corresponding to the adapted domain.

2. The large model combination system across knowledge domains as claimed in claim 1, characterized in that: The definition agent, selection agent and inspection agent all have corresponding definition modules, planning modules and action modules; In the definition agent: the definition module is used to define the role of the definition agent and store the domain keywords of all large models, the planning module is used to plan the execution steps of the definition agent, and the action module is used to execute the execution steps in the corresponding planning module; In the selection agent: the definition module is used to define the role of the selection agent, the planning module is used to plan the execution steps of the selection agent, and the action module is used to execute the execution steps in the corresponding planning module; In the inspection agent: the definition module is used to define the role of the inspection agent, the planning module is used to plan the execution steps of the inspection agent, and the action module is used to execute the execution steps in the corresponding planning module.

3. The large model combination system across knowledge domains as claimed in claim 2, characterized in that: The planning modules in the definition agent and the selection agent both include fixed task submodules, which are provided with fixed execution steps, and the corresponding action modules are used to execute the fixed execution steps in the corresponding planning modules to realize the functions of the corresponding agents.

4. The large model combination system across knowledge domains as claimed in claim 2, characterized in that: In the inspection agent, the planning module includes: A fixed task submodule is provided with fixed execution steps; the corresponding action module is used to execute the fixed execution steps in the corresponding execution planning module to realize the function of the inspection agent; and / or The non-fixed task submodule is used to call the corresponding large model to generate execution steps; the corresponding action module is used to cooperate with the non-fixed task submodule to execute the steps of calling the large model and executing the execution steps generated by the large model.

5. The large model combination system across knowledge domains as claimed in claim 4, characterized in that: Call the corresponding large model to generate execution steps, including: Send the target task to the corresponding large model and prompt the large model to decompose the task; Get the small tasks generated by decomposing the large model, send the small tasks to the corresponding large model again and prompt the large model to generate the corresponding execution steps; Get the execution steps for large model generation.

6. The large model combination system across knowledge domains as claimed in claim 1, characterized in that: The main large model and each of the secondary large models adopt a chain structure, and different functional components are mounted through the chain structure.

7. The large model combination system across knowledge domains as claimed in claim 1, characterized in that: It comprises a plurality of cascaded large model combinations, wherein the main large model of the large model combination of the latter level is the secondary large model of the large model combination of the former level; and the field of the large model combination of the latter level is a subdivided field of the field of the secondary large model of the former level.