Control method and device for multi-vertical-domain knowledge questions and answers and medium

Abstract

The invention discloses a control method and device for multi-vertical-domain knowledge questions and answers and a medium. The method comprises the steps that a subject type to which multi-modal data input by a user belongs is recognized through a pre-trained subject recognition model; and calling a target large model corresponding to the subject type from a model library. According to the multi-modal data, determining a target component which can be used for visualization in a component library; and determining an output data format corresponding to the target component. Taking the multi-modal data and the output data format as input of the target large model to generate an output result; and transmitting an output result to a front end for visualization. Therefore, the subject type to which the question proposed by the user belongs is recognized based on the subject recognition model of the pre-selection training, so that the target large model of the subject type is accurately called for processing, the accuracy of an output result is improved, and the requirements of a multi-subject scene are met. In addition, a visual component is called based on the multi-modal data, a visual display mode expected to be presented by the user is dynamically generated, and the diversified display requirements of the user are met.

Description

Technical Field

[0001] The present application relates to the technical field of large-model question-answering systems, and in particular to a control method, device and medium for multi-vertical domain knowledge question-answering. Background Art

[0002] With the continuous development of big model technology, it is gradually applied to various fields of life. For example, multimodal big models (MMM) can integrate information of multiple modes such as text, images, and audio, and can be applied to fields such as autonomous driving and intelligent medical diagnosis. In addition, big models are also increasingly used in multiple vertical fields such as education, medical care, finance, and government affairs.

[0003] At present, with the development of various disciplines, various fields have begun to vigorously develop large models of their own vertical domains, and the needs of different fields for visualization are also varied. It can be seen that the current large models often only serve a single discipline and can only present a single visualization method to users, which cannot meet the diverse needs of users for different fields, different disciplines, and different visualization displays.

[0004] It can be seen that how to achieve multi-domain reuse of large models and meet users' diverse visualization display needs is an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0005] In view of this, one aspect of the present application provides a control method for multi-vertical domain knowledge question answering, the method comprising: Obtain a subject identification model; Identify the subject type to which the multimodal data input by the user belongs through the subject identification model; and retrieve the target large model corresponding to the subject type from the model library; Determine, according to the multimodal data, a target component in a component library that can be used for visualization; and determine an output data format corresponding to the target component; Using the multimodal data and the output data format as input to the target macromodel to generate an output result; The output results are transmitted to the front end for visualization through the target component.

[0006] Optionally, the acquiring of the subject identification model includes: Collect subject data of multiple disciplines; and construct corresponding subject knowledge graphs based on the subject data; Performing data annotation and data enhancement on the subject knowledge graph through a three-level annotation system to obtain a training data set; After fine-tuning the structure of the pre-built BERT model, iteratively training the fine-tuned BERT model using the training data set to obtain a target BERT model; wherein the structural fine-tuning includes at least one of increasing hidden layers, reducing hidden layers, changing the size of hidden layers, and adding a fully connected layer classifier after the output layer; The target BERT model is optimized to obtain the subject recognition model.

[0007] Optionally, optimizing the target BERT model to obtain the subject recognition model includes: The target BERT model is subjected to a performance evaluation through a comprehensive evaluation function to calculate a comprehensive score of the target BERT model; wherein the components of the comprehensive evaluation function include: an exact match rate function for measuring output accuracy, an F1 score function for measuring the balance between output precision and recall, an interdisciplinary detection accuracy function for measuring the adaptability of the model in multidisciplinary tasks, and a knowledge relevance function for measuring the degree of relevance between the model output and the knowledge system; According to the comprehensive score, the target BERT model is optimized to obtain the subject recognition model.

[0008] Optionally, the model library includes a public vertical domain big model, a self-trained vertical domain big model and a universal big model; and the step of retrieving the target big model corresponding to the subject type from the model library includes: Determine whether there is a self-training vertical domain large model that meets the subject type in the model library; If it exists, the self-trained vertical domain macro model that conforms to the subject type is used as the target macro model; If it does not exist, determine whether there is a public vertical domain big model that meets the subject type in the model library; if it does exist, use the public vertical domain big model that meets the subject type as the target big model; if it does not exist, select the general big model as the target big model.

[0009] Optionally, determining a target component that can be used for visualization in a component library according to the multimodal data includes: Obtaining a list of available components in the component library according to the multimodal data; Assigning corresponding weights to component matching influencing factors; the component matching influencing factors at least include semantic recognition results of the multimodal data, historical context information, data compatibility information, and user component preferences; Calculating the matching confidence of each component in the available component list based on the component matching influence factor and the weight; The component with the highest matching confidence is used as the target component.

[0010] Optionally, the component library includes built-in components and custom components; the custom components are constructed according to component information of the component to be constructed input by the user; The custom component is constructed according to the component information of the component to be constructed input by the user, including: Obtain the component information; the component information at least includes the component name, installable component package, component required data structure description, component function description, component usage scenario description and component adaptation device; Determining whether the component information has a security risk; If it does not exist, assign a unique component ID to the component to be constructed; construct a component knowledge graph of the component to be constructed according to the component ID and the component information to obtain the custom component; If it exists, send modification information to the client.

[0011] Optionally, the using the multimodal data and the output data format as inputs of the target macro model to generate an output result comprises: Determining task description information according to the multimodal data; the task description information includes current context information, current user question and user request; The task description information and the output data format are rewritten by a pre-built Prompt project to obtain a target text; The target text is input into the target large model for reasoning to obtain the output result.

[0012] Optionally, transmitting the output result to the front end for visualization through the target component includes: After performing target processing on the output result, a target output result is obtained; wherein the target processing includes at least one of format verification, protocol conversion, data form conversion, data integration and data optimization; The target output result is transmitted to the front end so that the front end can call the target component; and the target component is rendered to display the target output result.

[0013] Another aspect of the present application provides a control device for multi-vertical domain knowledge question answering, the device comprising: A subject identification model acquisition module is used to acquire a subject identification model; A model retrieval module is used to identify the subject type to which the multimodal data input by the user belongs through the subject identification model; and retrieve the target large model corresponding to the subject type from the model library; An output data format determination module is used to determine a target component that can be used for visualization in a component library according to the multimodal data; and determine an output data format corresponding to the target component; An output result acquisition module, used for taking the multimodal data and the output data format as inputs of the target large model to generate an output result; The result transmission module is used to transmit the output result to the front end so as to be visualized through the target component.

[0014] Another aspect of the present application provides a control device for multi-vertical domain knowledge questions and answers, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the program, the steps of the control method for multi-vertical domain knowledge questions and answers are implemented.

[0015] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the control method of the multi-vertical domain knowledge question and answer.

[0016] The control method, device and medium of multi-vertical knowledge question and answer provided by the present application have the following beneficial effects: based on the pre-selected and trained subject recognition model, multi-modal data is deeply analyzed to identify the subject type to which the question raised by the user belongs, so as to accurately retrieve the target large model that can be used to process the subject type. Processing by the target large model corresponding to the subject can not only improve the accuracy of the output results, but also meet the needs of multi-field and multi-disciplinary application scenarios. In addition, user intent recognition is combined with the component library, and visualization components are retrieved based on the multi-modal data input by the user, and the visualization display method that the user expects to present is dynamically generated, thereby meeting the user's diverse visualization display needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a method for controlling multi-vertical domain knowledge question and answer provided in an embodiment of the present application; Figure 2 A schematic diagram of the architecture of a control system for multi-vertical domain knowledge question answering provided in an embodiment of the present application; Figure 3 A flowchart of a method for controlling multi-vertical domain knowledge question and answer provided by another embodiment of the present application; Figure 4 A flowchart of a method for controlling multi-vertical domain knowledge question and answer provided by another embodiment of the present application; Figure 5 A schematic diagram of a process for constructing a custom component provided in an embodiment of the present application; Figure 6A schematic diagram of the structure of a control device for multi-vertical domain knowledge question and answer provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of a control device for multi-vertical domain knowledge question and answer provided in another embodiment of the present application.

[0018] The reference numerals are as follows: 70 is a memory, 71 is a processor, 72 is a display screen, 73 is an input / output interface, 74 is a communication interface, 75 is a power supply, 76 is a communication bus, 701 is a computer program, 702 is an operating system, and 703 is data. DETAILED DESCRIPTION

[0019] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0020] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0021] Figure 1 A flowchart of a method for controlling multi-vertical domain knowledge question answering provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes: S10: Obtain subject identification model; S11: Identify the subject type to which the multimodal data input by the user belongs through the subject identification model; and retrieve the target large model corresponding to the subject type from the model library; In order to enable big models in different fields to better serve the corresponding fields (disciplines) and realize the reuse of big models, in a specific embodiment, the multimodal data input by the user is identified through the discipline recognition model to determine the discipline type to which the user input question belongs, and retrieve the target big model corresponding to the discipline type from the model library, that is, retrieve the big model specifically used to handle problems of this discipline type.

[0022] In an optional embodiment, the subject recognition model is a model pre-trained based on a large amount of knowledge in different subject areas, and is used to quickly identify different subject types. It should be noted that the multimodal data input by the user can be but is not limited to text, voice, picture, file, etc.

[0023] In addition, it should be noted that the model library stores large models commonly used in different vertical fields, general models, and large models trained by users themselves, etc. This application does not limit the type and number of models stored in the model library.

[0024] Figure 2 This is a schematic diagram of the architecture of a control system for multi-vertical domain knowledge question answering provided in an embodiment of the present application. For ease of understanding, the following will be combined with Figure 2 For explanation. Figure 2 As shown, in an optional embodiment, the control system includes two parts: a server and an (intelligent) agent, and the execution subject of the control method of multi-vertical domain knowledge question answering provided in the embodiment of the present application can be Figure 2 The control system composed of the server and agent shown.

[0025] In a specific embodiment, Figure 2 The front end shown receives the multimodal data input by the user through the user input request module, that is, the front end obtains the user request. Among them, the multimodal data supports different forms such as text, pictures, files, etc. The multimodal data is transmitted to the server through the network in the agreed protocol format. After the server receives the user request, it parses the requested multimodal data and passes it to the Agent. The Agent calls the pre-deployed subject recognition model to identify one or more subject types to which the multimodal data belongs, and calls the server interface for the determined subject type to store it. In addition, the corresponding target large model is retrieved from the model library according to the subject type.

[0026] S12: determining a target component that can be used for visualization in a component library according to the multimodal data; and determining an output data format corresponding to the target component; In order to meet the user's needs for diverse visualization, further, the target component that can be currently visualized by the user is determined based on the multimodal data, that is, the target component type that is finally displayed and the output data format corresponding to the target component are obtained in the component library according to the user's current needs.

[0027] Therefore, in the embodiment of the present application, text output is used as a fallback solution, a component dynamic adaptation mechanism is formulated, and the target component for final display is obtained according to user needs (ie, multimodal data).

[0028] S13: using the multimodal data and the output data format as input to a target large model to generate an output result; S14: The output results are transmitted to the front end for visualization through the target component.

[0029] Furthermore, after the agent retrieves the target large model, it uses the multimodal data and the output data format corresponding to the target component as the input of the target large model to generate the final answer to the user's request, that is, to generate the output result. The output result is transmitted to the front end, and the front end retrieves the target component from the component library for rendering to visualize the output result.

[0030] Therefore, the control method of multi-vertical knowledge question and answer provided in the embodiment of the present application deeply analyzes multimodal data based on the pre-selected and trained subject recognition model, thereby identifying the subject type to which the question raised by the user belongs, so as to accurately retrieve the target large model that can be used to process the subject type. Processing through the target large model corresponding to the subject can not only improve the accuracy of the output results, but also meet the needs of multi-field and multi-disciplinary application scenarios. In addition, the user intent recognition is combined with the component library, and the visualization component is retrieved based on the multimodal data input by the user, and the visualization display method that the user expects to present is dynamically generated, thereby meeting the user's diverse visualization display needs.

[0031] Figure 3 A flowchart of a method for controlling multi-vertical domain knowledge question and answer provided by another embodiment of the present application is shown as follows: Figure 3 As shown, a subject identification model is obtained, including: S30: Collect subject data from multiple disciplines; and build corresponding subject knowledge graphs based on the subject data; In a specific embodiment, subject data of multiple subjects can be collected through different channels such as subject textbooks, paper abstracts, professional dictionaries, academic literature, and network resources. It should be noted that the subject types may include but are not limited to biology, mechanics, and medicine. In a specific embodiment, subject data of subjects of interest can be collected according to different needs of users, and the subject recognition model can be trained according to actual needs to achieve accurate recognition of specific subjects.

[0032] After collecting the subject data, we construct the corresponding subject knowledge graph for different subject data. Specifically, we build the tree-like hierarchical structure of the subject through the existing public reference authoritative classification standards, and define the association rules between subjects.

[0033] Specifically, the data of disciplines, sub-disciplines, core concepts, and classical theories are used as nodes of the knowledge graph, the data of parent-child relationships, related fields, and cross-applications are used as edges of the knowledge graph, and the data of discipline codes, founding scholars, and landmark achievements are used as attributes of the knowledge graph, so as to construct triples of the discipline knowledge graph and obtain the discipline knowledge graph. It can be understood that different discipline data can construct a corresponding discipline knowledge graph, which is presented in the discipline knowledge graph as a triple structure.

[0034] S31: Data annotation and data enhancement are performed on the subject knowledge graph through a three-level annotation system to obtain a training data set; Furthermore, a three-level annotation system is used to annotate the subject knowledge graph constructed in step S30. At the same time, in order to ensure the recognition effect of the subject recognition model, in an optional embodiment, data enhancement is performed on the annotated data to obtain a training data set. The data enhancement may include but is not limited to term replacement, knowledge mixing, and adversarial samples.

[0035] S32: After fine-tuning the structure of the pre-built BERT model, iteratively train the fine-tuned BERT model using the training data set to obtain a target BERT model; wherein the structural fine-tuning includes at least one of increasing a hidden layer, reducing a hidden layer, changing a hidden layer size, and adding a fully connected layer classifier after an output layer; In an optional embodiment, a BERT model is pre-built, and in order to adapt to the recognition needs of different disciplines, the BERT model is structurally fine-tuned based on the recognition tasks of different disciplines. Specifically, the structural fine-tuning process may include but is not limited to adding hidden layers, reducing hidden layers, and changing the size of hidden layers. In addition, a fully connected layer classifier can be added after the output layer of the BERT model to map the features extracted by BERT to the subject categories. Furthermore, the structurally fine-tuned BERT model is iteratively trained using the training data set obtained in step S32 until the number of iterations is reached to obtain the target BERT model.

[0036] S33: Optimize the target BERT model to obtain a subject recognition model.

[0037] In order to further improve the subject recognition effect, the target BERT model is optimized to obtain a subject recognition model that can be directly used for subject recognition of multimodal data input by users.

[0038] Based on the above embodiment, as an optional embodiment, Figure 3 As shown, the target BERT model is optimized to obtain a subject recognition model, including: S330: Evaluate the performance of the target BERT model through a comprehensive evaluation function to calculate a comprehensive score of the target BERT model; wherein the components of the comprehensive evaluation function include: an exact match rate function for measuring output accuracy, an F1 score function for measuring the balance between output precision and recall, an interdisciplinary detection accuracy function for measuring the adaptability of the model in multidisciplinary tasks, and a knowledge relevance function for measuring the degree of relevance between the model output and the knowledge system; S331: Based on the comprehensive score, the target BERT model is optimized to obtain a subject recognition model.

[0039] In a specific embodiment, based on the fine-tuning of the BERT model structure, the model is further optimized to improve the performance of the subject recognition model. Specifically, in an optional embodiment, the target BERT model can be evaluated and measured by four indicators: the exact match rate function (EM), the F1 score function, the cross-disciplinary detection accuracy function, and the knowledge relevance function, and the target BERT model can be optimized according to the evaluation results (i.e., the calculated comprehensive score).

[0040] It is understandable that the exact match rate function can be used to measure the accuracy of the model output, the F1 score function can be used to measure the balance between output precision and recall, the cross-disciplinary detection accuracy function can be used to measure the adaptability of the model in multidisciplinary tasks, and the knowledge relevance function can be used to measure the degree of relevance between the model output and the knowledge system. The target BERT model is comprehensively evaluated through four indicators to improve the performance of the model in various dimensions.

[0041] Among them, the F1 score function can evaluate the model according to the F1 score value calculated by subject level weighting. The specific calculation formula is shown in formula (1): (1) in, is the F1 score value, The weight of each subject level is For the The accuracy of the subject, For the The recall rate of the subject, is the total number of subject levels, where the sum of the weights of all subject levels is equal to 1.

[0042] After obtaining the comprehensive score, the target BERT model is optimized based on the comprehensive score. Specifically, in an optional embodiment, the training strategy or hyperparameters of the target BERT model may be adjusted to further improve the performance, and finally obtain an optimized subject recognition model.

[0043] Furthermore, after obtaining the subject recognition model, the model is tested and deployed, and during operation, the database consisting of the knowledge graphs of each subject is regularly updated, the training data set is updated, and the model is continuously retrained and optimized. Figure 2 The Agent shown is called and managed in a unified manner.

[0044] As an optional embodiment, the model library may include but is not limited to public vertical domain big models, self-trained vertical domain big models and general big models. This application does not limit the number of big models of each type. Among them, the public vertical domain big model refers to a big model that can be called by the public, and the self-trained vertical domain big model refers to a model trained by the user according to actual business needs or specific field needs. The general big model may include but is not limited to LLM (Large Language Model).

[0045] On this basis, as an optional embodiment, the target large model corresponding to the subject type is retrieved from the model library, including: Determine whether there is a self-training vertical domain large model that meets the subject type in the model library; If it exists, the self-trained vertical domain large model that meets the subject type is used as the target large model; If it does not exist, determine whether there is a public vertical domain big model that meets the discipline type in the model library; if it does, use the public vertical domain big model that meets the discipline type as the target big model; if it does not exist, select the general big model as the target big model.

[0046] It can be understood that in order to meet the actual needs of users, that is, to understand the user's intentions as much as possible and generate accurate output results, in a specific embodiment, when calling the target big model according to the subject type, the user's self-trained vertical domain big model has a higher priority than the public vertical domain big model, and the public vertical domain big model has a higher priority than the general big model.

[0047] Therefore, in the specific embodiment of calling the target macromodel, it is first determined whether there is a self-training macromodel corresponding to the subject type generated by the subject recognition model in the model library. If so, the self-training macromodel is used as the target macromodel. If not, it is first determined whether there is a matching public vertical domain macromodel. If there is no matching public vertical domain macromodel, the general macromodel is selected as the target macromodel.

[0048] It should be noted that if the multimodal data input by the user belongs to multiple subject types, multiple different target large models may be called when calling the target large model. In this case, different target large models respond to the user's request, and when the results are finally displayed, the results can be integrated and displayed. In addition, it should be noted that in a specific embodiment, if Figure 2 As shown, the server also includes model permission management, which is used to manage the permissions of each large model in the model library.

[0049] Figure 4 A flowchart of a method for controlling multi-vertical domain knowledge question and answer provided by another embodiment of the present application is shown as follows: Figure 4 As shown, according to the multimodal data, the target components that can be used for visualization in the component library are determined, including: S40: Obtain a list of available components in a component library according to the multimodal data; Figure 2 After the server obtains the multimodal data (i.e., user questions, i.e., user requests) input by the user, it performs a preliminary screening of the components by querying the component library for a list of components that the user can use, i.e., an available component list, and passes the obtained available component list to the Agent.

[0050] S41: assigning corresponding weights to component matching influencing factors; the component matching influencing factors at least include semantic recognition results of multimodal data, historical context information, data compatibility information, and user component preferences; S42: Calculate the matching confidence of each component in the available component list based on the component matching influence factor and weight; Furthermore, the Agent calls the pre-built component evaluation algorithm model to score each component in the available component list, obtains the matching confidence list of all components, and sorts them.

[0051] Specifically, the component evaluation algorithm model is constructed by combining the semantic recognition results, historical context information, data compatibility information, and user component preference, four factors that affect component matching. That is, corresponding weights are assigned to different component matching influencing factors, and the sum of the weights of all factors is equal to 1.

[0052] The semantic recognition result refers to multi-modal speech recognition to determine the user's intention and select components based on the user's intention. For example, if you want to draw a line graph of data set A, when calling the target component, you should consider the component suitable for drawing a line graph.

[0053] The historical context information is used to determine the components that can be called currently in combination with the context. For example, please draw a line graph of data set A over the years. However, if it is found in combination with the historical context information that the user has not provided the corresponding data set A, the component that can draw the line graph cannot be selected.

[0054] Data compatibility considers the compatibility between the target component and the generated data, while user component preference considers the component type that the user prefers.

[0055] Therefore, the matching confidence of the components is calculated based on the above four dimensions. Specifically, based on the matching influence factors and weights of each component, a weighted sum calculation is performed to obtain the matching confidence of each component in the available component list.

[0056] S43: The component with the highest matching confidence is taken as the target component.

[0057] It can be understood that the higher the calculated matching confidence, the closer the representation is to the visualization method expected by the user. Therefore, in a specific embodiment, all matching confidences can be sorted in descending order, and the component with the highest confidence can be used as the target component for the final front-end visualization display.

[0058] In a specific embodiment, user feedback data on components can be continuously collected, and the weight of the component matching influencing factor can be continuously optimized based on the feedback data, so as to more accurately and efficiently determine the target component that is close to the user's visualization expectations. The feedback data may include, but is not limited to, text, voice, and like results of user feedback.

[0059] During the internal testing phase, a radio button can be set below the display results of each visualization component to indicate whether the component is appropriate for display. This allows users to directly provide feedback on the accuracy of the component during the internal testing phase, making it easier to continuously tune the weights of the component matching influencing factors, thereby improving the accuracy of component confidence calculations.

[0060] In an optional embodiment, the component library stores component knowledge graphs about different components, that is, the component library stores component-related information in the form of a knowledge graph. The component-related information includes, but is not limited to, component ID, component name, installable component package, component required data structure description, component function description, component usage scenario description, component adaptation device, component data format, component parameter requirements, keyword matching rules, and front-end rendering restrictions.

[0061] The component library established includes built-in components, that is, components that come with the system, and user-defined components. In a specific embodiment, a unique component ID is created for each component to uniquely identify and call each component. Among them, the built-in components are divided into two parts, one part is a public component without domain characteristics, including but not limited to a text component, a line graph component, and a bar graph component. The other part is a component with domain characteristics, which may include but not limited to a geological map component and a stereographic projection component.

[0062] Figure 5 A flowchart of a custom component construction provided in an embodiment of the present application is shown in FIG. 1 , but a custom component needs to be constructed through component information input by a user. Specifically, Figure 5As shown, the custom component is constructed according to the component information of the component to be constructed input by the user, including: S50: Obtain component information; the component information at least includes component name, installable component package, component required data structure description, component function description, component usage scenario description and component adaptation device; In a specific embodiment, after the user develops the component according to the interface specification, he / she registers on the component registration page and fills in the component information on the registration page. Specifically, it may include but is not limited to the component name, installable component package, data structure description required for the component, component function description, component usage scenario description, component adaptation device, output data format, component data format, component parameter requirements, keyword matching rules, and front-end rendering restrictions.

[0063] S51: Determine whether the component information has a security risk; if not, proceed to step S52; if so, proceed to step S53; S52: assign a unique component ID to the component to be constructed; construct a component knowledge graph of the component to be constructed according to the component ID and component information to obtain a custom component; S53: Send modification information to the client.

[0064] After obtaining the component information, the security of the component information is verified. In an optional embodiment, a verification method can be pre-set, and the control system automatically verifies. In another optional embodiment, the administrator can perform a security audit on the component information. This application does not limit the verification method.

[0065] After verification, if the component has security risks, the modification information is sent to the client so that the user can modify the component information. After modification, the user resubmits the new component information and enters step S50 again, and the cycle is repeated until the construction of the component to be constructed is completed.

[0066] Of course, if the component information of the component to be built passes the security risk review, the front-end component library upgrade program will be automatically triggered. At the same time, a uniquely identifiable component ID will be assigned to the component to be built, and the component knowledge graph of the component to be built will be constructed based on the component ID and the component information entered by the user. The component knowledge graph will be packaged and added to the component library, thereby completing the construction of the custom component.

[0067] After the front-end page is refreshed, the display of user-defined components is supported based on the latest component library configuration table. When the received data type matches the unique ID of the current component, the user-defined component is dynamically loaded for display so that users can view their customized components.

[0068] In a specific embodiment, after determining the target component based on the component library, the output data format corresponding to the target component can be obtained by querying the component knowledge graph corresponding to the target component.

[0069] In an optional embodiment, the multimodal data and the output data format are used as inputs to a target large model to generate output results, including: Determine task description information based on the multimodal data; the task description information includes current context information, current user questions and user requests; Rewrite the task description information and output data format through the pre-built Prompt project to obtain the target text; The target text is input into the target large model for reasoning to obtain the output result.

[0070] In a specific embodiment, the current task description information is determined based on the multimodal data, specifically including the current context information, the current user question and the user request. The current context information refers to the information of the user's question in the context, and the current user question refers to the specific problem or question that the user wants to solve, for example, how to make a cup of latte? A user request refers to a specific task or operation that the user wants the model to perform. For example, for the user question "What specific task or operation does the user want the model to perform?", the user request is "Please answer with step-by-step instructions."

[0071] Further, the task description information and output format data (i.e., the data format corresponding to the target component as the output data format requirement) are used as Figure 2 The input of the pre-built Prompt project is shown to rewrite the Prompt to obtain the target text.

[0072] Furthermore, the Agent inputs the rewritten target text into the target large model for reasoning and obtains the output result. Among them, the role of the prompt project is to guide the target large model to understand the task goal more accurately and generate the expected output by designing and optimizing the input prompt (Prompt). The prompt project is not only an "advance understanding" of the input, but also an active guidance and optimization process to help the model better complete specific tasks.

[0073] Based on the above embodiment, as an optional embodiment, the output result is transmitted to the front end for visualization through the target component, including: After target processing is performed on the output result, a target output result is obtained; wherein the target processing includes at least one of format verification, protocol conversion, data form conversion, data integration and data optimization; The target output result is transmitted to the front end so that the front end can call the target component; and the target component is rendered to display the target output result.

[0074] In a specific embodiment, after the server obtains the output result of the target large model, the output result can be processed as follows: Figure 2 The target processing shown, where the target processing may include but is not limited to verifying the data format, unifying the data protocol (i.e., protocol conversion), and generating a data format that can be displayed on the front end (i.e., data format conversion). The protocol format of the server-side organization front-end page is based on the OpenAI format, and an internal component list display protocol is added.

[0075] In addition, if there are multiple target large models, there are also multiple output results, so the output result data needs to be integrated and optimized. After processing, the target output results are transmitted to the front end.

[0076] Furthermore, the front-end visualization framework receives the target output results and Figure 2 The data parsing module shown is used for parsing. In the parsed data, the component ID field is obtained to determine the specific target component to be displayed, and the data protocol required by the component stored on the server is called, that is, Figure 2 The component library protocol management shown verifies the parsed target component data, and renders and displays it in the corresponding area after the verification passes.

[0077] In an optional embodiment, a feedback entry can be provided for the user to optimize the large model and calculation model involved in the control method of the entire multi-vertical domain knowledge question and answer. Specifically, the component feedback data of the user on the component can be obtained, and the weight of the component matching influencing factor can be continuously optimized according to the component feedback data, so as to more accurately and efficiently determine the target component that is close to the user's visualization expectation.

[0078] In addition, a feedback entry can be set for each target output result. For example, buttons for likes and dislikes can be set, or text, voice, pictures and other information input by the user can be directly obtained, thereby indirectly obtaining user satisfaction feedback on the target large model, so as to optimize and adjust the models in the model library according to the feedback results.

[0079] In an optional embodiment, when the online system is running, the ability to take screenshots and provide feedback is provided in the system settings, so that users can provide feedback on any issues with the system, including answer accuracy, display accuracy, functional issues, etc., and the optimization and adjustment of large models can also be achieved.

[0080] Therefore, by semantically understanding the question data asked by the user, the target big model corresponding to the subject type of the question is obtained for data processing, as well as the visualization presentation method of the target component desired by the user, while ensuring the accuracy of the output results, that is, realizing the reuse of the multi-vertical domain big model while meeting the diverse visualization needs of users, and facilitating the use of various vertical domains and multidisciplinary cross-research scenarios.

[0081] In the above embodiments, the control method for multi-vertical domain knowledge question and answer is described in detail. The present application also provides an embodiment corresponding to a control device for multi-vertical domain knowledge question and answer.

[0082] Figure 6 A schematic diagram of the structure of a control device for multi-vertical domain knowledge question answering provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device comprises: The subject identification model acquisition module 60 is used to acquire the subject identification model; The model retrieval module 61 is used to identify the subject type to which the multimodal data input by the user belongs through the subject identification model; and retrieve the target large model corresponding to the subject type from the model library; The output data format determination module 62 is used to determine the target component that can be used for visualization in the component library according to the multimodal data; and determine the output data format corresponding to the target component; An output result acquisition module 63, used to use the multimodal data and the output data format as inputs of the target large model to generate output results; The result transmission module 64 is used to transmit the output result to the front end for visualization through the target component.

[0083] In addition, the control device for multi-vertical domain knowledge question and answer provided in the embodiment of the present application also includes: The subject data collection module is used to collect subject data of multiple subjects and build corresponding subject knowledge graphs based on the subject data. The training data set acquisition module is used to annotate and enhance the subject knowledge graph through a three-level annotation system to obtain the training data set; A training module, configured to perform structural fine-tuning on a pre-built BERT model, and then iteratively train the fine-tuned BERT model using a training data set to obtain a target BERT model; wherein the structural fine-tuning includes at least one of increasing a hidden layer, decreasing a hidden layer, changing a hidden layer size, and adding a fully connected layer classifier after an output layer; The model optimization module is used to optimize the target BERT model to obtain a subject recognition model.

[0084] A comprehensive score calculation module, used to evaluate the performance of the target BERT model through a comprehensive evaluation function to calculate the comprehensive score of the target BERT model; wherein the components of the comprehensive evaluation function include: an exact match rate function for measuring output accuracy, an F1 score function for measuring the balance between output precision and recall, an interdisciplinary detection accuracy function for measuring the adaptability of the model in multidisciplinary tasks, and a knowledge relevance function for measuring the degree of relevance between the model output and the knowledge system; The model optimization module is also used to optimize the target BERT model according to the comprehensive score to obtain a subject recognition model.

[0085] The first processing module is used to determine whether there is a self-trained vertical domain big model that meets the subject type in the model library; if so, the self-trained vertical domain big model that meets the subject type is used as the target big model; if not, determine whether there is a public vertical domain big model that meets the subject type in the model library; if so, the public vertical domain big model that meets the subject type is used as the target big model; if not, select the general big model as the target big model.

[0086] An available component list acquisition module is used to obtain a list of available components in a component library based on multimodal data; A weight assignment module is used to assign corresponding weights to component matching influencing factors; the component matching influencing factors at least include semantic recognition results of multimodal data, historical context information, data compatibility information, and user component preferences; A matching confidence calculation module is used to calculate the matching confidence of each component in the available component list based on the component matching influence factor and weight; The target component determination module is used to take the component with the highest matching confidence as the target component.

[0087] The component information acquisition module is used to obtain component information; the component information at least includes the component name, installable component package, component required data structure description, component function description, component usage scenario description and component adaptation device; A security risk determination module is used to determine whether there is a security risk in the component information; if not, call the component construction model module; if so, call the information sending module; The component construction model module is used to assign a unique component ID to the component to be constructed; construct a component knowledge graph of the component to be constructed based on the component ID and component information to obtain a custom component; The information sending module is called to send the modification information to the client.

[0088] A task description information determination module is used to determine task description information based on multimodal data; the task description information includes current context information, current user question and user request; The target text acquisition module is used to rewrite the task description information and output data format through the pre-built Prompt project to obtain the target text; The reasoning module is used to input the target text into the target large model for reasoning and obtain the output result.

[0089] A second processing module is used to perform target processing on the output result to obtain a target output result; wherein the target processing includes at least one of format verification, protocol conversion, data form conversion, data integration and data optimization; The result transmission module is also used to transmit the target output result to the front end so that the front end can call the target component; and render the target component to display the target output result.

[0090] Figure 7 A schematic diagram of a control device for multi-vertical domain knowledge question answering provided by another embodiment of the present application is shown in FIG. Figure 7 As shown, the control device for multi-vertical domain knowledge question and answer includes: a memory 70 for storing a computer program; The processor 71 is used to implement the steps of the control method of multi-vertical domain knowledge question and answer as mentioned in the above embodiment when executing the computer program.

[0091] The control device for the multi-vertical domain knowledge question and answer provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer or a desktop computer.

[0092] Among them, the processor 71 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 71 can be implemented in at least one hardware form of a digital signal processor (Digital Signal Processor, referred to as DSP), a field programmable gate array (Field-Programmable Gate Array, referred to as FPGA), and a programmable logic array (Programmable Logic Array, referred to as PLA). The processor 71 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (Central Processing Unit, referred to as CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 71 may be integrated with a graphics processing unit (Graphics Processing Unit, referred to as GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 71 may also include an artificial intelligence (Artificial Intelligence, referred to as AI) processor, which is used to process computing operations related to machine learning.

[0093] The memory 70 may include one or more computer-readable storage media, which may be non-transitory. The memory 70 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 70 is at least used to store the following computer program 701, wherein, after the computer program is loaded and executed by the processor 71, it can implement the relevant steps of the control method of the multi-vertical domain knowledge question and answer disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 70 may also include an operating system 702 and data 703, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 702 may include Windows, Unix, Linux, etc. Data 703 may include, but is not limited to, relevant data involved in the control method of the multi-vertical domain knowledge question and answer.

[0094] In some embodiments, the control device for multi-vertical knowledge questions and answers may also include a display screen 72, an input and output interface 73, a communication interface 74, a power supply 75 and a communication bus 76.

[0095] Those skilled in the art will understand that Figure 7 The structure shown in does not constitute a limitation on the control device for multi-vertical domain knowledge quiz and may include more or fewer components than shown in the figure.

[0096] The control device for multi-vertical domain knowledge question and answer provided in an embodiment of the present application includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the control method for multi-vertical domain knowledge question and answer in the above embodiment.

[0097] It should be noted that, although the operations are depicted in a specific order in the accompanying drawings, this should not be understood as requiring these operations to be performed in the specific order shown or to be performed sequentially, or requiring all illustrated operations to be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

Claims

1. A control method for multi-domain knowledge question answering, characterized in that: The method comprises: Obtain a subject identification model; Identify the subject type to which the multimodal data input by the user belongs through the subject identification model; and retrieve the target large model corresponding to the subject type from the model library; Determine, according to the multimodal data, a target component in a component library that can be used for visualization; and determine an output data format corresponding to the target component; Using the multimodal data and the output data format as input to the target macromodel to generate an output result; The output results are transmitted to the front end for visualization via the target component.

2. The control method for multi-domain knowledge question answering according to claim 1, characterized in that: The obtaining of the subject identification model comprises: Collect subject data of multiple disciplines; and construct corresponding subject knowledge graphs based on the subject data; Perform data annotation and data enhancement on the subject knowledge graph through a three-level annotation system to obtain a training data set; After fine-tuning the structure of the pre-built BERT model, iteratively training the fine-tuned BERT model using the training data set to obtain a target BERT model; wherein the structural fine-tuning includes at least one of increasing hidden layers, reducing hidden layers, changing the size of hidden layers, and adding a fully connected layer classifier after the output layer; The target BERT model is optimized to obtain the subject recognition model.

3. The control method for multi-domain knowledge question answering according to claim 2, characterized in that: The step of optimizing the target BERT model to obtain the subject recognition model includes: The target BERT model is subjected to a performance evaluation through a comprehensive evaluation function to calculate a comprehensive score of the target BERT model; wherein the components of the comprehensive evaluation function include: an exact match rate function for measuring output accuracy, an F1 score function for measuring the balance between output precision and recall, an interdisciplinary detection accuracy function for measuring the adaptability of the model in multidisciplinary tasks, and a knowledge relevance function for measuring the degree of relevance between the model output and the knowledge system; According to the comprehensive score, the target BERT model is optimized to obtain the subject recognition model.

4. The control method for multi-domain knowledge question answering according to claim 1, characterized in that: The model library includes a public vertical domain big model, a self-trained vertical domain big model and a general big model; the step of retrieving the target big model corresponding to the subject type from the model library includes: Determine whether there is a self-training vertical domain large model that meets the subject type in the model library; If it exists, the self-trained vertical domain macro model that conforms to the subject type is used as the target macro model; If it does not exist, determine whether there is a public vertical domain big model that meets the subject type in the model library; if it does exist, use the public vertical domain big model that meets the subject type as the target big model; if it does not exist, select the general big model as the target big model.

5. The control method for multi-domain knowledge question answering according to claim 1, characterized in that: Determining a target component in a component library that can be used for visualization according to the multimodal data includes: Obtaining a list of available components in the component library according to the multimodal data; Assigning corresponding weights to component matching influencing factors; the component matching influencing factors at least include semantic recognition results of the multimodal data, historical context information, data compatibility information, and user component preferences; Calculating the matching confidence of each component in the available component list based on the component matching influence factor and the weight; The component with the highest matching confidence is used as the target component.

6. The control method for multi-domain knowledge question answering according to claim 1, characterized in that: The component library includes built-in components and custom components; the custom components are constructed according to the component information of the component to be constructed input by the user; The custom component is constructed according to the component information of the component to be constructed input by the user, including: Obtain the component information; the component information at least includes the component name, installable component package, component required data structure description, component function description, component usage scenario description and component adaptation device; Determining whether the component information has a security risk; If it does not exist, assign a unique component ID to the component to be constructed; construct a component knowledge graph of the component to be constructed according to the component ID and the component information to obtain the custom component; If it exists, send modification information to the client.

7. The control method for multi-domain knowledge question answering according to claim 1, characterized in that: The using the multimodal data and the output data format as inputs of the target large model to generate an output result comprises: Determining task description information according to the multimodal data; the task description information includes current context information, current user question and user request; The task description information and the output data format are rewritten by a pre-built Prompt project to obtain a target text; The target text is input into the target large model for reasoning to obtain the output result.

8. The control method for multi-domain knowledge question answering according to claim 1, characterized in that: The transmitting the output result to the front end for visualization through the target component comprises: After performing target processing on the output result, a target output result is obtained; wherein the target processing includes at least one of format verification, protocol conversion, data form conversion, data integration and data optimization; The target output result is transmitted to the front end so that the front end can call the target component; and the target component is rendered to display the target output result.

9. A control device for multi-vertical domain knowledge question and answer, characterized in that: The device comprises: A subject identification model acquisition module is used to acquire a subject identification model; A model retrieval module is used to identify the subject type to which the multimodal data input by the user belongs through the subject identification model; and retrieve the target large model corresponding to the subject type from the model library; An output data format determination module is used to determine a target component that can be used for visualization in a component library according to the multimodal data; and determine an output data format corresponding to the target component; An output result acquisition module, used for taking the multimodal data and the output data format as inputs of the target large model to generate an output result; The result transmission module is used to transmit the output result to the front end so as to be visualized through the target component.

10. A control device for multi-domain knowledge question and answer, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps of the control method of multi-vertical domain knowledge question and answer according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the control method of multi-vertical domain knowledge question and answering as described in any one of claims 1 to 8 are implemented.

Images