Large language model structured multi-modal response method and device for vertical field

Abstract

The application provides a large language model structured multi-mode response method and device for vertical fields, comprising: based on a natural language question input by a user, identifying a task mode to which the natural language question belongs, and extracting parameters corresponding to the task mode from a stored task mode-parameter mapping table; calculating the similarity of the natural language question and the structural label set and high-dimensional semantic vector of each knowledge fragment in the knowledge base, and fusing to obtain a similarity score; if the maximum value of the similarity score is greater than or equal to a confidence threshold, directly executing answer content generation, otherwise inferring the answer to the natural language question according to inference rules, taking the inference conclusion as a response benchmark text, and labeling the source information of the inference conclusion; generating the response content according to the domain structured template, and labeling the source information. The application has higher accuracy.

Description

Technical Field

[0001] This invention relates to computer intelligent question answering technology, and more particularly to a structured multimodal response method and device for large language models in vertical domains. Background Technology

[0002] In recent years, with the rapid development of Large Language Models (LLMs) in fields such as natural language processing, question answering, intelligent writing, and decision support, significant progress has been made in language understanding and generation capabilities in general scenarios. However, in applications within vertical domains (such as humanities, ethics, medicine, law, engineering, and finance), general-purpose LLMs often face challenges such as a lack of domain knowledge, insufficient structured processing capabilities, and inaccurate contextual understanding. Vertical domain tasks typically rely on extensive professional background knowledge, complex logical reasoning chains, and verifiable factual evidence. Relying solely on the corpus and reasoning capabilities of general-purpose models often fails to meet user expectations under requirements of high accuracy, high professionalism, and high traceability. To address these issues, the industry generally adopts a technical approach of fine-tuning vertical domain models and enhancing generation through retrieval.

[0003] 1. Data preparation and pre-training modification: Collect and clean data from vertical domains and mix them with general corpora for training to ensure a balance between domain capabilities and general capabilities and avoid "catastrophic forgetting".

[0004] 2. Fine-tuning and instruction data construction: Incremental pre-training or supervised fine-tuning of the base model is performed by manually writing or generating instruction sets from large models to improve its performance in domain tasks.

[0005] 3. Enhanced Retrieval Generation: During the model reasoning stage, the model's contextual understanding ability is enhanced by combining the knowledge base retrieval results and matching the most relevant text blocks or knowledge fragments.

[0006] While this method is relatively mature in its overall system, it still has shortcomings such as insufficient structure awareness (segmentation across chapters), inconsistent tagging system, uncontrolled generation in no-hit scenarios, and unverifiable output.

[0007] In actual deployment scenarios, the interaction needs of vertical fields are diverse, including knowledge Q&A for single facts, as well as case analysis involving multiple theoretical perspectives, cross-role analysis, and interpretation of institutional background.

[0008] However, current technologies generally suffer from the following pain points: all inputs are processed according to a single question-and-answer logic; the retrieval channel is limited, with most relying solely on semantic vector matching; illusory information is easily introduced in scenarios with no knowledge hit or low confidence, reducing the accuracy of the answer; and the academic rigor and verifiability of the output answer are insufficient. Summary of the Invention

[0009] To address the problems existing in the prior art, the purpose of this invention is to provide a more accurate structured multimodal response method and device for large language models in vertical domains.

[0010] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0011] A structured multimodal response method for large language models in vertical domains includes the following steps:

[0012] (1) The problem of obtaining natural language input from users;

[0013] (2) Convert the natural language question input by the user into a high-dimensional semantic vector, identify the task mode to which the natural language question belongs, and extract the parameters corresponding to the task mode from the stored task mode-parameter mapping table. The parameters include fusion weight coefficient, confidence threshold, inference rules and domain structure template.

[0014] (3) Calculate the similarity between the natural language problem and the set of structural labels and the high-dimensional semantic vector of each knowledge fragment in the knowledge base, and fuse them based on the corresponding fusion weight coefficient to obtain the similarity score between each knowledge fragment and the natural language problem. Store the knowledge fragments with the highest similarity scores into the candidate knowledge fragment list. The knowledge base stores several knowledge fragments related to each task mode, as well as the set of structural labels and the high-dimensional semantic vector of each knowledge fragment.

[0015] (4) If the maximum similarity score in the candidate knowledge fragment list is greater than or equal to the confidence threshold, then the knowledge fragments with similarity scores greater than the confidence threshold are used as the answer reference text, and step (5) is executed directly. Otherwise, the answer to the natural language question is inferred based on the candidate knowledge fragment list according to the inference rules, the inference conclusion is used as the answer reference text, and the source information of the inference conclusion is marked, and then step (5) is executed.

[0016] (5) Generate answer content from the baseline text of the answer according to the domain structured template, and mark the source information.

[0017] Furthermore, step (2) specifically includes:

[0018] (2.1) Convert the natural language question q input by the user into a high-dimensional semantic vector with contextual semantic relevance. ;

[0019] (2.2) Transform high-dimensional semantic vectors Input a pre-trained task classifier to determine the task mode to which the natural language problem belongs:

[0020]

[0021] In the formula, High-dimensional semantic vector The task mode to which it belongs. Represents a task classifier. Indicates in Given a given condition, the probability that the task mode is m, where M represents the set of task modes;

[0022] (2.3) Based on the task mode to which the natural language problem belongs Find the task mode from the task mode-parameter mapping table. The corresponding parameters.

[0023] Furthermore, step (3) specifically includes:

[0024] (3.1) Extract a set of structural labels for natural language questions based on the task pattern, wherein the structural labels are structured identifiers used to characterize the structural features of the text in the whole text;

[0025] (3.2) Calculate the set similarity between the set of structure labels of the natural language problem and the set of structure labels of each knowledge fragment in the knowledge base; wherein, the knowledge base stores several knowledge fragments related to each task mode, as well as the set of structure labels of each knowledge fragment and a high-dimensional semantic vector with contextual semantic relevance;

[0026]

[0027] In the formula, The set of structural labels representing the natural language problem q The set of structural tags for the i-th knowledge fragment in the knowledge base set similarity, Represents a knowledge base. Indicates the number of elements in the set;

[0028] (3.3) Obtain the high-dimensional semantic vector of the natural language question and calculate the semantic similarity with the high-dimensional semantic vector of each knowledge fragment in the knowledge base:

[0029]

[0030] In the formula, A high-dimensional semantic vector representing a natural language problem q With the high-dimensional semantic vector of the i-th knowledge fragment semantic similarity, Represents the norm;

[0031] (3.4) Weighted fusion of set similarity and semantic similarity yields the similarity score between the natural language question and each knowledge fragment:

[0032]

[0033] In the formula, This represents the natural language question q and the i-th knowledge fragment in the knowledge base. Similarity score, Indicates the fusion weight coefficient;

[0034] (3.5) Sort the knowledge fragments from high to low according to their similarity scores, and store the top few knowledge fragments with the highest similarity scores in the candidate knowledge fragment list. .

[0035] Furthermore, step (4) specifically includes:

[0036] (4.1) Obtain the list of candidate knowledge fragments Maximum similarity score ,like Indicates the confidence threshold, then the list of candidate knowledge fragments will be... The knowledge snippets in the text are used as the baseline text for the answer, and source tags are added. Then proceed to step (5); otherwise proceed to step (4.2).

[0037] (4.2) Obtain the inference rules corresponding to the task mode, and input them into the inference function along with the natural language question q to obtain the inference conclusion;

[0038]

[0039] In the formula, Indicates an inferred conclusion. Represents the inference function. The inference rules are defined, including the logical scope and boundary conditions of the inference. This represents a set of facts, specifically a list of candidate knowledge fragments. ;

[0040] (4.3) Add source tags to the inference conclusions List of candidate knowledge fragments Add tags to knowledge fragments This generates a baseline text for the response after source differentiation, and records the source information.

[0041] Furthermore, step (5) specifically includes:

[0042] (5.1) Obtain the domain structure template corresponding to the task mode;

[0043] (5.2) Extract source tags from the baseline text of the answer Part of the content is used as an inference, source tags The content is treated as the factual part, and the inference part and the factual part are embedded in the corresponding positions according to the requirements of the domain structured template, and the source information identifier is added to each key point;

[0044] (5.3) The filled domain structure template is used to form a structured and verifiable text output, and the output text and all source information are recorded in the system log.

[0045] Furthermore, the task classifier is specifically a multilayer perceptron, used to determine the task mode to which the input belongs.

[0046] Furthermore, the inference function is specifically a large-scale language model inference mechanism based on Chain-of-Thought (CoT) prompts, which uses a set of facts... with inference rules The information is structured into a prompt template containing background knowledge and logical constraints, which is then input into a large language model to guide it in generating a step-by-step reasoning process and outputting the final inference conclusion. .

[0047] A computer program product includes a computer program that, when executed by a processor, implements the above-described method.

[0048] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described above.

[0049] A computer-readable storage medium having a computer program stored thereon, the computer program implementing the above-described method when executed by a processor.

[0050] Compared with the prior art, the beneficial effects of this invention are:

[0051] 1. Automatic task type recognition and intelligent mode switching: The task classifier automatically maps user input to a set of preset task modes, enabling real-time switching between multiple modes. The mode parameters and confidence scores are embedded in the configuration object to provide the optimal matching processing path for subsequent retrieval and generation, thereby improving accuracy.

[0052] 2. Dual-channel weighted fusion retrieval mechanism: The dual-channel parallel and fusion of structural tag retrieval and semantic vector retrieval is adopted to improve the hit rate and context integrity across chapters and multiple levels, solve the problem of insufficient coverage and context gaps in single-channel matching, and improve accuracy;

[0053] 3. Controllable inference generation and source differentiation mechanism for low-hit scenarios: Define a confidence threshold, and make inferences based on the known set of facts and inference rules in low-hit scenarios. In the output, the inference and facts are clearly distinguished by the source, which effectively suppresses the risk of "illusion" and improves accuracy.

[0054] 4. Domain-specific structured output templates and end-to-end traceability annotations: The corresponding domain-specific structured templates are invoked according to the task mode, and the source is embedded in the output to realize verifiable and citationable academic and business applications, thereby improving credibility.

[0055] Based on the above analysis, this invention can achieve the dual goals of high precision and high reliability in vertical domain tasks. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating the structured multimodal response method for large language models oriented towards vertical domains provided in the embodiments of the invention.

[0057] Figure 2 This is a schematic diagram of the pattern recognition determination process proposed in an embodiment of the present invention;

[0058] Figure 3 This is a flowchart of the task type identification and pattern selection process proposed in an embodiment of the present invention;

[0059] Figure 4 This is a flowchart of the dual-channel fusion retrieval process proposed in an embodiment of the present invention;

[0060] Figure 5 This is a flowchart illustrating the controllable inference and source differentiation process proposed in this embodiment of the invention;

[0061] Figure 6 This is a flowchart of the structured response and source tracing annotation output proposed in an embodiment of the present invention. Detailed Implementation

[0062] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0063] Example 1

[0064] This invention provides a structured multi-modal response method for large language models in vertical domains, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0065] (1) The problem of obtaining natural language input from users.

[0066] (2) Convert the natural language question input by the user into a high-dimensional semantic vector, identify the task mode to which the natural language question belongs, and extract the parameters corresponding to the task mode from the stored task mode-parameter mapping table.

[0067] like Figure 3 As shown, this step specifically includes:

[0068] (2.1) Convert the natural language question q input by the user into a high-dimensional semantic vector with contextual semantic relevance. The conversion process employs an encoder function based on a pre-trained language model. accomplish;

[0069] (2.2) Transform high-dimensional semantic vectors Input a pre-trained task classifier to determine the task mode to which the natural language problem belongs:

[0070]

[0071] In the formula, High-dimensional semantic vector The task mode to which it belongs. This represents a task classifier, specifically a multilayer perceptron (MLP) with a self-attention mechanism. Indicates in Given a task pattern m, M represents the set of task patterns; M = {FactQA, CaseAnalysis, Impference, GeneeralQA}, where FactQA, CaseAnalysis, Impference, and GeneeralQA represent fact-based question answering, case analysis, controlled inference, and non-domain short answering, respectively.

[0072] (2.3) Based on the task mode to which the natural language problem belongs Find the task mode from the task mode-parameter mapping table. The corresponding parameters include fusion weight coefficients, confidence thresholds, inference rules (including inference priority, inference permission, and scope restrictions), and domain structure templates.

[0073] The core of this step lies in semantic parsing and structured feature extraction of the natural language questions input by users. By using a trained task classifier, the task type (fact-based question answering, case analysis, controllable inference, non-domain short answer, etc.) is automatically determined, thereby triggering the corresponding matching algorithm and output template in the subsequent retrieval and generation process. This achieves optimal matching between task patterns and information processing paths, significantly improving the relevance, structure, and accuracy of question answering.

[0074] (3) Calculate the similarity between the natural language question and the set of structural labels and high-dimensional semantic vectors of each knowledge fragment in the knowledge base, and fuse them based on the corresponding fusion weight coefficients to obtain the similarity score between each knowledge fragment and the natural language question. Store the knowledge fragments with the highest similarity scores (e.g., the first three) into the candidate knowledge fragment list.

[0075] The knowledge base stores several knowledge fragments related to each task mode, along with a set of structural tags and a high-dimensional semantic vector for each knowledge fragment. The knowledge fragments are verified facts, possessing authority and verifiability. The structural tags are structured identifiers used to characterize the structural features of the text within the entire document, such as chapter numbers, legal clause numbers, and disease classification numbers. The high-dimensional semantic vectors are high-dimensional floating-point arrays that preserve the contextual relevance of the content in the semantic space and can be used for nearest neighbor search in the vector database. Specifically, the high-dimensional semantic vectors of the knowledge fragments can be extracted using an encoder embedding function.

[0076] like Figure 4 As shown, step (3) specifically includes:

[0077] (3.1) Extract the set of structural labels for the natural language problem based on the task mode to which the natural language problem belongs;

[0078] (3.2) Calculate the set similarity between the set of structure labels for a natural language question and the set of structure labels for each knowledge fragment in the knowledge base;

[0079]

[0080] In the formula, The set of structural labels representing the natural language problem q The set of structural tags for the i-th knowledge fragment in the knowledge base set similarity, Represents a knowledge base. Indicates the number of elements in the set;

[0081] (3.3) Obtain the high-dimensional semantic vector of the natural language question and calculate the semantic similarity with the high-dimensional semantic vector of each knowledge fragment in the knowledge base:

[0082]

[0083] In the formula, A high-dimensional semantic vector representing a natural language problem q With the high-dimensional semantic vector of the i-th knowledge fragment semantic similarity, Represents the norm;

[0084] (3.4) Weighted fusion of set similarity and semantic similarity yields the similarity score between the natural language question and each knowledge fragment:

[0085]

[0086] In the formula, This represents the natural language question q and the i-th knowledge fragment in the knowledge base. Similarity score, Indicates the fusion weight coefficient;

[0087] (3.5) Sort the knowledge fragments from high to low according to their similarity scores, and store the top few knowledge fragments with the highest similarity scores in the candidate knowledge fragment list. .

[0088] This step aims to simultaneously utilize structural tag retrieval (discrete matching based on knowledge base chapter tags) and semantic vector retrieval (continuous spatial matching based on embedding models) to recall task-related candidate knowledge fragments from the knowledge base, and to mathematically optimize the relevance and coverage of the retrieval results through a weighted ranking fusion algorithm, thereby significantly improving the hit rate and contextual integrity of cross-chapter and multi-level information.

[0089] (4) If the maximum similarity score in the candidate knowledge fragment list is greater than or equal to the confidence threshold, then the knowledge fragments with similarity scores greater than the confidence threshold are used as the answer reference text, and step (5) is executed directly. Otherwise, the answer to the natural language question is inferred based on the candidate knowledge fragment list according to the inference rules, the inference conclusion is used as the answer reference text, and the source information of the inference conclusion is marked, and then step (5) is executed.

[0090] like Figure 5 As shown, step (4) specifically includes:

[0091] (4.1) Obtain the list of candidate knowledge fragments Maximum similarity score ,like , Indicates the confidence threshold, then the list of candidate knowledge fragments will be... Knowledge fragments with similarity scores greater than the confidence threshold are used as the baseline text for the answer, and source tags are added. Then proceed to step (5); otherwise proceed to step (4.2).

[0092] (4.2) Obtain the inference rules corresponding to the task mode, and input them into the inference function along with the natural language question q to obtain the inference conclusion;

[0093]

[0094] In the formula, Indicates an inferred conclusion. Represents the inference function. The inference rules are defined, including the logical scope and boundary conditions of the inference. This represents a set of facts, specifically a list of candidate knowledge fragments. The derivation process of the inference function is defined by the mapping function and constraints to ensure that the generated content is closed within the logical space. Specifically, the inference function is a large-scale language model reasoning mechanism based on Chain-of-Thought (CoT) prompts, which integrates the fact set... with inference rules The information is structured into a prompt template containing background knowledge and logical constraints, which is then input into a large language model to guide it in generating a step-by-step reasoning process and outputting the final inference conclusion. ;

[0095] (4.3) Add labels to the inference conclusions List of candidate knowledge fragments Add source tags to knowledge fragments This forms the baseline text for the responses after source differentiation. It also records source information, including source path, chapter index, and tag set, for subsequent verification.

[0096] In cases where the confidence level of the search results is insufficient, this step executes a controlled inference generation process and clearly distinguishes between content sources based on knowledge base facts and those based on logical inference. By constraining the logical scope and citation boundaries of the generation process, it effectively suppresses the "illusion" phenomenon and ensures the verifiability and compliance of the answer content. This is especially suitable for high-risk vertical fields such as medicine, law, and ethics.

[0097] (5) Generate answer content from the baseline text of the answer according to the domain structured template, and mark the source information.

[0098] like Figure 6 As shown, step (5) specifically includes:

[0099] (5.1) Obtain the domain structure template corresponding to the task mode; for example, the factual question and answer mode adopts the "conclusion first + point explanation" structure, and the case analysis mode adopts the six-dimensional structure (background - conflict point - multi-theory analysis - stakeholders - institutional background - comprehensive judgment and suggestions) to ensure that the template is highly matched with the task mode;

[0100] (5.2) Extract source tags from the baseline text of the answer Part of the content is used as an inference, source tags The content is treated as the factual part. The inference part and the factual part are embedded in the corresponding positions (paragraphs and sub-points) according to the requirements of the domain structured template. The logical hierarchy is clear and the sentence structure conforms to the academic expression requirements of the domain. Source information identifiers (such as ID number, original knowledge base chapter path, tag set) are added to each key point.

[0101] (5.3) The populated domain structure template is transformed into structured, verifiable text output, ensuring a fixed content format that can be automatically parsed by the program. The output text and all source information are recorded in the system log to support subsequent auditing and performance analysis. The final generated structured text response object, containing factual and inference content, full-link source annotations, and a fixed structure template format, is returned through the user interface, and logs are simultaneously recorded for performance evaluation and knowledge base maintenance.

[0102] This step calls the corresponding domain-specific structured output template according to the task mode, organically combines factual and inferential content, generates the final answer according to the academic and structured standards of professional scenarios, and embeds source annotation information throughout the entire chain to ensure the traceability and compliance of the output content, making it directly usable in high-requirement scenarios such as academia, law, and ethics.

[0103] The following comparison between this invention and some other patents in the same field demonstrates the innovativeness of this invention through the differences and innovations found:

[0104] Comparative Example 1: Patent CN 120069098 A: "A method, apparatus, and medium for fine-tuning a large-scale model in a vertical domain." This solution constructs and optimizes a large-scale model for a specific vertical domain. The core process is as follows: Knowledge Graph Triple Extraction: Extracting subject-relation-predicate triples from the knowledge graph of the specified domain; Prompt Engineering: Guiding the large-scale model based on preset prompts, converting triples into natural language question-answer pairs while preserving information integrity as much as possible; Completion and Quality Scoring: Using masked reasoning to complete information on some triples, and generating quality scores through similarity scoring; Data Filtering and Fine-tuning: Filtering question-answer pairs with quality scores exceeding a threshold as training data to fine-tune the large-scale model in the vertical domain.

[0105] The limitations of this patent include: heavy reliance on the quality and coverage of knowledge graph triples; once the domain documents are unstructured long texts (legal precedents, ethics research reports, etc.), the efficiency and accuracy of triple extraction will significantly decrease; it cannot handle cross-chapter and cross-paragraph contextual relationships, and its performance is insufficient for tasks requiring multi-source information integration (such as case analysis and legal application analysis); it has a single mode, only able to generate a single question-and-answer logic, and does not support automatic switching for different task types (fact-based question answering, structured case analysis, inference generation); it lacks inference and illusion control mechanisms in low-confidence scenarios, and once the knowledge base is not matched, the system can hardly maintain high-quality output; it ignores the tag system, original text positioning, and traceability, and the final answer cannot be directly cited or verified in high-requirement situations such as academia and law.

[0106] Differences and innovations: This invention does not rely on a single triple extraction. It has directory awareness and semantic hybrid segmentation for unstructured long documents, and is equipped with task pattern recognition, dual retrieval channels, low hit inference control and source labeling, and the whole chain is traceable.

[0107] Comparative Example 2: Patent CN 118260393 B: "A Question Answering Method and System Based on a Large Model for Vertical Domains". This scheme consists of three main stages: Incremental pre-training: Selecting a base model and training it by mixing general corpora with vertical domain datasets (approximately a 1:5 ratio) to avoid degradation of general capabilities; Instruction fine-tuning (SFT): Segmenting long texts into short texts according to semantics or rules to generate instruction sets, and then using GPT to enhance and generate more instruction sets for model fine-tuning; Retrieval-enhanced generation (RAG): By constructing a parent-child document structure (main document block + child document block), when a user's question matches a child document block, the content of the main document block and the question are sent to the model to generate the answer.

[0108] The limitations of this invention are as follows: Although it uses parent-child block division, the two-layer structure is only maintained in the retrieval stage, while the generation stage still follows a unified question-and-answer mode, making it unable to intelligently identify task types and customize the output structure; the retrieval relies solely on single-channel similarity matching, lacks structural tag fusion, and does not employ multi-source weighted sorting, resulting in limited cross-chapter / cross-topic hit rates; instruction set generation relies on GPT or local models, and its quality is greatly affected by the stability of the prompt, making it uncontrollable and lacking in verifiability; there is no confidence-driven inference mode, and when knowledge is not matched, the output quality heavily relies on model illusions, making it difficult to guarantee factual accuracy; the output does not embed source IDs, chapter paths, or other traceability information, making it impossible to verify in academic and legal scenarios.

[0109] Differences and Innovations: This invention not only utilizes a dual-channel retrieval system of directory awareness and semantic tags, but also automatically switches between task modes during the generation stage to maintain structured output, and embeds source labeling and anti-illusion mechanisms throughout the process.

[0110] Comparative Example 3: Patent CN 119150998 A: "A Multimodal RAG Knowledge Question Answering Method and Device Applied to Vertical Domains". This solution supports multimodal inputs such as text, images, audio, and video. The process includes: Multimodal conversion: Non-text data is converted into text data through a preset model; Knowledge fragment retrieval: Multiple retrieval models are used for parallel retrieval (such as BM25, vector retrieval), and the ranking results are fused (RRF inverse ranking fusion algorithm); Weight re-ranking: The ranking weights are adjusted based on preset knowledge frequency information to obtain a second ranking list; Reward model reasoning: The best knowledge fragments are combined from the ranking list to generate a prompt, and the highest-scoring prompt is selected using a reward model; LLM output generation: An answer is generated based on the highest-scoring prompt, and similar question-and-answer options can be provided if the user's needs are not met.

[0111] The limitations of this patent are as follows: the search results are only enhanced in terms of ranking fusion, but there is no precise matching based on structural tags, making it difficult to achieve accurate chapter-level citations; after multimodal data fusion, it only participates in unified question-and-answer generation and cannot select the optimal structured output for different task types; the task mode determination is missing, and case analysis questions and factual questions and answers use the same generation template, resulting in fragmented answers and insufficient academic rigor; although there is a reward model to filter prompts, it does not combine confidence with the distinction between "knowledge base-based" and "inference-based" sources, resulting in low verifiability of the generated data; and the source path and tag set are not embedded in the output, limiting the traceability capability in professional scenarios.

[0112] Differences and innovations: This invention combines task mode switching, with the retrieval end using dual-channel weighting of structural tags and semantic vectors, and the generation end selecting a six-dimensional analysis or conclusion-first template according to the mode, while locking the source annotation, thus improving academic and business usability.

[0113] Example 2

[0114] This invention also provides a computer program product, such as an app on a mobile phone or tablet, or an installer on a computer. This product includes a computer program / instructions that, when executed by a processor, implement the method described in Embodiment 1. The code for the computer-executable program used to perform the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0115] Example 3

[0116] This invention provides a computer device that provides services for implementing the method described in Embodiment 1. The device may include: a memory storing a computer-executable program; a processor coupled to the memory; and the processor calling the computer-executable program stored in the memory to execute the steps of the method described in Embodiment 1.

[0117] The memory may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the memory may be used to read and write non-removable, non-volatile magnetic media (commonly referred to as a "hard disk drive"). A program / utility having a set (at least one) of program modules may be stored in, for example, memory. Such program modules include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The computer-executable program of the program modules typically performs the functions and / or methods described in the embodiments of the present invention.

[0118] The processor executes various functional applications and data processing by running programs stored in memory, such as the method provided in Embodiment 1 of the present invention.

[0119] The code of a computer executable program can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages.

[0120] Example 4

[0121] This invention provides a storage medium containing a computer-executable program, which, when executed by a computer processor, is used to perform the method of Embodiment 1.

[0122] The storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0123] Of course, the computer-executable program in the storage medium provided in the embodiments of the present invention is not limited to the above-described method operations, but can also perform related operations in the methods provided in any embodiment of the present invention.

[0124] It should be understood that the embodiments and descriptions above are only the principles, main features and advantages of the present invention. Various changes and modifications can be made to the present invention without departing from the spirit and scope of the invention, and all such changes and modifications fall within the protection scope of the present invention.

Claims

1. A structured multi-modal response method for large language models in vertical domains, characterized in that, Includes the following steps: (1) The problem of obtaining natural language input from users; (2) Convert the natural language question input by the user into a high-dimensional semantic vector, identify the task mode to which the natural language question belongs, and extract the parameters corresponding to the task mode from the stored task mode-parameter mapping table. The parameters include fusion weight coefficient, confidence threshold, inference rules and domain structure template. (3) Calculate the similarity between the natural language problem and the set of structural labels and the high-dimensional semantic vector of each knowledge fragment in the knowledge base, and fuse them based on the corresponding fusion weight coefficient to obtain the similarity score between each knowledge fragment and the natural language problem. Store the knowledge fragments with the highest similarity scores into the candidate knowledge fragment list. The knowledge base stores several knowledge fragments related to each task mode, as well as the set of structural labels and the high-dimensional semantic vector of each knowledge fragment. (4) If the maximum similarity score in the candidate knowledge fragment list is greater than or equal to the confidence threshold, then the knowledge fragments with similarity scores greater than the confidence threshold are used as the answer reference text, and step (5) is executed directly. Otherwise, the answer to the natural language question is inferred based on the candidate knowledge fragment list according to the inference rules, the inference conclusion is used as the answer reference text, and the source information of the inference conclusion is marked, and then step (5) is executed. (5) Generate answer content from the baseline answer text according to the domain structured template, and annotate the source information; Step (3) specifically includes: (3.1) Extract the set of structural labels for the natural language problem according to the task mode to which the natural language problem belongs, wherein the structural labels are structured identifiers used to characterize the structural features of the text in the whole text; (3.2) Calculate the set similarity between the set of structure labels for a natural language question and the set of structure labels for each knowledge fragment in the knowledge base; ， In the formula, The set of structural labels representing the natural language problem q The set of structural labels for the i-th knowledge fragment in the knowledge base set similarity, Represents a knowledge base. Indicates the number of elements in the set; (3.3) Obtain the high-dimensional semantic vector of the natural language question and calculate the semantic similarity with the high-dimensional semantic vector of each knowledge fragment in the knowledge base: ， In the formula, A high-dimensional semantic vector representing a natural language problem q With the high-dimensional semantic vector of the i-th knowledge fragment semantic similarity, Represents the norm; (3.4) Weighted fusion of set similarity and semantic similarity yields the similarity score between the natural language question and each knowledge fragment: ， In the formula, This represents the natural language question q and the i-th knowledge fragment in the knowledge base. Similarity score, Indicates the fusion weight coefficient; (3.5) Sort the knowledge fragments from high to low according to their similarity scores, and store the top few knowledge fragments with the highest similarity scores in the candidate knowledge fragment list. .

2. The structured multi-modal response method for large language models oriented towards vertical domains as described in claim 1, characterized in that, Step (2) specifically includes: (2.1) Convert the natural language question q input by the user into a high-dimensional semantic vector with contextual semantic relevance. ; (2.2) Transform high-dimensional semantic vectors Input a pre-trained task classifier to determine the task mode to which the natural language problem belongs: ， In the formula, High-dimensional semantic vector The task mode to which it belongs. Represents a task classifier. Indicates in Given a given condition, the probability that the task mode is m, where M represents the set of task modes; (2.3) Based on the task mode to which the natural language problem belongs Find the task mode from the task mode-parameter mapping table. The corresponding parameters.

3. The structured multi-modal response method for large language models oriented towards vertical domains as described in claim 1, characterized in that, Step (4) specifically includes: (4.1) Obtain the list of candidate knowledge fragments Maximum similarity score ,like , Indicates the confidence threshold, then the list of candidate knowledge fragments will be... Knowledge fragments with similarity scores greater than the confidence threshold are used as the baseline text for the answer, and source tags are added. Then proceed to step (5); otherwise proceed to step (4.2). (4.2) Obtain the inference rules corresponding to the task mode, and input them into the inference function along with the natural language question q to obtain the inference conclusion; ， In the formula, Indicates an inferred conclusion. Represents the inference function. The inference rules are defined, including the logical scope and boundary conditions of the inference. This represents a set of facts, specifically a list of candidate knowledge fragments. ; (4.3) Add source tags to the inference conclusions List of candidate knowledge fragments Add source tags to knowledge fragments This generates a baseline text for the response after source differentiation, and records the source information.

4. The structured multi-modal response method for large language models oriented towards vertical domains as described in claim 1, characterized in that, Step (5) specifically includes: (5.1) Obtain the domain structure template corresponding to the task mode; (5.2) Extract source tags from the baseline text of the answer Part of the content is used as an inference, source tags The content is treated as the factual part, and the inference part and the factual part are embedded in the corresponding positions according to the requirements of the domain structured template, and the source information identifier is added to each key point; (5.3) The filled domain structure template is used to form a structured and verifiable text output, and the output text and all source information are recorded in the system log.

5. The structured multi-modal response method for large language models oriented to vertical domains according to claim 2, wherein the task classifier is specifically a multilayer perceptron used to determine the task mode to which the input belongs.

6. The structured multi-modal response method for large language models oriented towards vertical domains as described in claim 3, wherein the inference function is specifically a large language model inference mechanism based on chain-like thinking prompts, which inference mechanism integrates the fact set... with inference rules The information is structured into a prompt template containing background knowledge and logical constraints, which is then input into a large language model to guide it in generating a step-by-step reasoning process and outputting the final inference conclusion. .

7. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, it implements the method of any one of claims 1-6.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor executes the computer program to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program, when executed by a processor, implements the method of any one of claims 1-6.

Images