A large model question and answer method and device based on bidirectional sorting and a storage medium

Abstract

This invention discloses a large-scale question-answering method, apparatus, and storage medium based on bidirectional ranking. The method includes: generating a first candidate logical form list for a natural language target question using a large language model; transcribing each candidate logical form into a corresponding natural language question using a transcription model; calculating the semantic similarity between each transcribed question and the natural language target question; reordering the candidate logical forms according to the semantic similarity to obtain a second candidate logical form list; fusing the first and second candidate logical form lists using a credibility-aware Polda fusion method to obtain a third candidate logical form list; and querying the answer corresponding to the target question in a knowledge base based on the third candidate logical form list. While ensuring candidate coverage, this method significantly improves the position of the most valuable logical form in the fused list, effectively enhancing the efficiency and accuracy of large-scale question-answering.

Description

Technical Field

[0001] This invention relates to the field of natural language large model question answering technology, and in particular to a large model question answering method, apparatus and storage medium based on bidirectional ranking. Background Technology

[0002] For a long time, knowledge-based question answering (KBQA) based on large models has been a core technology for interpreting and responding to user queries due to its structured knowledge reserves, and has received widespread attention in both academia and industry. Typical knowledge bases such as Freebase, Wikidata, and DBpedia are based on knowledge graphs and store information in the form of triples.

[0003] Semantic parsing, as one of the core methods of large-scale question answering, aims to effectively transform natural language questions into logical forms to support answer generation, and has achieved considerable success in recent years. Early work involved translating questions into intermediate logical forms, such as S-expressions, before execution.

[0004] Recently, large language models such as Codex and GPT-4 have demonstrated powerful context learning capabilities. After learning from a small number of examples (natural language target question, logical form), they can perform complex reasoning tasks based on the natural language target question. Previous methods typically consisted of two stages: first, a list of candidate logical forms was generated using the large language model; then, through entity and relation links, an executable query language corresponding to each candidate logical form was obtained. The answer selection employed a self-consistency principle, i.e., after executing the query languages ​​of all candidate logical forms, a majority voting strategy was used to determine the final answer; or, a first validity principle was used, sequentially querying the knowledge base for each current candidate logical form according to its ranking list, stopping and outputting the current answer as soon as a non-empty answer was returned.

[0005] However, the above methods face significant challenges in practical applications. While the self-consistency principle can achieve high accuracy through majority voting, it requires executing all candidate logical forms, typically involving thousands of queries, leading to wasted computational resources and slow response times, failing to meet the needs of real-time applications. Although the first validity principle reduces time costs, its effectiveness is highly dependent on the initial ranking quality of the candidate logical form list. When the "golden logical form" ranks low in the initial list, the system may prematurely adopt incorrect answers, resulting in a decrease in question-answering accuracy. Analysis of the closest prior art reveals that, for example, Chinese Patent Application No. 202510892622.8 discloses a question-answering method and system for a large language model knowledge graph that combines semantic correction. Although it provides a post-event correction mechanism for logical forms, its initial execution stage still directly relies on the ranking of the candidate logical form list generated by the large language model, lacking forward-looking optimization of the ranking quality of candidate logical forms.

[0006] The core problem in the existing technology is that there is a lack of a mechanism to actively optimize and semantically verify the candidate logical form list before execution. Specifically, it is manifested in the following ways: (1) it is impossible to effectively verify the reliability of the initial candidate logical form list in order of avoiding a large number of invalid executions; (2) there is a lack of sorting optimization methods specifically for structured logical forms. In particular, when fusing multi-dimensional sorting signals such as generation probability and semantic similarity, the existing methods usually adopt static or fixed fusion strategies, which cannot be dynamically adjusted according to the specific question context and the real-time credibility of the model output. As a result, the final candidate logical form list sorting fails to reach the optimal level, thus restricting the further improvement of question answering efficiency and accuracy. Summary of the Invention

[0007] The purpose of this invention is to provide a large-scale question-answering method, apparatus, and storage medium based on bidirectional sorting, so as to at least solve the technical problem that traditional question-answering methods in the prior art cannot simultaneously achieve both execution efficiency and question-answering accuracy.

[0008] According to one aspect of the present invention, a large-scale model question answering method based on bidirectional ranking is provided, comprising: acquiring a natural language target question input by a user, and generating a first candidate logical form list of the natural language target question using a pre-trained large-scale language model; using a pre-trained transcription model to transcribe each candidate logical form in the first candidate logical form list into a corresponding natural language question, thereby obtaining a transcription question list; calculating the semantic similarity between each natural language question in the transcription question list and the natural language target question; reordering each candidate logical form in the first candidate logical form list according to the semantic similarity, thereby obtaining a second candidate logical form list; fusing the first candidate logical form list and the second candidate logical form list to obtain a third candidate logical form list; and querying an answer corresponding to the natural language target question in a knowledge base according to the third candidate logical form list.

[0009] According to another aspect of the present invention, a computer-readable storage medium is also provided, the storage medium including a stored program, wherein the large model question-answering method based on bidirectional ranking is executed by a processor when the program is run.

[0010] According to another aspect of the present invention, a large-scale model question answering device based on bidirectional ranking is also provided, comprising: a first generation module, configured to acquire a natural language target question input by a user and generate a first candidate logical form list of the natural language target question using a pre-trained large-scale language model; a transcription module, configured to transcribe each candidate logical form in the first candidate logical form list into a corresponding natural language question using a pre-trained transcription model, thereby obtaining a transcribed question list; a similarity calculation module, configured to calculate the semantic similarity between each natural language question in the transcribed question list and the natural language target question; a second generation module, configured to reorder each candidate logical form in the first candidate logical form list according to the semantic similarity, thereby obtaining a second candidate logical form list; a third generation module, configured to merge the first candidate logical form list and the second candidate logical form list, thereby obtaining a third candidate logical form list; and a query module, configured to query an answer corresponding to the natural language target question in a knowledge base according to the third candidate logical form list.

[0011] Furthermore, in the third generation module, the first candidate logical form list and the second candidate logical form list are fused using a credibility-aware Polda fusion method.

[0012] According to another aspect of the present invention, a large model question answering apparatus based on bidirectional ranking is also provided, comprising: a processor and a memory; the processor executes instructions of the large model question answering method based on bidirectional ranking.

[0013] This application constructs a bidirectional semantic verification mechanism and a multi-dimensional ranking fusion mechanism in a large-scale model question answering scenario. Specifically, it achieves precise optimization of the candidate logical form ranking list through the following steps: First, a first candidate logical form list for the natural language target question is generated based on a large language model; then, the candidate logical forms are reverse-converted into natural language questions using a transcription model to construct a transcription question set corresponding to the natural language target question; further, semantic similarity calculation is used to evaluate the semantic matching degree between each natural language question in the transcription question set and the natural language target question, and the first candidate logical form list is ranked according to semantic consistency to form a semantically enhanced second candidate logical form list; finally, a credibility-aware Porta fusion method is used to fuse the ranking weights of the first and second candidate logical form lists to generate a third candidate logical form list that balances generation quality and semantic consistency, and a knowledge base query is performed based on this list.

[0014] Compared with the prior art, the beneficial effects of the present invention are:

[0015] The innovation of the bidirectional semantic verification mechanism in this invention lies in its use of a closed-loop architecture of "generation-transcription-verification" (candidate logical form → natural language question → semantic similarity evaluation) to verify the semantic fidelity of the generated results from the large language model. The innovation of the multi-dimensional ranking and fusion mechanism lies in its fusion of the first candidate logical form list (ranked based on generation probability) and the second candidate logical form list (ranked based on semantic similarity). This preserves the diversity of the initial candidate logical form list while strengthening the ranking priority of the golden logical form through semantic similarity. Thus, while ensuring candidate coverage, it significantly improves the position of the golden logical form in the fusion list, effectively enhancing the efficiency and accuracy of large-scale model question answering, thereby solving the technical problem in existing large-scale model question answering methods that cannot simultaneously achieve both execution efficiency and question answer accuracy.

[0016] This invention constructs a "bidirectional semantic verification" mechanism to cross-validate semantics from both positive and negative directions. This provides a high-quality, semantically consistent ranking list of candidate logical forms for subsequent fusion, resulting in a significant improvement in question-answering performance. Experiments show that, thanks to this mechanism, on complex CWQ and WebQSP datasets, this invention achieves an average 4.8% improvement in F1 score compared to existing technologies (such as KB-Coder and Interactive-KBQA). In terms of execution efficiency, the answer retrieval time is reduced by an average of 13.7% on both datasets.

[0017] 3. The reliability-aware fusion method used in this invention achieves results surpassing simple sorting and stacking, and its advantages are specifically reflected in:

[0018] Dynamic optimization and precise weighting: Compared to traditional static fusion methods, this method intelligently perceives the context of each question and answer. Specifically, when the average generation confidence of candidate logical forms generated by the large language model is high, the system assigns higher weights to the initial ranking, effectively relying on the model's "intuition." Conversely, when the semantic similarity ranking list obtained through bidirectional verification shows significant differentiation, the ranking of candidates with high semantic consistency is strengthened. This dynamic weighting mechanism can make judgments like an experienced expert, ensuring that the "golden logical form" is accurately promoted to the top of the list whenever any ranking signal is particularly reliable, thereby directly improving the accuracy and efficiency of predicted answers.

[0019] More robust to complex scenarios: The large-model question answering method of this invention has a more prominent advantage when dealing with the following complex problems:

[0020] To address the issue of model "hesitation": when large language models generally generate low confidence levels for certain complex or uncommon questions, traditional methods relying on initial ranking are prone to failure. However, the question-answering method of this invention automatically reduces the weight of unreliable initial ranking, instead relying more on the results of the second candidate logical form list generated by bidirectional semantic verification. Utilizing high-precision semantic similarity as a safety net, it effectively corrects model bias and maintains the system's accuracy.

[0021] To address the issue of "low semantic discriminability": When the target question has multiple reasonable logical representations, resulting in multiple transcribed natural language questions having very high and close semantic similarity scores to the natural language target question, and low dispersion in similarity scores, the question answering method of this invention can identify this "low discriminability" characteristic. Therefore, it prioritizes the ranking tendency of the first candidate logical form list based on the probability generated by the large language model, avoiding blindly selecting from several close options in the second candidate logical form list, thus improving the rationality of the decision.

[0022] In summary, this invention not only ensures the quality foundation of the candidate logical form list through bidirectional semantic verification, but also achieves precise fine-tuning of the final ranking through its core intelligent fusion algorithm (trustworthiness-aware Porta fusion method), thereby surpassing existing technologies in both overall performance and robustness in complex scenarios.

[0023] 4. This invention specifically trains a sequence-to-sequence (Seq2Seq) transcription model. The training data for this model consists of a high-quality parallel corpus composed of pairs of (second sample logical form, sample question). This is a dedicated transcription model. By supervising the fine-tuning of the language model on this specific task, the language model not only learns language generation patterns but also learns more deeply the precise semantic mapping rules from logical form to natural language questions. This ensures that the transcribed natural language question can highly reproduce the query intent of the logical form, providing a reliable foundation for subsequent semantic similarity evaluation. The transcription model of this invention ensures that the transcribed natural language question is semantically as equivalent as possible to the input second sample logical form, while also being a natural expression that is understandable to humans and conforms to language habits. By training the transcription model, the probability that a simple sequence-to-sequence model might generate grammatically correct but semantically distorted questions (e.g., incorrectly changing entity relationships or ignoring query conditions) is reduced, thereby mitigating the situation where subsequent semantic similarity calculations become meaningless. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0025] Figure 1 This is a flowchart illustrating an embodiment of the large-scale question-answering method based on bidirectional ranking of the present invention;

[0026] Figure 2 This is a hardware structure block diagram of an embodiment of the large model question answering device based on bidirectional sorting of the present invention, used to implement the large model question answering method based on bidirectional sorting of the present invention.

[0027] Figure 3 This is a schematic diagram of the structure of one embodiment of the large-scale question-answering device based on bidirectional sorting of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that comprises a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Terminology Explanation:

[0031] 1. Knowledge Base: A knowledge base (KB) is a collection containing multiple triples. s is an entity, r is a binary relation, and o can be an entity or a literal, where each triple represents a fact.

[0032] The formal definition of a knowledge base is:

[0033]

[0034] in, It is a set of entities, representing an entity set; It is a set of literals; It is a set of binary relations, representing a set of relations; K represents the knowledge base.

[0035] Entity set Each entity in Each entity has a unique MID (Machine Identifier, also known as an Entity Identifier) ​​and a corresponding surface name. For example, the machine identifier m.02r0hl7 corresponds to the "Afghan national anthem". (Relationship set) Each relation in It consists of multi-level tags, such as r = ''government.national_anthem_of_a_country.anthem'', where each "." separates different levels of tags. Furthermore, each literal... It is usually an integer, a floating-point number, or a date / time value.

[0036] 2. Logical form is the structured representation of a natural language question in a knowledge base, such as an S-expression. S-expressions will be used subsequently to represent... The logical form of the problem above. S-expressions use function operations based on set semantics and eliminate the use of variables. Because it balances readability and compactness, it is a suitable representation for KBQA tasks.

[0037] 3. For the KBQA task, given a natural language problem... Q and a knowledge base First of all Q Convert to logical form ,in It is a semantic parsing function that takes a natural language question as input and outputs the corresponding logical form. The logical form is merely an intermediate expression and cannot be directly used in the semantic parsing function. Therefore, the logical form needs to be converted to something that can be executed on top of it. SPARQL query language executed on ,Right now ,in It is a fixed transformation function. Finally, through the transformation function... implement Get the final answer set ,in It is the query execution function.

[0038] Example 1

[0039] According to this embodiment, an embodiment of a large model question-answering method based on bidirectional sorting is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0040] According to the first aspect of this embodiment, a large model question answering method based on bidirectional ranking is provided. Figure 1 A flowchart illustrating the method is shown, which includes:

[0041] S301: Obtain the natural language target question input by the user, and use a pre-trained large language model to generate a list of first candidate logical forms of the natural language target question;

[0042] S302: Using a pre-trained transcription model, each candidate logical form in the first candidate logical form list is transcribed into a corresponding natural language question to obtain a transcription question list;

[0043] S303: Calculate the semantic similarity between each natural language question in the transcription problem list and the natural language target question;

[0044] S304: Based on the semantic similarity, reorder each candidate logical form in the first candidate logical form list to obtain a second candidate logical form list;

[0045] S305: Using the credibility-aware Porta fusion method, the first candidate logical form list and the second candidate logical form list are fused to obtain the third candidate logical form list;

[0046] S306: Based on the third candidate logical form list, query the knowledge base for the answer corresponding to the natural language target question.

[0047] The method described in this embodiment can be executed on a server or similar computing device.

[0048] Example 2

[0049] Figure 2 A hardware block diagram of a large-scale question-answering device based on bidirectional ranking, according to one embodiment, is shown. Figure 2 As shown, the large-scale question-and-answer device may include one or more processors (such as processor s1, processor s2, ..., processor sp; in this embodiment, the number of processors is p, and the processors may include, but are not limited to, microprocessors (MCUs) or programmable logic devices (FPGAs), a memory 2 for storing data, a transmission device 3 for communication functions, and an input / output interface 4. The memory, transmission device, and input / output interface are connected to the processors via a bus 1. In addition, it may also include a display 7, a keyboard 6, and a cursor control device 5 connected to the input / output interface. Those skilled in the art will understand that... Figure 2 The structure shown is for illustrative purposes only and does not limit the structure of the large-scale question-and-answer device described above. For example, the large-scale question-and-answer device may also include components larger than... Figure 2 The more or fewer components shown, or having the same Figure 2 The different configurations shown.

[0050] It should be noted that the aforementioned one or more processors may generally be referred to herein as "data processing circuitry". This data processing circuitry may be wholly or partially embodied in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be wholly or partially integrated into any other element in the large model question-answering device. As involved in the embodiments of the present invention, this data processing circuitry acts as a processor control, for example, the selection of a variable resistor termination path connected to input / output interface 4.

[0051] The memory can be used to store software programs and modules of application software, such as program instructions 9 corresponding to the bidirectional ranking-based large model question-and-answer method in this embodiment of the invention, or data storage device 8 storing the aforementioned program instructions. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remotely located memories can be connected to the processor via a network. The network includes, but is not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0052] The transmission device 3 is used to receive or send data via a network. The network may specifically include a wireless network provided by a communications provider. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0053] The display 7 can be a touch-screen liquid crystal display (LCD) that allows users to interact with it through a user interface.

[0054] It should be noted here that in some optional embodiments, Figure 2 The large-scale question-and-answer device shown may also include hardware components (including circuitry), software components (including computer code stored on a computer-readable medium), or a combination of both hardware and software components. It should be noted that... Figure 2 This is just one instance of a specific particular instance, and is intended to illustrate the types of components that can exist in a large model question-and-answer device.

[0055] Example 3

[0056] In this embodiment, when a user inputs a natural language target question (such as "In which capital of Australia is the Victorian District Court located?") through a terminal, the large-scale question answering device invokes a pre-trained large-scale language model and utilizes the large-scale language model's context learning capability to generate a list of first candidate logical forms arranged in descending order of generation probability. This list covers the logical forms of S-expressions, enabling the rapid construction of a list of candidate logical forms containing the golden logical form, laying the foundation for subsequent optimization and querying.

[0057] Then, using a pre-trained transcription model, each candidate logical form in the first candidate logical form list is reverse-reconstructed into a natural language problem, resulting in a transcription problem list. For example, the three candidate logical forms in the first candidate logical form list generated by the trained large language model for the target question are converted into natural language questions through a transcription model. For instance, the first candidate logical form "(add(relation [location, location, contained in])(add[law, court jurisdiction, court][Victoria Magistrates' Court]))" is reverse-reconstructed into the natural language question "What places are included in the jurisdiction of the Victoria Magistrates' Court?", the second candidate logical form "(add(relation [base, region, region, administrative region, capital]) (add[law, court jurisdiction, court][Victoria Magistrates' Court]))" is reverse-reconstructed into the natural language question "What is the capital of the Victoria Magistrates' Court?", and the third candidate logical form "(add(relation [location, country, capital]) (add[law, court jurisdiction, court]) [Victoria Magistrates' Court])" is reverse-reconstructed into the natural language question "Where is the Victoria Magistrates' Court located?", thus obtaining a transcription question list of the first candidate logical form list. The transcription problem list consists of natural language problems obtained from reconstruction, thus constructing a reverse channel from "logical form to natural language" through the transcription model, realizing the interpretability conversion of logical form, and providing a comparable text carrier for semantic consistency verification.

[0058] Next, the semantic similarity between each natural language question in the transcription question list and the natural language target question is calculated. Specifically, for the transcription question list transcribed by the transcription model, a similarity calculation model is used to map the user-input natural language target question and each natural language question in the transcription question list into semantic vectors. By calculating the cosine similarity of the semantic vectors, a similarity list is obtained. For example, the semantic similarity between the first transcribed natural language question (What places are included in the jurisdiction of the Victorian Magistrates' Court?) and the user's natural language target question (In which capital of Australia is the Victorian Magistrates' Court located?) is approximately 5.974; the semantic similarity between the second transcribed natural language question (What is the capital of the Victorian Magistrates' Court?) and the user's natural language target question (In which capital of Australia is the Victorian Magistrates' Court located?) is approximately 8.064. By quantifying the consistency strength between each natural language question and the target question through similarity calculation, quantifiable semantic verification evidence is generated, laying the foundation for subsequent re-ranking.

[0059] Next, based on the semantic similarity, the candidate logical forms in the first candidate logical form list are reordered to obtain the second candidate logical form list. Specifically, the first candidate logical form list is rearranged in descending order based on the calculated semantic similarity to generate the second candidate logical form list sorted in descending order of semantic similarity.

[0060] Then, using a credibility-aware Poldahl fusion method, the first candidate logic form list and the second candidate logic form list are fused to obtain a third candidate logic form list. Specifically: First, the total length L of the first candidate logic form list (e.g., L=3) is used as the full score benchmark. The basic Poldahl score for each candidate logic form in both lists (the first candidate logic form list and the second candidate logic form list) is calculated, i.e., the first basic Poldahl score = L - the index of the candidate logic form in the first candidate logic form list, and the second basic Poldahl score = L - the index of the candidate logic form in the second candidate logic form list. Subsequently, the first fusion weight of the first candidate logic form list is dynamically calculated. Second fusion weights with the second candidate logical form list . The calculation is based on the average confidence of all candidate logical forms when the large language model generates the first candidate logical form list; The calculation is based on the semantic similarity score variance (i.e., dispersion) of the top M candidate logical forms in the second candidate logical form list (i.e., Top-M, where M is a positive integer and not greater than the number of candidate logical forms n; in this embodiment, M is 3). Next, the two logical form lists are fused based on the fusion weight. Specifically, the first base reach score of each candidate logical form in the first candidate logical form list is multiplied by... Calculate its weighted first part score; multiply its second fundamental Borda score in the second candidate logical form list by . The weighted second part score is obtained. The weighted first part score is added to the weighted second part score to obtain the final Polda count score of the candidate logic form. Finally, the candidate logic forms are rearranged in descending order according to the final Polda count scores of all candidate logic forms to generate a third list of candidate logic forms that has been intelligently weighted and takes into account both the reliability of generation probability and semantic consistency.

[0061] Finally, based on the third candidate logical form list, the answer corresponding to the target question is queried in the knowledge base. The logical form is then converted into SPARQL query language through entity and relation linking to obtain the answer.

[0062] As described in the background section, large language models demonstrate powerful context learning capabilities. After learning from a small number of examples (natural language target question, logical form), they can perform complex reasoning tasks based on the natural language target question. Previous methods typically consist of two stages: first, the large language model generates a list of candidate logical forms; then, through entity and relational links, an executable query language is obtained. The answer selection uses either the self-consistency principle (executes all query languages ​​and uses a majority voting strategy to determine the final answer) or the first validity principle (executes candidate query languages ​​sequentially and uses the first valid answer as the final answer). However, the high accuracy of the majority voting strategy comes at the cost of a large number of unreliable candidates, often requiring thousands of SPARQL queries and several minutes to obtain an answer, significantly increasing runtime. Applications based on large language models cannot afford such resource waste and slow response times. Although the first validity principle reduces the time cost caused by the self-consistency principle to some extent, the most valuable logical form may not be in the first position of the valid execution list, leading to increased execution time and answer error rate in large model question answering.

[0063] In view of this, this application constructs a bidirectional semantic verification mechanism and a multi-dimensional ranking fusion mechanism in the large-scale model question answering scenario. Specifically, it achieves accurate optimization of candidate logical forms through the following steps: First, a list of first candidate logical forms for the natural language target question is generated based on the large language model (generation); then, the candidate logical forms are converted back into natural language questions using a transcription model, constructing a list of transcribed questions corresponding to the natural language target question (transcription); further, the semantic similarity is calculated to evaluate the semantic matching degree between the transcribed questions (i.e., natural language questions) and the natural language target question, and the first candidate logical form list is sorted in descending order of semantic similarity accordingly to form a second candidate logical form list with semantic enhancement (verification), thus realizing the bidirectional semantic verification mechanism; finally, the ranking results of the first and second candidate logical form lists are fused using a credibility-aware Porta fusion method, which is the multi-dimensional ranking fusion mechanism, generating a third candidate logical form list that takes into account both generation quality and semantic consistency, and performing knowledge base queries based on this list.

[0064] The innovation of the bidirectional semantic verification mechanism lies in its use of a closed-loop architecture of "generation-transcription-verification" (candidate logical form → natural language question → semantic similarity evaluation) to verify the semantic fidelity of the generated results from the large language model. The innovation of the multi-dimensional ranking and fusion mechanism lies in its use of a credibility-aware Porta fusion method to fuse the first candidate logical form list (ranked based on generation probability) and the second candidate logical form list (ranked based on semantic similarity). This preserves the diversity of the initial candidate list while strengthening the ranking priority of the golden logical form through semantic similarity. Thus, while ensuring candidate coverage, it significantly improves the position of the golden logical form in the fusion list, effectively enhancing the efficiency and accuracy of large-scale model question answering, thereby solving the technical problem of traditional large-scale model question answering methods failing to balance execution efficiency and question answer accuracy.

[0065] Example 4

[0066] In this embodiment, the large language model is trained through efficient parameter fine-tuning, and the specific steps are as follows: obtaining the first sample logical form corresponding to each sample question from the knowledge base; replacing the entities in the first sample logical form with the surface names corresponding to the entities; decomposing the multi-level labels in the first sample logical form; constructing a second sample logical form based on the surface names corresponding to the entities and the decomposed labels; constructing sample pairs consisting of sample questions and corresponding second sample logical forms; and using the sample pairs to fine-tune the large language model.

[0067] Specifically, a number of sample questions are first obtained, and then the first sample logical form corresponding to each sample question is retrieved from the knowledge base. Considering that entities in the logical forms existing in the knowledge base are often represented in the form of machine identifiers, this symbolic representation is not conducive to reasoning and learning. For example, in the question: "where are the NFL Redskins from?", the entity "redskins" is represented in the dataset as m.084l5 (machine identifier). Such machine identifiers cannot guide large language models to perform effective reasoning. Based on this, this embodiment replaces the entities in the first sample logical form with their corresponding surface names. Furthermore, since the relations in the first sample logical form are composed of multi-level labels, this increases the difficulty of generating large language models. Therefore, the multi-level labels are decomposed. For example, the relation in the question "where are thenflredskins from?" is represented in the knowledge base as "sports.sports_team.location", which this embodiment formats as "sports, sports team, location". Finally, enclose the entities and relations in square brackets to obtain the second sample logical form, the overall example of which is as follows: "(JOIN(R[sports,sports team, location] ) [Washington Redskins])".

[0068] Next, sample pairs are constructed, consisting of sample questions and corresponding second sample logical forms. These sample pairs are then used to fine-tune the large language model. This approach achieves precise alignment between natural language questions and symbolic knowledge representations, significantly improving the large language model's ability to generate logical forms and providing a high-quality foundation for the generation of subsequent candidate logical forms.

[0069] Example 5

[0070] In this embodiment, the large language model is adjusted based on few-shot context learning. The specific steps are as follows: extract N similar sample questions (N is a positive integer) from the preset training set whose similarity to the natural language target question is greater than a preset threshold; obtain the second sample logic form corresponding to each similar sample question; take the combination of the similar sample question and the corresponding second sample logic form as an example, construct prompt information according to the real-time generated instructions, the example and the natural language target question; and adjust the large language model using the prompt information.

[0071] Specifically, this embodiment of the invention has found that large language models possess powerful generalization capabilities, enabling few-shot context learning to allow LLMs to complete complex reasoning tasks while observing a small amount of labeled data. Therefore, this embodiment adopts a few-shot context learning approach to adjust the large language model, enabling it to transform natural language problems into logical forms. Specifically, a natural language target problem is first determined, and then N similar sample problems with a similarity greater than a preset threshold are extracted from a preset training set. Then, the second sample logical form corresponding to each similar sample problem is obtained.

[0072] Next, taking the combination of a similar sample problem and the corresponding second sample logical form as an example, based on the real-time generated generation instructions, the example, and the natural language target problem, a prompt message is constructed to guide the large language model to generate a list of candidate logical forms for the natural language target problem. The format of the corresponding generation logic is as follows:

[0073] ;

[0074] in, C A list of candidate logical forms generated for a large language model. To generate instructions, Ex For example, Q For the natural language objective problem, LLM stands for Large Language Model.

[0075] The above methods achieve precise alignment between natural language questions and symbolic knowledge representations, significantly improving the logical form generation capability of large language models and providing a high-quality foundation for the generation of subsequent candidate logical forms.

[0076] Example 6

[0077] In this embodiment, the transcription model is based on a sample pair consisting of a sample question and a second sample logical form obtained in Example 4. The second sample logical form is regarded as the input and the sample question as the output. The pre-trained language model is then fine-tuned and trained.

[0078] Specifically, after obtaining the first candidate logical form list, this embodiment found that over 21% of the golden logical forms in the candidate logical form list did not appear at the top of the candidate logical form list, leading to errors. Through data analysis, this embodiment found that the golden logical forms contain specific pattern information from the knowledge base, and using them as auxiliary clues may help solve this problem. Based on this, this embodiment proposes a two-way semantic verification mechanism (RTAM). Based on sample pairs consisting of second sample logical forms and sample questions in the knowledge base, the pre-trained language model is fine-tuned to give it the ability to perceive from logical forms to natural language questions. Then, the trained transcription model is used to transcribe each candidate logical form in the first candidate logical form list into a natural language question.

[0079] Then, using transcribed natural language questions and the problem of natural language objectives Q Calculate similarity score The formula is as follows:

[0080] ;

[0081] in, This represents any algorithm for determining the similarity between two natural language sentences, such as natural language sentences... Q and After converting them into semantic vectors, the similarity between the two semantic vectors is determined using the Cosine function, and the similarity score is assigned to a value. .

[0082] Based on similarity score The candidate logical forms in the first candidate logical form list are reordered to obtain the second candidate logical form list. The formula is as follows:

[0083] ;

[0084] in, This indicates a rearrangement operation, sorting candidates by similarity score from highest to lowest; n is the number of candidate logical forms.

[0085] Thus, through the closed-loop architecture of "generation-transcription-verification" (candidate logical form → natural language problem → semantic similarity evaluation), the semantic fidelity of the generated results of the large language model is verified.

[0086] Optionally, the operation of fusing the first candidate logical form list and the second candidate logical form list using the credibility-aware Polda fusion method to obtain the third candidate logical form list includes: using the total length of the first candidate logical form list as the full score of the credibility-aware Polda fusion method; traversing the first candidate logical form list and subtracting the index of each candidate logical form in the first candidate logical form list from the full score to obtain the first basic Polda score for each candidate logical form. Traverse the second candidate logic form list, and subtract the index of each candidate logic form in the second candidate logic form list from the full score to obtain the second basic Borda score for each candidate logic form. Dynamically calculate the first fusion weight of the first candidate logical form list. And the second fusion weight of the second candidate logical form list. ,in and The sum is 1;

[0087] Multiply the first basic Porta score of each candidate logical form by the first fusion weight. The weighted first part score is obtained, and the second basic Porta score corresponding to the candidate logical form is multiplied by the second fusion weight. The weighted second part score is obtained; the weighted first part score of each candidate logic form is added to the weighted second part score to obtain the final Polda count score; based on the final Polda count score, the candidate logic forms are reordered to obtain a list of third candidate logic forms.

[0088] In this embodiment of the invention, considering the importance of the order in the first candidate logical form list, a bidirectional reordering strategy for logical problems is proposed to balance the influence of the first and second candidate logical form lists on the final answer. This embodiment uses a credibility-aware Poda fusion method, a ranking-based voting mechanism where the ranking in the list represents the preference order for each candidate logical form, including:

[0089] S1. Calculate the basic Porta score: Take the total length L of the first candidate logic form list as the maximum score. Traverse the first candidate logic form list and subtract the index of each candidate logic form in the first candidate logic form list from the maximum score L to obtain its first basic Porta score. Traverse the second candidate logic form list, and subtract the index of each candidate logic form in the second candidate logic form list from the full score L to obtain its second basic Borda score. The calculation formula is as follows:

[0090]

[0091] in, Indicates the first i One candidate logical form, for Index in the first candidate logical form list for Index in the list of second candidate logical forms.

[0092] S2. Dynamically calculate fusion weights:

[0093] Calculate the first fusion weight of the first candidate logical form list. And the second fusion weight of the second candidate logical form list. And satisfy .

[0094] First fusion weight The calculation is based on the average generation confidence when generating the first candidate logical form list using the large language model. Specifically, first, obtain each candidate logical form from the first candidate logical form list. Generation confidence This confidence score is output by the large language model during generation and is the average generation probability of each word in the candidate logical forms; subsequently, the average generation confidence score of all n candidate logical forms is calculated, i.e. and use this value as Basic calculated value .

[0095] The second fusion weight is calculated based on the variance of the semantic similarity scores corresponding to the top M candidate logical forms in the second candidate logical form list. Specifically, firstly, the top M (Top-M) candidates in semantic similarity are selected from the second candidate logical form list to obtain their similarity score set. Then calculate the variance of the score set. , where µ is the average of the M scores, and this variance is used as . Basic calculated value .

[0096] Subsequently, the basic weights are normalized based on the amplification factor to obtain the final weights used for fusion:

[0097] .

[0098] in, βIt is the amplification factor, a variance "magnitude balancer" calculated from the second candidate logic form, with a value range of [1, 10]. The simpler the logical structure of the natural language problem, the better. β The larger the value, the more likely the natural language problem will be input into the LLM, which will then score and determine the outcome.

[0099] S3. Calculate the weighted final score and generate the final list:

[0100] For each candidate logic form, multiply its first base arrival score by its first fusion weight to obtain its weighted first part score; multiply its second base arrival score by its second fusion weight to obtain its weighted second part score. Add the two parts score to obtain the final arrival count score for that candidate logic form. The calculation formula is:

[0101]

[0102] The final score based on the Borda count of all candidate logical forms. Reorder the results in descending order to obtain a list of third candidate logical forms.

[0103] In this embodiment of the invention, considering that the order of the two candidate logical form lists is dynamically changing in context under different problem scenarios, a credibility-aware Polda fusion method is proposed, which introduces a dynamic weight-aware mechanism on top of the traditional ranking-based voting mechanism.

[0104] Specifically, the innovation of the credibility-aware PODA fusion method lies in the fact that it not only converts rankings into scores, but also further integrates weights. and The reliability of the first candidate logical form list (ranked by generation probability) and the second candidate logical form list (ranked by semantic similarity) was dynamically perceived and quantified under the current problem. This was based on the average generation confidence of the large language model. High time, The increase in variance makes the method more reliant on its original ranking; when the variance of the semantic similarity scores of the Top-M candidate logical forms in the semantic similarity ranking increases... When it is big, The method relies more on the results of semantic verification.

[0105] The credibility-aware Poda fusion method introduces weight awareness, enabling intelligent weighted fusion of the first and second candidate logical forms, rather than simply adding up scores. This method can dynamically adapt to different question-answering scenarios, accurately improving the ranking of the best logical form, thereby significantly improving the accuracy and efficiency of question answering while preserving candidate diversity.

[0106] This invention innovatively proposes a "trustworthiness-aware Porta fusion method": This invention does not stop at traditional static ranking fusion, but further introduces a trustworthiness-aware mechanism. The core advantage of this method is that it not only avoids the "incommensurability" problem of heterogeneous scores, but also dynamically evaluates the reliability of different ranking lists in the current context and performs intelligent weighting. This fusion method first calculates a first basic Porta score based on the ranking of the candidate logical form in the first candidate logical form list, and then calculates a second basic Porta score based on the ranking of the candidate logical form in the second candidate logical form list. Based on this, a first fusion weight for the first candidate logical form list is dynamically calculated. Second fusion weights with the second candidate logical form list : The algorithm is dynamically adjusted based on the average generation confidence score of the large language model. The algorithm dynamically adjusts the semantic similarity scores of the Top-M candidate logical forms based on their dispersion in the semantic similarity ranking. Then, the first basic arrival score of each candidate logical form is multiplied by a first fusion weight to obtain a weighted first part score, and the second basic arrival score of the corresponding candidate logical form is multiplied by a second fusion weight to obtain a weighted second part score. Finally, the weighted first part score and the weighted second part score of each candidate logical form are added together to obtain the final arrival count score for each candidate logical form, and the final ranking is generated accordingly. This method represents a leap from "static aggregation" to "dynamic intelligent fusion." It no longer treats all "votes" equally, but assigns higher "weights" to more reliable voters, thus more intelligently balancing the "intuition" of the generative model with the "evidence" of semantic verification. This significantly improves the ranking probability and stability of the "golden logical form" in the final candidate logical form list, and can perceive and respond to the dynamic changes in model confidence and semantic discriminancy in each question-and-answer session, stably producing better rankings in various scenarios.

[0107] In this embodiment, the value of M is set according to the actual situation and is a positive integer.

[0108] Example 7

[0109] In this embodiment, the operation of querying the answer corresponding to the natural language target question in the knowledge base according to the third candidate logical form list includes: extracting the surface names corresponding to all entities from each candidate logical form in the third candidate logical form list; linking each surface name extracted from each candidate logical form to an entity identifier in the knowledge base, and determining a target entity identifier from the linked entity identifiers; replacing the surface names in each candidate logical form with the corresponding target entity identifier, converting the replaced candidate logical forms into a first query language of the knowledge base, and executing the first query language in the knowledge base to obtain the corresponding answer; determining whether the obtained answer meets preset requirements; for the current candidate logical form that does not obtain an answer that meets the requirements, obtaining all relations of each entity in the current candidate logical form within a two-hop range in the knowledge base to obtain a set of limited relations for each entity; and optimizing each relation entry in the current candidate logical form according to the set of limited relations, converting the optimized current candidate logical form into a second query language of the knowledge base, and executing the second query language in the knowledge base to obtain the corresponding answer.

[0110] In this embodiment of the invention, the surface entities in the candidate logical form do not support direct execution. Therefore, this embodiment performs pattern matching on the knowledge base for relations and entities.

[0111] Specifically, to determine the accurate Entity Identifier (MID) of the entities in the problem, the surface names of each entity are first extracted from each candidate logical form in the third candidate logical form list. Then, all candidate MIDs exceeding a threshold are filtered using the FACC1 tool. If no candidate MID exceeds the threshold, the first MID obtained from the FACC1 tool is taken as the target entity identifier. Furthermore, if multiple surface names are detected in the candidate logical forms, all permutations of their combinations are considered.

[0112] Next, the surface name in each candidate logical form is replaced with the corresponding target entity identifier to achieve batch matching of entity patterns. Since the ranking of the candidate logical forms that are more consistent with the knowledge base pattern in the third candidate logical form list has risen, in order to better balance time cost and performance in large-scale retrieval, the replaced candidate logical forms are directly converted into the first query language of the knowledge base, and the first query language is executed in the knowledge base to obtain the corresponding answer.

[0113] Next, it is determined whether the obtained answer meets the preset requirements. Only for current candidate logical forms that do not yield valid answers is relational pattern matching performed. Specifically, first, all relations (e.g., "province of origin → administrative affiliation → judicial jurisdiction") of entities appearing in the current candidate logical form (e.g., "Victorian District Court") within a two-hop range in the knowledge base are obtained. The first-hop relations (direct associations) and second-hop relations (associations through intermediate nodes) of each entity are merged to form a set of limited relations (e.g., a set containing relations such as "court type", "jurisdiction", and "administrative level"). For each relation entry in the current candidate logical form, semantic similarity is used to obtain the top-K most likely matching relation entries from the limited relation set and replace them, resulting in an optimized current candidate logical form. Furthermore, if multiple relation entries are detected in the logical form, all permutations of their combinations are also considered.

[0114] Finally, the optimized current candidate logic is transformed into the second query language of the knowledge base, and the second query language is executed in the knowledge base to obtain the corresponding answer.

[0115] Example 8

[0116] To evaluate the effectiveness of the core components of the proposed bidirectional ranking-based large-scale question answering method (hereinafter referred to as RankKBQA), the detailed settings of RankKBQA and its comparison results with the baseline will be presented below. It should be noted that this experiment mainly verifies the performance improvement brought by the bidirectional semantic verification mechanism (i.e., steps S301 to S304 in Example 1). Under this experimental configuration, after RankKBQA generates a second candidate logical form list through transcription and semantic reordering, it does not apply the more advanced "trustworthiness-aware hop count fusion method," but instead uses a standard fusion strategy (traditional hop count method) to generate a third candidate logical form list, thereby isolating and highlighting the value of the bidirectional semantic verification itself. This setting aims to first solidify the effectiveness of the basic architecture. The trustworthiness-aware hop count fusion method proposed in this invention is an advanced strategy that achieves further performance leaps based on this verified reliability.

[0117] (1) Dataset: In this embodiment, RankKBQA was evaluated on two standard KBQA datasets as follows:

[0118] WebQuestionsSP (WebQSP) is a widely recognized KBQA dataset containing 4,737 natural language questions.

[0119] The Complex WebQuestions (CWQ) dataset extends WebQSP by incorporating four types of complex questions: Conjunctions, Compositions, Comparisons, and Superlatives.

[0120] (2) Baseline:

[0121] Methods based on large language model agents include KD-CoT, Interactive-KBQA, and QueryAgent.

[0122] Fine-tuning-based methods include RnG-KBQA, ArcaneQA, DecAF, Uni-Parser, FC-KBQA, HGNet, ChatKBQA, and RoG.

[0123] Methods based on few-shot context learning include KB-BINDER, KB-Coder, FlexKBQA, and ToG.

[0124] (3) Evaluation metrics: F1 and Hits@1 were used as performance metrics on the WebQSP and CWQ datasets. In addition, this embodiment reports the average time cost (ATC) for each dataset in the ablation experiments.

[0125] (4) Experimental details

[0126] In this fine-tuning-based approach, Llama-3-8b was selected as the base large language model, with a learning rate of 5e-5 and other parameters kept at their default values, trained for 10 epochs. Furthermore, the beam search size was set to 15 for the WebQSP dataset and to 8 for the CWQ dataset. The ChatKBQA method was also reproduced based on this environment.

[0127] In the few-shot context learning approach, Llama-3-8b is still used as the base large language model, with the model temperature set to 0.7 and the maximum number of words generated by the model set to 8192. For the WebQSP dataset, the sample size N for the few shots is set to 100, meaning 100 similar samples with a similarity greater than a preset threshold to the natural language target question are extracted from the training set. For the CWQ dataset, the sample size N for the few shots is set to 40, meaning 40 similar samples with a similarity greater than a preset threshold are extracted from the training set.

[0128] During the transcription phase, this embodiment uses a T5 family model as the pre-trained language model for transcription. For the WebQSP dataset, T5-base is used, and for the CWQ dataset, T5-large is used, with a learning rate of 3e-5. Furthermore, the bge-reranker-v2-m3 model without fine-tuning is used to calculate the semantic similarity score between the transcription problem and the natural language target problem.

[0129] In the knowledge base pattern matching process, this embodiment uses a threshold to control the number of entities to be matched. For relation pattern matching, the SimCSE instance unsup-simcse-roberta-large is used to obtain a dense representation. Considering that multiple relation entries may appear in the candidate logical form, each relation entry is allowed to have K=15 relations to be matched.

[0130] In addition, to verify that a larger model can bring more substantial benefits, based on this experimental environment, we replaced Llama-3-8b with ChatGPT (gpt-3.5-turbo) under API calls for experiments.

[0131] Table 1 below shows a performance comparison of the large model question answering method RankKBQA based on bidirectional ranking and different baselines on two KBQA datasets.

[0132] Table 1

[0133]

[0134] Note: In Table 1, 'w' indicates usage.

[0135] (5) Experimental results

[0136] As shown in Table 1, Few-shot ICL and LLM-based agents have inherent disadvantages compared to fine-tuning-based methods. Therefore, RankKBQA based on few-shot context learning is not compared with the baseline of fine-tuning-based methods during the comparison process.

[0137] In employing a few-shot context-based learning approach, RankKBQA achieved an F1 score of 78.5 on the WebQSP dataset using the ChatGPT pedestal, outperforming KB-Coder by 2.9 points using the same pedestal.

[0138] RankKBQA also outperformed KB-BINDER, which uses the Codex platform, as well as Interactive-KBQA and QueryAgent, which use the GPT-4 platform.

[0139] In employing a few-shot context learning-based approach, RankKBQA achieved 82.0 Hits@1 on the WebQSP dataset using the ChatGPT pedestal, surpassing ToG 6.2 which also uses ChatGPT. ToG utilizes Oracle entity annotations, which theoretically outperforms the FACC1 preference retrieval algorithm used by RankKBQA.

[0140] It is worth noting that on the CWQ dataset, for both F1 and Hit@1, RankKBQA using the ChatGPT base outperforms all baselines, including LLM-based Agent and Few-shot ICL, achieving an F1 score of 64.4 and a Hit@1 score of 69.8. Furthermore, RankKBQA using Llama-3-8b also performs admirably. On the WebQSP dataset, its F1 score (73.4) is only 2.2 lower than existing techniques (KB-Coder), and in Hit@1, it is 0.6 higher than ToG using ChatGPT.

[0141] In the fine-tuning-based approach, RankKBQA achieved an F1 score of 79.6 on the WebQSP dataset and a competitive performance (83.5) for Hit@1, only 2.2 below the current standard (RoG). On the CWQ dataset, RankKBQA achieved F1 scores of 75.9 and 80.2 for Hit@1, respectively. Overall, RankKBQA demonstrates an average F1 improvement of 4.8% compared to the highest performance baseline model across both non-fine-tuning and fine-tuning methods.

[0142] (6) Ablation analysis

[0143] This embodiment conducted an ablation study to understand the impact of the two-way semantic verification mechanism (RTAM) on the final F1 score and ATC. Experiments were performed on both few-shot context learning-based and fine-tuning-based methods. In the few-shot context learning-based method, Llama-3-8b and ChatGPT were selected as base models. Table 2 shows the different cases with and without RTAM, and reports the obtained F1 and ATC scores.

[0144] Table 2

[0145]

[0146] Note: In Table 2, w / o indicates removal.

[0147] In the few-shot context learning-based approach, this embodiment uses Llama-3-8b as the base for RankKBQA. Whether on the WebQSP or CWQ datasets, this method shows improvements in F1 scores. For ATC, RankKBQA improves by 3.4 seconds on the WebQSP dataset and by 14.3 seconds on the CWQ dataset. For RankKBQA using ChatGPT, ATC improves by 2.5 seconds on the WebQSP dataset and by 5.1 seconds on the CWQ dataset. Furthermore, in the fine-tuning-based approach, RankKBQA using RTAM shows improvements in both F1 and ATC scores.

[0148] Overall, with the addition of a two-way semantic verification mechanism, the baseline models, both fine-tuned and non-fine-tuned, showed an average speed improvement of 13.7% on both datasets.

[0149] In summary, the RankKBQA framework proposed in this invention is a novel approach that further optimizes the capabilities of KBQA systems based on large language models through a transcriptional reordering method. Extensive experiments on various datasets demonstrate the effectiveness of RankKBQA, outperforming all baseline methods in both few-shot context learning and fine-tuning scenarios. Furthermore, compared to other baseline methods, the bidirectional ranking-based large model question answering method in this embodiment significantly reduces the average time cost. In the era of large language models, it meets users' requirements for KBQA systems in terms of both response speed and performance.

[0150] The RankKBQA framework proposed in this invention improves the sorting list in the golden logic form, while reducing the candidate search space to improve execution efficiency (average processing speed improvement of 13.7%) and question answering accuracy (average improvement of F1 score of 4.8%).

[0151] Furthermore, according to a second aspect of this embodiment, a storage medium is provided. The storage medium includes a stored program, wherein, when the program is executed, a processor performs the method described in any of the above embodiments.

[0152] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0153] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0154] Example 9

[0155] This embodiment provides a large-model question-answering device based on bidirectional ranking (hereinafter referred to as a large-model question-answering device or question-answering device, see [link]). Figure 3 The large-scale question-and-answer device includes:

[0156] The first generation module 1110 is used to obtain the natural language target question input by the user and generate a first candidate logical form list of the target question using a pre-trained large language model.

[0157] Transcription module 1120 is used to transcribe each candidate logical form in the first candidate logical form list into a corresponding natural language question using a pre-trained transcription model, thereby obtaining a transcription question list.

[0158] Similarity calculation module 1130 is used to calculate the semantic similarity between each natural language question in the transcription question list and the natural language target question;

[0159] The second generation module 1140 is used to reorder each candidate logical form in the first candidate logical form list according to the semantic similarity to obtain a second candidate logical form list.

[0160] The third generation module 1150 is used to fuse the first candidate logical form list and the second candidate logical form list using a credibility-aware Polda fusion method to obtain a third candidate logical form list.

[0161] And a query module 1160, used to query the answer corresponding to the natural language target question in the knowledge base according to the third candidate logical form list.

[0162] Example 10

[0163] This embodiment provides a large-scale question-answering device based on bidirectional sorting, including: a processor and a memory;

[0164] Memory 2 is connected to the processor and provides instructions to the processor to perform the following processing steps: acquiring a natural language target question input by the user, and generating a first candidate logical form list of the natural language target question using a pre-trained large language model; using a pre-trained transcription model to transcribe each candidate logical form in the first candidate logical form list into a corresponding natural language question, obtaining a transcription question list; calculating the semantic similarity between each natural language question in the transcription question list and the natural language target question; reordering each candidate logical form in the first candidate logical form list according to the semantic similarity, obtaining a second candidate logical form list; fusing the first candidate logical form list and the second candidate logical form list using a credibility-aware Polda fusion method, obtaining a third candidate logical form list; and querying the knowledge base for the answer corresponding to the natural language target question based on the third candidate logical form list.

[0165] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0166] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0167] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0168] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0169] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0170] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0171] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

[0172] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A large-scale question-answering method based on bidirectional ranking, characterized in that, The question-and-answer method includes: Obtain the natural language target question input by the user, and use a pre-trained large language model to generate a list of first candidate logical forms of the natural language target question; Using a pre-trained transcription model, each candidate logical form in the first candidate logical form list is transcribed into a corresponding natural language question, thus obtaining a transcription question list; Calculate the semantic similarity between each natural language question in the transcription question list and the natural language target question; Based on the semantic similarity, the candidate logical forms in the first candidate logical form list are reordered to obtain the second candidate logical form list. The first candidate logical form list and the second candidate logical form list are merged to obtain the third candidate logical form list; and Based on the third candidate logical form list, query the knowledge base for the answer corresponding to the natural language target question; Using a credibility-aware Polarization method, the first candidate logical form list and the second candidate logical form list are fused to obtain a third candidate logical form list, which specifically includes: The total length of the first candidate logical form list is taken as the full score; Traverse the first candidate logical form list, and subtract the index of each candidate logical form in the first candidate logical form list from the full score to obtain the first basic Borda score for each candidate logical form; Traverse the second candidate logical form list, and subtract the index of each candidate logical form in the second candidate logical form list from the full score to obtain the second basic Borda score for each candidate logical form; The first fusion weight of the first candidate logical form list and the second fusion weight of the second candidate logical form list are dynamically calculated, wherein the sum of the first fusion weight and the second fusion weight is 1; Multiply the first basic arrival score of each candidate logical form by the first fusion weight to obtain the weighted first part score, and multiply the second basic arrival score of the corresponding candidate logical form by the second fusion weight to obtain the weighted second part score; Add the weighted first part score and the weighted second part score of each candidate logic form to obtain the final Borda count score for each candidate logic form; and Based on the final Polda count score of each candidate logic form, each candidate logic form is reordered to obtain a third candidate logic form list.

2. The large-scale question-answering method based on bidirectional ranking according to claim 1, characterized in that, The large language model is trained through the following steps: Obtain the first sample logical form corresponding to each sample question from the knowledge base; Replace the entities in the first sample logical form with the surface names corresponding to the entities, decompose the multi-level tags in the first sample logical form, and construct the second sample logical form based on the surface names corresponding to the entities and the decomposed tags. Construct sample pairs consisting of sample questions and corresponding second sample logical forms; as well as The large language model is fine-tuned using the sample pairs.

3. The large-scale question-answering method based on bidirectional ranking according to claim 2, characterized in that, The large language model is obtained through the following steps: Extract N similar sample questions from the preset training set whose similarity to the natural language target question is greater than a preset threshold; Obtain the logical form of the second sample corresponding to each similar sample problem; Taking the combination of similar sample questions and corresponding second sample logical forms as an example, prompt information is constructed based on the real-time generated instructions, the example, and the natural language target question; as well as The large language model is adjusted using the provided prompts.

4. The large-scale question-answering method based on bidirectional ranking according to claim 2, characterized in that, The transcription model is obtained by fine-tuning the pre-trained language model based on the sample pair, taking the logical form of the second sample as input and the sample question as output.

5. The large-scale question-answering method based on bidirectional ranking according to claim 1, characterized in that, Dynamically calculate the first fusion weight and the second fusion weight, including: The first fusion weight is calculated based on the average generation confidence when the large language model generates the first candidate logical form list; the second fusion weight is calculated based on the variance of the semantic similarity scores corresponding to the top-M candidate logical forms in the second candidate logical form list.

6. The large-scale question-answering method based on bidirectional ranking according to claim 1, characterized in that, Based on the third candidate logical form list, the answer for each current candidate logical form is queried in the knowledge base in order of ranking in the third candidate logical form list. Once a non-empty answer is returned, the query is stopped immediately and the current answer is output, including: Extract the surface names corresponding to all entities from the current candidate logical forms in the third candidate logical form list; Each surface name extracted from the current candidate logical form is linked to an entity identifier in the knowledge base, and a target entity identifier is determined from the linked entity identifiers. Replace the surface name in the current candidate logical form with the corresponding target entity identifier, convert the replaced current candidate logical form into the first query language of the knowledge base, and execute the first query language in the knowledge base to obtain the corresponding answer; Determine whether the obtained answer meets the preset requirements; For a current candidate logical form where no matching answer is found, obtain all relations of each entity in the current candidate logical form within a two-hop range in the knowledge base, thus obtaining a set of limited relations for each entity; and Based on the defined relation set, each relation entry in the current candidate logical form is optimized, the optimized current candidate logical form is transformed into the second query language of the knowledge base, and the second query language is executed in the knowledge base to obtain the corresponding answer.

7. The large-scale question-answering method based on bidirectional ranking according to claim 6, characterized in that, The operation of optimizing each relation entry in the current candidate logical form based on the defined relation set includes: For each relation entry in the current candidate logical form, calculate the semantic vector representation of the relation entry; Based on the semantic vector representation, semantic similarity retrieval is performed in the limited relation set to obtain the Top-K candidate knowledge base relations whose semantic similarity reaches a preset threshold; and Replace the corresponding relation entries in the current candidate logical form with the retrieved Top-K candidate knowledge base relations to obtain the optimized current candidate logical form.

8. A computer-readable storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, a processor performs the large model question-answering method based on bidirectional sorting as described in any one of claims 1 to 7.

9. A large-scale question-answering device based on bidirectional ranking, characterized in that, The question-and-answer device includes: The first generation module is used to obtain the natural language target question input by the user and generate a first candidate logical form list of the natural language target question using a pre-trained large language model. The transcription module is used to transcribe each candidate logical form in the first candidate logical form list into a corresponding natural language question using a pre-trained transcription model, thereby obtaining a transcription question list. The similarity calculation module is used to calculate the semantic similarity between each natural language question in the transcription question list and the natural language target question; The second generation module is used to reorder each candidate logical form in the first candidate logical form list according to the semantic similarity to obtain a second candidate logical form list. The third generation module is used to merge the first candidate logical form list and the second candidate logical form list to obtain a third candidate logical form list; and The query module is used to query the knowledge base for the answer corresponding to the natural language target question based on the third candidate logical form list; Using a credibility-aware Polarization method, the first candidate logical form list and the second candidate logical form list are fused to obtain a third candidate logical form list, which specifically includes: The total length of the first candidate logical form list is taken as the full score; Traverse the first candidate logical form list, and subtract the index of each candidate logical form in the first candidate logical form list from the full score to obtain the first basic Borda score for each candidate logical form; Traverse the second candidate logical form list, and subtract the index of each candidate logical form in the second candidate logical form list from the full score to obtain the second basic Borda score for each candidate logical form; The first fusion weight of the first candidate logical form list and the second fusion weight of the second candidate logical form list are dynamically calculated, wherein the sum of the first fusion weight and the second fusion weight is 1; Multiply the first basic arrival score of each candidate logical form by the first fusion weight to obtain the weighted first part score, and multiply the second basic arrival score of the corresponding candidate logical form by the second fusion weight to obtain the weighted second part score; Add the weighted first part score and the weighted second part score of each candidate logic form to obtain the final Borda count score for each candidate logic form; and Based on the final Polda count score of each candidate logic form, each candidate logic form is reordered to obtain a third candidate logic form list.

Images