A large model output generation method based on fusion information, medium and equipment

Abstract

The present application relates to the field of large language models, in particular to a large model output generation method based on fusion information, medium and equipment. Including: if there is any question and answer reference record also includes a travel query mark, obtaining travel information; according to the similarity, the multiple question and answer reference information is sorted and spliced to generate initial reference information; the travel information and the initial reference information are spliced to generate secondary reference information; after splicing the initial input and the secondary reference information, the second target input is generated; the second target input is loaded into the corresponding special large language model of the target field to generate the corresponding output. In the present application, according to the travel query mark, the answer wanted to be obtained is more accurate information. Therefore, by whether carrying the travel query mark, it is determined whether the first target input is loaded into the special large language model or the general large language model. In order to overcome the "illusion" problem of large language model, the influence on the user.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of large language models, in particular to a large model output generation method based on fusion information, medium and equipment. BACKGROUND

[0002] Large language models (LLMs) have shown great potential in various fields due to their strong language understanding and generation capabilities. The "hallucination" problem of large language models refers to the inaccurate, self-contradictory, or inconsistent with input information phenomenon that may occur when these models generate text. Although large language models (LLMs) have demonstrated excellent capabilities in numerous natural language processing tasks, such as language understanding, text generation, and question-answering systems, they sometimes generate seemingly reasonable but actually incorrect or irrelevant content, which is referred to as "hallucination". The causes of hallucination include: the model training data may not cover all knowledge domains or have biases; the model may over-generalize the patterns learned from the training data, leading to incorrect application in new contexts; the model cannot access real-time information, so it may generate outdated or incorrect content; and the pursuit of the best probability of predicting the next word may compromise global logical consistency or factual accuracy.

[0003] Currently, large language models are used in some professional application scenarios to provide more accurate and diverse business services. Typically, to meet the ability to answer professional and non-professional domain questions, existing application scenarios integrate general large language models and specialized large language models. In some general scenarios with low accuracy requirements (such as story writing scenarios), the "hallucination" problem of large language models has little impact. However, in some vertical field scenarios with high accuracy requirements (such as medical, financial, and civil aviation scenarios), users often require more accurate and accurate domain-specific knowledge from the model to solve corresponding problems. Due to the "hallucination" problem of large language models in existing technologies, users cannot accurately obtain professional knowledge, making it difficult to better adapt to vertical fields with high accuracy requirements. SUMMARY

[0004] To solve the above technical problems, the technical solution adopted by the present application is as follows:

[0005] According to one aspect of the present application, a large model output generation method based on fusion information is provided, the method comprising the following steps:

[0006] The initial input is matched with a plurality of categories of external knowledge bases in the target field respectively to recall a plurality of categories of external knowledge bases corresponding to the question and answer reference records respectively; the question and answer reference records include question and answer reference information and confidence; and the external knowledge bases include long text knowledge bases, short text knowledge bases, structured data knowledge bases and intent knowledge bases.

[0007] If there is any question and answer reference record that also includes a travel query mark, the travel information of the user corresponding to the initial input is obtained; the travel query mark indicates that the content of the answer part in the question and answer reference information corresponding to the question and answer reference record is related to the value corresponding to at least one field in the travel information of the user;

[0008] A target similarity calculation model is used to generate the similarity between the initial input and the sentence vector corresponding to each question and answer reference information; the target similarity calculation model is a RoBERTa model trained based on a SimCSE framework using corpus corresponding to the target field;

[0009] The plurality of question and answer reference information is sorted and spliced in descending order of similarity to generate initial reference information;

[0010] The travel information and the initial reference information are spliced to generate secondary reference information;

[0011] The initial input and the secondary reference information are spliced to generate a second target input;

[0012] The second target input is loaded into a special large language model corresponding to the target field to generate a corresponding output; the special large language model is obtained by training a general large language model using corpus corresponding to the target field.

[0013] According to a second aspect of the present application, a non-transitory computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the above-mentioned large model output generation method based on fused information.

[0014] According to a third aspect of the present application, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the above-mentioned large model output generation method based on fused information when executing the computer program.

[0015] The present application has at least the following advantages:

[0016] The prompt information text in the Prompt engineering with higher quality and higher relevance is formed by using various types of professional prior knowledge in the target field and the itinerary information together with the itinerary fusion in the present application. The prompt information is used to clearly define the task requirements, constraint conditions or expected answer format, so as to guide the large language model (LLM) to generate high-quality text output that meets the specific requirements and expectations, and further overcome the probability of the large language model appearing "hallucination".

[0017] In addition, the general large language model is more likely to have "hallucination" problem when answering the corresponding field problem than the special large language model. Therefore, in the present application, whether the itinerary query mark is carried in the recall result is used to directly determine whether the first target input is loaded into the special large language model or the general large language model. Since the itinerary query mark is used to represent the relevance of the content of the recall result and the itinerary information of the user, the answer that the initial input wants to obtain is more close to professional and accurate information according to the itinerary query mark. That is, the model "hallucination" problem needs to be overcome more effectively. Then, the special large language model can be used to generate more accurate output with higher understanding ability of the target field, so as to further overcome the "hallucination" problem of the large language model and improve its applicability in the vertical field with high precision requirement. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 A flowchart of a large model output generation method based on external knowledge provided by an embodiment of the present application is provided.

[0020] Figure 2 A flowchart of a large model output generation method based on fusion information provided by an embodiment of the present application is provided. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0022] As a possible implementation of the present application, as Figure 1As shown, a large model output generation method based on external knowledge is provided, which comprises the following steps:

[0023] S100: Use the initial input to match with multiple categories of external knowledge bases in the target field respectively to recall multiple categories of external knowledge bases corresponding to the question and answer reference records respectively. The question and answer reference records include question and answer reference information and confidence. The external knowledge bases include long text knowledge bases, short text knowledge bases, structured data knowledge bases and intent knowledge bases. Specifically, the target field in the embodiment can be the civil aviation field. The long text knowledge base can be the existing long document text in the civil aviation field.

[0024] The short text knowledge base can be the question and answer pair knowledge in the civil aviation field. The structured data knowledge base can be the existing knowledge graph in the civil aviation field.

[0025] Specifically, when the initial input (i.e. query) is matched with the professional prior knowledge of the civil aviation field in the long text knowledge base, the short text knowledge base, the structured data knowledge base and the intent knowledge base, the following methods can be used:

[0026] When the external knowledge base is a short text knowledge base.

[0027] Using the initial input to match with multiple categories of external knowledge bases respectively includes:

[0028] S110: Obtain the first similarity between the sentence vector corresponding to the initial input and the sentence vector corresponding to each question and answer pair in the short text knowledge base through sentence vector retrieval. The sentence vector is obtained by RoBERTa model trained based on the SimCSE framework using the target field corresponding corpus.

[0029] S111: Each question and answer pair corresponding to the first similarity greater than the first similarity threshold and the corresponding first similarity are respectively taken as the recalled question and answer reference information and the corresponding confidence.

[0030] The model trained based on the SimCSE framework using the RoBERTa model on the civil aviation corpus can be used to generate the sentence vector of the text and compare the similarity between them. SimCSE is an unsupervised sentence representation learning method, which mainly constructs positive and negative sample pairs of sentences, and uses the output of the BERT model to learn better sentence representation through contrastive learning.

[0031] Through the above method, the initial input can be matched with the question and answer pair in the short text knowledge base to select one or more recall results. When selecting the recall results, the top few in the similarity ranking can be selected, or the similarity threshold can be used for screening.

[0032] When the external knowledge base is a long text knowledge base.

[0033] The initial input is matched with the external knowledge base of each category, including:

[0034] S120: Based on the BM25 algorithm, the initial input is keyword matched with each text in the long text knowledge base, to generate the second similarity corresponding to each text in the long text knowledge base.

[0035] S121: Each text corresponding to the second similarity greater than the second similarity threshold and the corresponding second similarity are respectively taken as the recalled question and answer reference information and the corresponding confidence.

[0036] In this embodiment, the matching of the initial input and the long document is realized by ES (Elasticsearch) matching. Elasticsearch is an open source search engine, which is built based on Lucene library and provides a distributed, RESTful style full-text search engine. In Elasticsearch, "matching" is a core concept in the query process, which involves how to find the documents that meet the user's query conditions from the index.

[0037] In this step, the keyword matching based on the BM25 algorithm is a widely used relevance scoring method in the field of information retrieval and text mining, which is used to evaluate the matching degree between the keywords in the query and the documents. BM25 algorithm is an improvement of the classic TF-IDF model, which not only considers the term frequency (Term Frequency, TF) and inverse document frequency (Inverse Document Frequency, IDF), but also introduces document length normalization and query frequency factors, so as to more accurately reflect the importance of keywords in the document, in order to improve the accuracy of matching.

[0038] When the external knowledge base is a structured data knowledge base.

[0039] The initial input is matched with the external knowledge base of each category, including:

[0040] S130: Based on KB-QA, the key entity in the initial input is extracted, and the corresponding relationship between the key entity and the entity in the structured data knowledge base is determined to generate target entity information.

[0041] S131: The target entity information is taken as the recalled question and answer reference information.

[0042] S132: The similarity between the key entity extracted from the initial input and the target entity information is taken as the corresponding confidence.

[0043] Since the structured data in this embodiment is a knowledge graph in the corresponding field, KB-QA (Knowledge Base Question Answering) can be used for entity recognition matching and entity similarity calculation to determine the recall results.

[0044] In this embodiment, the matching of the structured knowledge base in S130-S132 above can be specifically:

[0045] Entity Recognition: Utilize Named Entity Recognition (NER) techniques to identify entities with specific meanings from user queries (i.e., initial input), such as names, locations, times, etc. This can be achieved through pre-trained models such as BERT, which can understand context and accurately label entity types and boundaries.

[0046] Entity Linking: Map the identified entities to the entities in the knowledge base. This step involves finding the most matching entries from the knowledge base for the entities mentioned in the question, usually requiring similarity calculation between entities.

[0047] Entity Similarity Calculation: Calculate the similarity between entities in the question and the descriptions of entities in the knowledge base. Common methods include: pre-trained model-based similarity calculation: such as SimBERT, BGE, GTE, etc. These models can represent entities as high-dimensional vectors, and then calculate the cosine similarity or other similarity measures between vectors to evaluate the similarity between entities. Semantic distance-based methods: iterative alignment strategy, calculate semantic distance through joint vector generation model, such as distance defined by energy function. Rule and pattern matching: in some cases, predefined rules or patterns can also be used to directly match entities.

[0048] When the external knowledge base is an intent knowledge base.

[0049] The initial input is matched with multiple categories of external knowledge bases, including: using the intent recognition data processing system with patent number: ZL202110934400.X for matching calculation. The question and answer reference information in this embodiment can be the intent recognition result generated by the intent classification model; the confidence in this embodiment is the confidence generated by the intent classification model.

[0050] After the matching operation of S100, if the user input question is a question that needs to be accurately answered in a professional field, there will be at least one recall result and a high confidence. At this time, the recall result can be used to generate a corresponding Prompt (i.e., the content in the subsequent S200-S500 steps) to help the professional large language model better understand the question and accurately output the answer.

[0051] If no question and answer reference information is matched after the matching operation of S100 (i.e., the recall result is an empty set), it indicates that the user input question is not a question that needs to be accurately answered in a professional field. At this time, the initial input can be directly loaded into a general large language model according to the method of S101 to generate a corresponding output.

[0052] S101: If no question and answer reference information is matched, load the initial input into a general large language model to generate a corresponding output.

[0053] S200: Use a target similarity calculation model to generate the similarity between the initial input and the sentence vector corresponding to each question and answer reference information. The target similarity calculation model is a RoBERTa model trained based on the SimCSE framework using a target field corpus.

[0054] S300: Sort and splice the multiple question and answer reference information in descending order of similarity to generate an initial reference information.

[0055] Step S200 is used to recalculate the similarity between each recall result and the initial input, thereby laying the foundation for the subsequent splicing and sorting. S300 sorts all the recall results to generate a Prompt for the initial input. The more effective references in the Prompt are placed in a more forward position, which can facilitate the large language model to better reference to generate more accurate answers.

[0056] S400: After splicing the initial input and the initial reference information, generate a first target input.

[0057] The generated first target input is an input with a Prompt. For example, the user input question (i.e., the initial input) is: What kind of flight is more likely to ride a ferry. After sorting the multiple references according to the similarity, the result is:

[0058] Reference 1: Q: How to check if the flight needs to take the shuttle bus? A: You can check if the flight needs to take the shuttle bus on the xxx app. After opening the app, if there is a shuttle bus icon in the lower right corner of the ticket page, it means there will be a shuttle bus to pick up. The shuttle bus is a transportation tool connecting the airport lounge and the remote aircraft. If the aircraft is parked in a remote location, passengers need to take the shuttle bus to board the aircraft.

[0059] Reference 2: Q: How to check if the flight needs to take the shuttle bus? A: You can check if the flight needs to take the shuttle bus on the xxx app. After opening the app, if there is a shuttle bus icon in the lower right corner of the ticket page, it means there will be a shuttle bus to pick up. The shuttle bus is a transportation tool connecting the airport lounge and the remote aircraft. If the aircraft is parked in a remote location, passengers need to take the shuttle bus to board the aircraft.

[0060] Reference 3: Flights that arrive before 22:00 but do not take off until after 10:00 the next morning will generally use the shuttle bus. Because these flights will occupy the bridge for a long time, which may affect the punctual arrival of flights during the morning rush hour the next day.

[0061] The first target input content generated based on the above information is as follows:

[0062] According to the reference answers provided below, answer the question: Reference 1: Q: How to check if the flight needs to take the shuttle bus? A: You can check if the flight needs to take the shuttle bus on the xxx app. After opening the app, if there is a shuttle bus icon in the lower right corner of the ticket page, it means there will be a shuttle bus to pick up. The shuttle bus is a transportation tool connecting the airport lounge and the remote aircraft. If the aircraft is parked in a remote location, passengers need to take the shuttle bus to board the aircraft. Reference 2: Q: How to check if the flight needs to take the shuttle bus? A: You can check if the flight needs to take the shuttle bus on the xxx app. After opening the app, if there is a shuttle bus icon in the lower right corner of the ticket page, it means there will be a shuttle bus to pick up. The shuttle bus is a transportation tool connecting the airport lounge and the remote aircraft. If the aircraft is parked in a remote location, passengers need to take the shuttle bus to board the aircraft. Reference 3: Flights that arrive before 22:00 but do not take off until after 10:00 the next morning will generally use the shuttle bus. Because these flights will occupy the bridge for a long time, which may affect the punctual arrival of flights during the morning rush hour the next day. Question: What kind of flight is more likely to take the shuttle bus?

[0063] S500: If there is any confidence value greater than the confidence threshold value corresponding to the category among the confidence values of multiple question and answer reference information, load the first target input into the special large language model corresponding to the target domain to generate the corresponding output. The special large language model is obtained by training the general large language model using the corpus corresponding to the target domain.

[0064] S600: If each of the plurality of confidence degrees corresponding to the question and answer reference information is less than or equal to the confidence degree threshold value of the corresponding category, the first target input is loaded into the general large language model to generate the corresponding output.

[0065] In the embodiment, the confidence degree threshold values respectively include confidence degree threshold values corresponding to the long text knowledge base, the short text knowledge base, the structured data knowledge base and the intent knowledge base. The threshold values can be determined according to the setting values used in actual use, so that the results output by the large language model are more accurate.

[0066] In the embodiment, whether the first target input is loaded into the special large language model or the general large language model is determined according to the relationship between the confidence degree of each type of result and the corresponding threshold value. Since the recall results are some professional and accurate prior knowledge in the corresponding field, the higher the corresponding confidence degree is, the more the answer that the initial input wants to obtain is closer to professional and accurate information. That is, the model "illusion" problem needs to be overcome more effectively. Therefore, in the present application, if any confidence degree is greater than the confidence degree threshold value of the corresponding category, the first target input is loaded into the special large language model, and then the special large language model can be used to generate more accurate output with higher understanding ability of the target field, to further overcome the "illusion" problem of the large language model and improve its applicability in vertical fields with high precision requirements.

[0067] As another possible embodiment of the present application, as shown in Figure 2 a large model output generation method based on fusion information is also provided, which includes the following steps:

[0068] W100: The initial input is matched with a plurality of categories of external knowledge bases in the target field respectively to recall a plurality of categories of external knowledge bases corresponding to question and answer reference records. The question and answer reference records include question and answer reference information and confidence degrees. The external knowledge bases include long text knowledge bases, short text knowledge bases, structured data knowledge bases and intent knowledge bases.

[0069] Specifically, the target field in the embodiment is also the civil aviation field. The specific implementation of W100 is the same as the implementation of S100 described above. In addition, when establishing the external knowledge base, whether the itinerary query mark needs to be set in advance for the corresponding question and answer reference record can be determined according to the collected relevant reference information in advance, whether the itinerary information of the user needs to be referred to in order to output more accurate and appropriate standards. The content of the answer part in the question and answer reference information corresponding to the question and answer reference record is related to the value corresponding to at least one field in the itinerary information of the user. The itinerary query mark is usually added to some question and answer reference information that involves the use of the user's itinerary information to further determine the output content. For example, in the following reference:

[0070] The question part in the question and answer reference information is: Is there compensation for flight delay or cancellation?

[0071] The answer part in the question and answer reference information is: (1) If the airline itself causes the flight to be delayed or canceled at the departure airport due to maintenance, flight allocation, crew, etc., the airline will provide food or accommodation services to passengers.

[0072] (2) If the flight is delayed or canceled at the departure airport due to weather, unexpected events, air traffic control, security checks, and passengers, the carrier shall assist passengers in arranging food and accommodation, and the expenses shall be borne by the passengers.

[0073] (3) If the domestic flight is delayed or canceled at the stopover, regardless of the reason, the carrier shall provide food or accommodation services to the stopover passengers.

[0074] As can be seen from the above question and answer reference information example, when answering the corresponding question, if more accurate user itinerary information can be provided, the large language model can output an answer that is more relevant to the user's itinerary in combination with the itinerary information, rather than just outputting the judgment rule provided in the answer part of the question and answer reference information.

[0075] W200: If there is any question and answer reference record that also includes an itinerary query mark, obtain the itinerary information of the user corresponding to the initial input.

[0076] Specifically, the itinerary information includes multiple fields. For example, it can include: flight number, departure airport, destination, departure time, arrival time, cabin type, seat number, boarding gate, food and luggage information, etc.

[0077] W300: Use a target similarity calculation model to generate the similarity between the sentence vectors corresponding to each question and answer reference information and the initial input. The target similarity calculation model is a RoBERTa model trained based on the SimCSE framework using the corpus corresponding to the target field.

[0078] W400: In the order of similarity from large to small, the plurality of question and answer reference information is sorted and spliced to generate initial reference information.

[0079] The specific implementation of W300 and W400 in this embodiment is the same as the implementation of S200 and S300 described above, and will not be repeated here.

[0080] W500: Splice the itinerary information and the initial reference information to generate secondary reference information.

[0081] When splicing the itinerary information and the initial reference information, there is no limitation on the splicing position of the reference information, and it can be directly spliced at the head, tail of the initial reference information or behind the question and answer reference information with the itinerary query mark.

[0082] However, in order to further enable the itinerary information to provide more effective reference, more important fields in the itinerary information will be arranged in a more forward position. For example, the content expressed by the fields such as flight number, seat number, departure place, destination, and departure time is more important than the content expressed by the fields such as luggage and food. Therefore, it will also be arranged in a more forward position.

[0083] Specifically, the arrangement order of the fields in the itinerary information can be determined in the following manner:

[0084] Firstly:

[0085] W501: According to the importance of each field to the itinerary in the civil aviation field, generate the field priority of each field. The importance can be determined by the technical personnel in the field.

[0086] W502: In the order of field priority from high to low, sort each field to generate a second itinerary field sequence.

[0087] W503: According to the second itinerary field sequence, obtain the itinerary information of the user corresponding to the initial input.

[0088] Secondly:

[0089] W510: According to the relevance of each field and the content of the answer part in each type of question and answer reference information, generate the field priority corresponding to each type of question and answer reference information.

[0090] W511: In the order of field priority from high to low, sort each field to generate a first itinerary field sequence corresponding to each type of question and answer reference information.

[0091] W512: According to the first itinerary field sequence corresponding to each type of question and answer reference information, obtain the itinerary information of the user corresponding to the initial input of the corresponding type.

[0092] In this way, the first journey field sequence with different field orders is designed according to the types of problems, so that more detailed and accurate journey information can be provided for each type of problem.

[0093] W600: splice the initial input and the secondary reference information to generate a second target input.

[0094] W600 is to combine the Prompt information formed by the secondary reference information with the initial input (i.e., qes) to form a new input.

[0095] Specifically, an example of the generated second target input is as follows:

[0096] I am a Chinese, and for me, the domestic is China. My last flight itinerary information is as follows: {info} Reference answer: {wenwen_res} Please answer the following questions according to the reference answer combined with my itinerary information: {qes}, please write according to your itinerary information in the answer, do not write according to the reference answer.

[0097] Among them, {info} is the itinerary information, {wenwen_res} is the initial reference information, and {qes} is the initial input.

[0098] W700: load the second target input into the special large language model corresponding to the target domain to generate the corresponding output. The special large language model is obtained by training the general large language model using the corpus corresponding to the target domain.

[0099] Generally, if the recalled record contains the itinerary query information, it indicates that the user needs more accurate answers. Therefore, in this embodiment, whether the itinerary query mark is carried in the recall result is used to directly determine whether the first target input is loaded into the special large language model or the general large language model. Since the itinerary query mark is used to indicate the relevance of the content of the recall result to the user's itinerary information. Therefore, according to the itinerary query mark, it can be determined that the answer the initial input wants to obtain is also more close to professional and accurate information. That is, it is necessary to more effectively overcome the model "illusion" problem. Then, the special large language model can be used to generate more accurate output with higher understanding ability of the target domain, to further overcome the "illusion" problem of the large language model and improve its applicability in vertical fields with high precision requirements.

[0100] Moreover, although individual steps of the methods in the present disclosure are described in a particular order in the figures, this is not required or implied as to the order of execution of the steps, nor is it required that all of the steps be executed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into a single step, a single step can be broken into multiple steps, etc.

[0101] From the above description of the embodiments, those skilled in the art will easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to make a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) execute the method according to the embodiments of the present disclosure.

[0102] In the example embodiments of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0103] Those skilled in the art can understand that each aspect of the present disclosure can be implemented as a system, a method or a program product. Therefore, each aspect of the present disclosure can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system".

[0104] The electronic device according to this embodiment of the present disclosure. The electronic device is merely an example, and should not bring any limitation to the function and use range of the embodiments of the present disclosure.

[0105] The electronic device is in the form of a general computing device. The components of the electronic device can include, but are not limited to, the at least one processor described above, the at least one storage described above, and a bus connecting different system components (including storage and processor).

[0106] The storage stores program code that can be executed by the processor, so that the processor executes the steps according to various example embodiments of the present disclosure described in the "example method" section of the present specification.

[0107] The storage can include a readable medium in the form of a volatile storage, such as a random access memory (RAM) and / or a cache memory, and can further include a read-only memory (ROM).

[0108] The storage can also include a program / utility, having a set of program modules that include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof, can include implementation of a networking environment.

[0109] The bus can represent one or more of several types of bus structures, including a storage bus or

[0110] The electronic device can also communicate with one or more external devices such as a keyboard or a pointing device, through an I / O interface. Additionally, the electronic device can communicate with one or more devices that enable a user to interact with the electronic device, and / or one or more devices (e.g., a router, a modem, a server, etc.) that enable the electronic device to communicate with one or more other computing devices. Such communication can occur via an I / O interface. Still yet, the electronic device can communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or the Internet) through a network adapter. It should be appreciated that for purposes of clarity, not all of the hardware and / or software modules are shown in the figures. For example, the electronic device can include any number of additional hardware and / or software modules that facilitate the electronic device's functionality, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0111] Those skilled in the art will readily understand that the example embodiments described herein can be implemented by software and / or by hardware coupled with software, as described above. Thus, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, etc.) or a network, and includes a number of instructions that cause a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0112] In the example embodiments of the present disclosure, a computer readable storage medium is also provided, which stores a program product capable of implementing the method described above. In some possible embodiments, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program codes for causing a terminal device to perform the steps described in the "example method" section of the present disclosure according to various example embodiments of the present disclosure when the program product is run on the terminal device.

[0113] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0114] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0115] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0116] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0117] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0118] It should be noted that, although several modules or units of the devices for action execution are mentioned in the above detailed description, the division into such modules or units is not mandatory. Indeed, according to an embodiment of the present disclosure, the features and functionalities of two or more of the above-described modules or units can be embodied in one module or unit. Conversely, the features and functionalities of one of the above-described modules or units can be further divided into several modules or units.

[0119] The above merely shows the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed by the present application can be easily conceived by those skilled in the art, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A large model output generation method based on fusion information, characterized by, The method comprises the following steps: The initial input is matched with a plurality of categories of external knowledge bases in the target field respectively to recall a plurality of categories of external knowledge bases corresponding to the question and answer reference records respectively; The question and answer reference records comprise question and answer reference information and confidence; and the external knowledge bases comprise long text knowledge bases, short text knowledge bases, structured data knowledge bases and intent knowledge bases; If there is any question and answer reference record further comprising a travel query mark, travel information of a user corresponding to the initial input is obtained; the travel query mark represents that the content of the answer part in the question and answer reference information corresponding to the question and answer reference record corresponds to the value of at least one field in the travel information of the user; A target similarity calculation model is used to generate the similarity between the initial input and the sentence vector corresponding to each question and answer reference information; the target similarity calculation model is a RoBERTa model trained based on a SimCSE framework using a corpus corresponding to the target field; The plurality of question and answer reference information are sorted and spliced in descending order of the similarity to generate initial reference information; The travel information and the initial reference information are spliced to generate secondary reference information; After the initial input and the secondary reference information are spliced, a second target input is generated; The second target input is loaded into a special large language model corresponding to the target field to generate a corresponding output; the special large language model is obtained by training a general large language model using a corpus corresponding to the target field.

2. The method of claim 1, wherein, The target field comprises a civil aviation field.

3. The method of claim 2, wherein, The travel information comprises a plurality of fields; According to the relevance between each field and the content of the answer part in each type of question and answer reference information, a field priority corresponding to each type of question and answer reference information is generated; Each field is sorted in descending order of the field priority to generate a first travel field sequence corresponding to each type of question and answer reference information; According to the first travel field sequence corresponding to each type of question and answer reference information, the travel information of the user corresponding to the initial input of the corresponding type is obtained.

4. The method of claim 2, wherein, The travel information comprises a plurality of fields; According to the importance of each field to the travel in the civil aviation field, a field priority of each field is generated; Each field is sorted in descending order of the field priority to generate a second travel field sequence; According to the second travel field sequence, the travel information of the user corresponding to the initial input is obtained.

5. The method of claim 1, wherein, When the external knowledge base is a short text knowledge base; The initial input is matched with a plurality of categories of external knowledge bases, comprising: The first similarity between the sentence vector corresponding to the initial input and the sentence vector corresponding to each question and answer pair in the short text knowledge base is obtained through sentence vector retrieval; the sentence vector is obtained by a RoBERTa model trained based on a SimCSE framework using a corpus corresponding to the target field; Each question and answer pair corresponding to the first similarity greater than a first similarity threshold and the corresponding first similarity are taken as the recalled question and answer reference information and the corresponding confidence respectively.

6. The method of claim 1, wherein, When the external knowledge base is a long text knowledge base; The initial input is matched with a plurality of categories of external knowledge bases, comprising: Based on the BM25 algorithm, the initial input is matched with each text in the long text knowledge base to generate a second similarity corresponding to each text in the long text knowledge base; Each text corresponding to the second similarity greater than the second similarity threshold and the corresponding second similarity are respectively taken as the recalled question and answer reference information and the corresponding confidence.

7. The method of claim 1, wherein, When the external knowledge base is a structured data knowledge base; The initial input is matched with a plurality of categories of external knowledge bases respectively, including: Based on KB-QA, the key entity in the initial input is extracted, and the corresponding relationship between the key entity and the entity in the structured data knowledge base is determined to generate target entity information; The target entity information is taken as the recalled question and answer reference information; The similarity between the key entity in the initial input and the target entity information is taken as the corresponding confidence.

8. The method of claim 4, wherein, The fields include: flight number, departure place, destination, departure time, arrival time, cabin type, seat number, meal and luggage. 9.A non-transitory computer-readable storage medium storing a computer program, the computer program comprising instructions configured to cause a processor to perform the method according to any one of claims 1 to 8. The computer program is executed by the processor to implement the large model output generation method based on fusion information according to any one of claims 1-8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the large model output generation method based on fusion information according to any one of claims 1-8.

Images