Question answering method based on large model, and electronic device

Abstract

The present application relates to the technical field of artificial intelligence, and in particular to a question answering method based on a large model, and an electronic device. The electronic device comprises a communication interface, a memory, and a processor, the memory is configured to store a computer instruction, and the processor is configured to execute a computer program to enable the electronic device to: determine, on the basis of each pre-stored text block, a target text block matched with a question to be answered; obtain, if non-text data including a picture and / or a table is pre-stored for the target text block in advance, stored summary text of the non-text data; and input the question to be answered, the target text block, and the summary text into a first large model on the basis of a preset format to obtain answer information. Since the summary text is text that summarizes all content in the non-text data, the first large model does not need to process the picture and / or the table and still can ensure that the content recorded in the picture and / or the table is taken into consideration during generation of the answer information, thereby improving the model-based question answering accuracy.

Description

Question answering methods and electronic devices based on large models

[0001] Cross-reference of related applications

[0002] This application claims priority to Chinese patent applications filed on May 31, 2024, application number 202410702697.0; filed on June 28, 2024, application number 202410870481.5; filed on June 28, 2024, application number 202410865861.X; and filed on August 30, 2024, application number 202411219258.0, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of artificial intelligence technology, and in particular to a question-answering method and electronic device based on a large model. Background Technology

[0004] When using document-based knowledge-based question answering models, Retrieval-Augmented Generation (RAG) models are typically employed to retrieve relevant document fragments. However, RAG models cannot directly retrieve images and tables. Larger models can only generate responses based on the question and document fragments, failing to consider the content of images or tables during response generation, resulting in lower question-answering accuracy. Furthermore, even when relevant images and tables are input into a large model, it cannot effectively process them directly.

[0005] Therefore, improving the accuracy of Q&A responses to images and tables in documents has become an urgent problem to be solved. Summary of the Invention

[0006] Some embodiments of this application provide a question-answering method and electronic device based on a large model. During the question-answering process, a target text block matching the question to be answered is determined based on each pre-saved text block, that is, the text related to the question to be answered is determined. If non-text data such as images and / or tables are pre-saved for the target text block, the summary text of the saved non-text data is obtained. The question to be answered, the target text block, and the summary text are input into a first large model according to a preset format to obtain the answer information. Since the summary text is a summary of all the content in the non-text data, inputting it into the first large model allows the first large model to avoid directly processing the images and / or tables, while ensuring that it considers the content recorded in the images and / or tables when generating the answer information, thereby improving the question-answering accuracy of the large model.

[0007] In a first aspect, an electronic device according to some embodiments of this application includes: a communication interface; a memory configured to store a computer program; and a processor connected to the memory and the communication interface, configured to execute the computer program to cause the electronic device to perform: determining a target text block matching a question to be answered based on each pre-saved text block; if non-text data is pre-saved for the target text block, obtaining an overview text of the saved non-text data, the non-text data including at least one of images and tables, the overview text being a summary of all content in the non-text data; and inputting the question to be answered, the target text block, and the overview text into a first large model according to a preset format to obtain response information.

[0008] Secondly, in some embodiments of this application, a question-answering method based on a large model is provided. The method includes: determining a target text block that matches the question to be answered based on each pre-saved text block; if non-text data is pre-saved for the target text block, obtaining an overview text of the saved non-text data, wherein the non-text data includes at least one of images and tables, and the overview text is a summary of all content in the non-text data; and inputting the question to be answered, the target text block, and the overview text into a first large model according to a preset format to obtain answer information.

[0009] Thirdly, in some embodiments of this application, a question-answering device based on a large model is provided. The device includes: a determining module, configured to determine a target text block matching a question to be answered based on each pre-saved text block; an obtaining module, configured to obtain an overview text of the saved non-text data if non-text data is pre-saved for the target text block, wherein the non-text data includes at least one of images and tables, and the overview text is a summary of all content in the non-text data; and a question-answering module, configured to input the question to be answered, the target text block, and the overview text into a first large model according to a preset format to obtain answer information.

[0010] Fourthly, this application also provides a computer-readable non-volatile storage medium storing a computer program that, when executed by a processor, implements the steps of the question-answering method based on the large model as described above. Attached Figure Description

[0011] Figure 1 is a schematic diagram of a RAG model in a related art provided by some embodiments of this application;

[0012] Figure 2 is a flowchart illustrating a question-and-answer process based on a large model, provided in some embodiments of this application.

[0013] Figure 3 is a schematic diagram of a question-and-answer process based on a large model provided in some embodiments of this application;

[0014] Figure 4 is a schematic diagram of a table format conversion provided in some embodiments of this application;

[0015] Figure 5 is a schematic diagram of an enhanced label screening process provided in some embodiments of this application;

[0016] Figure 6 is a schematic diagram of an enhanced label screening process provided in some embodiments of this application;

[0017] Figure 7 is a schematic diagram of an enhanced label determination process provided in some embodiments of this application;

[0018] Figure 8 shows an image to be stored according to some embodiments of this application;

[0019] Figure 9 is a flowchart illustrating another question-and-answer process based on a large model provided in some embodiments of this application;

[0020] Figure 10 is a schematic diagram of a LoRA fine-tuning process provided in some embodiments of this application;

[0021] Figure 11 is a schematic diagram of the structure of a reward model provided in some embodiments of this application;

[0022] Figure 12 is a schematic diagram of a word vector encoding provided in some embodiments of this application;

[0023] Figure 13 is a schematic diagram illustrating the principle of a hybrid retrieval process provided in some embodiments of this application;

[0024] Figure 14 is a schematic diagram of a classification model structure provided in some embodiments of this application;

[0025] Figure 15 is a schematic diagram of word vector encoding of input data for a classification model provided in some embodiments of this application;

[0026] Figure 16 is a schematic diagram of a target candidate text determination process provided by some embodiments of this application;

[0027] Figure 17 is a schematic diagram of a response information determination process provided in some embodiments of this application;

[0028] Figure 18 is a schematic diagram of a response information quality evaluation principle provided by some embodiments of this application;

[0029] Figure 19 is a schematic diagram of the principle of response information quality evaluation provided by some embodiments of this application;

[0030] Figure 20 is a schematic diagram of a question-and-answer process based on a large model provided in some embodiments of this application;

[0031] Figure 21 is a flowchart illustrating another large-model-based question-and-answer process provided in some embodiments of this application;

[0032] Figure 22 is a schematic diagram of an encoder structure provided in some embodiments of this application;

[0033] Figure 23 is a schematic diagram of a model input provided in some embodiments of this application;

[0034] Figure 24 is a schematic diagram of the structure of an embedded model provided in some embodiments of this application;

[0035] Figure 25 is a schematic diagram of the training of an embedding model provided in some embodiments of this application;

[0036] Figure 26 is a similarity diagram provided by some embodiments of this application;

[0037] Figure 27 is a schematic diagram of an architecture for fine-tuning training provided in some embodiments of this application;

[0038] Figure 28 is a schematic diagram of an architecture for fine-tuning training provided in some embodiments of this application;

[0039] Figure 29 is a schematic diagram of a similar text determination principle provided by some embodiments of this application;

[0040] Figure 30 is a schematic diagram of a similar text determination principle provided by some embodiments of this application;

[0041] Figure 31 is a schematic diagram illustrating the generation principle of a problem and overview text provided in some embodiments of this application;

[0042] Figure 32 is a schematic diagram illustrating the principle of generating response information according to some embodiments of this application;

[0043] Figure 33 is a flowchart illustrating a fourth large-model-based question-and-answer process provided in some embodiments of this application;

[0044] Figure 34 is a schematic diagram of a road hazard detection terminal provided in some embodiments of this application;

[0045] Figure 35 is a schematic diagram of the structure of a road hazard detection terminal provided in some embodiments of this application;

[0046] Figure 36 is a schematic diagram showing a list of locations to be investigated according to some embodiments of this application;

[0047] Figure 37 is a schematic diagram showing the investigation results provided in some embodiments of this application;

[0048] Figure 38 is a schematic diagram of the back of a terminal provided in some embodiments of this application;

[0049] Figure 39 is a schematic diagram of on-site investigation record input provided in some embodiments of this application;

[0050] Figure 40 is a schematic diagram of an image upload page provided in some embodiments of this application;

[0051] Figure 41 is a schematic diagram of on-site investigation record input provided by some embodiments of this application;

[0052] Figure 42 is a schematic diagram of on-site investigation record input provided by some embodiments of this application;

[0053] Figure 43 is a schematic diagram illustrating the determination of a target supplementary investigation location provided by some embodiments of this application;

[0054] Figure 44 is a schematic diagram showing the investigation results provided in some embodiments of this application;

[0055] Figure 45 is a schematic diagram of an electronic device structure provided in some embodiments of this application. Detailed Implementation

[0056] To make the objectives, solutions, and advantages of this application clearer, the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described drawings are merely some embodiments of this application. Those skilled in the art can obtain other drawings based on these drawings without any creative effort. The described embodiments are only a part of the embodiments of this application, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0057] It should be noted that the terms "comprising" and "having" and their variations used in this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0058] The terms "first" and "second" used in this document are for descriptive purposes only and should not be construed as indicating relative importance or implying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of some embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0059] Knowledge retrieval is the core process of the Retrieval-augmented Generation (RAG) model and the most important part of an enterprise knowledge management system. Its purpose is to ensure that users can quickly and accurately find the information they need. Figure 1 is a schematic diagram of a RAG model in a related art provided by some embodiments of this application. As shown in Figure 1, the text is pre-segmented into small segments, and then the Transformer encoder model, i.e., the embedding model, is used to convert these text segments into vector form and store them in memory. All vectors are then aggregated into a Vector store index. Finally, a cue word is defined for the Large Language Model (LLM). This cue word guides the LLM to answer the user's query based on the context information found in the database. The process of searching for context information, i.e., the knowledge retrieval process, generally includes a basic retrieval process and a complex retrieval process. Among them, a typical complex retrieval process includes steps such as pre-retrieval processing, knowledge retrieval, post-retrieval processing, and answer generation. Each search step involves numerous technical details, and the basic search process often omits pre- and post-processing steps, resulting in low search accuracy.

[0060] Some embodiments of this application provide a question-answering method and electronic device based on a large model. In this method, a target text block matching the question to be answered is determined according to each pre-saved text block. If non-text data is pre-saved for the target text block, an overview text of the saved non-text data is obtained. The non-text data includes at least one of images and tables. The overview text is a summary of all the contents of the non-text data. The question to be answered, the target text block, and the overview text are input into a first large model according to a preset format to obtain the answer information, thereby improving the accuracy of question answering.

[0061] Figure 2 is a flowchart illustrating a question-answering process based on a large model, as shown in Figure 1. This process includes, but is not limited to, the following steps:

[0062] S101: Based on each pre-saved text block, determine the target text block that matches the question to be answered.

[0063] The question-answering method based on a large model provided in some embodiments of this application is applied to electronic devices, such as servers, PCs, and mobile terminals.

[0064] Because large models are typically trained using general knowledge, their accuracy will not be high when applied to question-answering in a specific domain. Furthermore, they may provide incorrect answers because they haven't learned the relevant knowledge during pre-training. The large model can be any model with language processing capabilities, such as the aforementioned LLM.

[0065] To improve the accuracy of question-answering based on large models, in some embodiments of this application, the text that may be involved in the question-answering process can be pre-saved. Since the technical field applied to question-answering is known, documents, books, news, etc., related to that technical field can be collected in advance and saved. When saving, each document can be treated as a text block, or a document can be split into multiple text segments, and these multiple text segments can be saved as text blocks. In some embodiments, when segmenting the entire document, it can be segmented according to the number of characters, punctuation, or headings, for example, with N characters as a text block.

[0066] Upon receiving a question to be answered, target text blocks matching the question can be determined based on each pre-saved text block. In other words, to ensure accurate responses from the large model, one or more target text blocks related to the content of the question can be found within each pre-saved text block. When determining target text blocks, the similarity between the question and each text block can be determined, and text blocks with similarity greater than a set threshold can be identified as target text blocks. Alternatively, one or more text blocks with the highest or greatest similarity among those with similarity greater than a preset threshold can be selected as target text blocks. How to determine target text blocks matching the question is a related technology, which will not be elaborated upon in this embodiment.

[0067] S102: If non-text data is pre-saved for the target text block, then obtain the summary text of the saved non-text data. The non-text data includes at least one of images and tables. The summary text is a text that summarizes all the contents of the non-text data.

[0068] Because text block similarity matching can only compare the similarity between texts, it cannot compare the similarity between text and non-text data. This non-text data can be at least one of images or tables, and can also be video or audio. If a text segment corresponds to non-text data, it indicates that the non-text data contains more important information. However, large models cannot access the content recorded in the corresponding non-text data during question-and-answer processing, and therefore cannot consider the content recorded in images and / or tables when generating response information, resulting in low question-and-answer accuracy. Therefore, in some embodiments of this application, if a collected text block corresponds to non-text data, the corresponding non-text data can be saved for that text block. Since large models cannot directly process this non-text data, a summary text of the non-text data can be saved simultaneously when saving it. This summary text can be understood as a summary of all the content in the non-text data. This summary text can be manually annotated by relevant personnel after collection.

[0069] S103: Input the question to be answered, the target text block, and the overview text into the first large model according to a preset format to obtain the answer information.

[0070] After obtaining the target text block and its non-textual data summary text, the question to be answered, the target text block, and the summary text block can be input into the first main model according to a preset format to obtain the answer information. Since the summary text is a summary of all the content in the relevant non-textual data, the first main model can understand all the content recorded in the non-textual data based on the summary text, and thus generate a more accurate answer to the question to be answered based on the summary text and the target text.

[0071] In some embodiments of this application, during the question-and-answer process, a target text block matching the question to be answered is determined based on each pre-saved text block, i.e., a target text block related to the question to be answered is determined. If non-text data such as images and / or tables are pre-saved for the target text block, the summary text of the saved non-text data is obtained, and the question to be answered, the target text block, and the summary text are input into the first large model according to a preset format to obtain the answer information. Since the summary text is a summary of all the content in the non-text data, inputting it into the first large model allows the first large model to avoid directly processing the images and / or tables, while ensuring that it considers the content recorded in the images and / or tables when generating the answer information, thereby improving the question-and-answer accuracy of the large model.

[0072] To further improve the accuracy of question answering, based on the above embodiments, in some embodiments of this application, if the non-text data pre-saved for the target text block includes a target table, after obtaining the summary text of the saved non-text data and before inputting the question to be answered, the target text block, and the summary text into the first large model according to a preset format, the method further includes: obtaining enhanced tags pre-saved for each entry in the target table, wherein the enhanced tags are key information obtained by summarizing the content of the corresponding entry; determining a first similarity between the question to be answered and the enhanced tags corresponding to each entry; determining a target entry matching the question to be answered based on each first similarity, and obtaining the target index of the target entry. When inputting the question to be answered, the target text block, and the summary text into the first large model according to a preset format to obtain answer information, the method includes: inputting the question to be answered, the target text block, the summary text, all table content of the target table, and the target index into the first large model according to a preset format to obtain answer information.

[0073] Since the question to be answered may be directed at a specific item in the table, and the table's overview text summarizes the table as a whole, it may weaken the content recorded in a particular item when determining the overview information. This could lead to the first model being unable to obtain more information about that item when determining the answer, resulting in an incorrect answer to the question. Therefore, to further improve the accuracy of question answering, in some embodiments of this application, if it is determined that a table exists in the non-text data pre-saved for the target text block, after obtaining the overview text of the non-text data, the items in the existing table can be processed before inputting the relevant information into the first model. For ease of description, the existing table will be referred to as the target table. The items in the table can be understood as detailed categories grouped by content. For ease of understanding, the items are explained below with reference to Table 1.

[0074] Table 1

[0075] As shown in Table 1, each row records information about the same region. For example, row 2 records the traffic congestion index for Qingdao in April 2024, which was 1.790, a year-on-year decrease of 10.76%. Row 3 records the traffic congestion index for Jinan in April 2024, which was 1.769, a year-on-year increase of 13.62%. Although rows 2 and 3 both record traffic conditions, they reflect information from different cities. Therefore, row 2 and row 3 of Table 1 can be considered as one entry each.

[0076] In some embodiments of this application, after obtaining a summary of the saved non-text data, if it is determined that a target table exists in the non-text data, enhanced tags that have been pre-saved for each entry in the target table can be obtained. These enhanced tags can be understood as key information obtained by summarizing the content of the corresponding entry. For example, if an entry is "Scenario: Safety Status, Indicator: Traffic Accident Count, Indicator Value: 0", then the key information obtained by summarizing all the content recorded in that entry could be: traffic safety, traffic accidents, zero accidents, accident statistics, and road safety.

[0077] To determine whether a question to be answered is directed at a specific entry in the target table, in some embodiments of this application, a first similarity between the question to be answered and the enhanced tag corresponding to each entry can be determined. Each enhanced tag corresponding to each entry can be manually labeled by the data collector.

[0078] After determining each first similarity score, the target entry matching the question to be answered can be determined based on each first similarity score. That is, it is determined which entry record is related to the content of the question to be answered, and that related entry is identified as the target entry.

[0079] To facilitate the subsequent identification of which entry in the target table the question in the first major model is referring to, the target index of that entry can be obtained after the target entry has been determined. The target index can be the row number within the target table.

[0080] In some embodiments, the feature vector of the enhanced label corresponding to each entry in the target table is obtained. It should be noted that if an entry corresponds to multiple enhanced labels, then a single feature vector is determined based on a combination of these multiple enhanced labels. In some embodiments, the feature vector of each enhanced label can be determined separately. After obtaining the feature vector of each enhanced label, these feature vectors are fused to obtain the feature vector corresponding to the entry. For example, the sum or average of the feature vectors can be used as the feature vector of the corresponding entry. Assume a table includes three entries, m1, m2, and m3, where the feature vector corresponding to m1 is vm1: [0,1,0,1,1], the feature vector corresponding to m2 is vm2: [1,1,0,1,1], and the feature vector corresponding to m3 is vm3: [1,1,1,1,1]. The feature vector of the question to be answered is denoted as vu. The first similarity between the question to be answered and the enhanced label corresponding to each entry is determined. This can be achieved by calculating the first similarity between mi and vu using cosine similarity, where i is an integer representing the entry number. After determining each first similarity, find the maximum value of the first similarity and obtain the target index (Index) of the target entry through the maximum value. Here, Index can be represented as Index = max(cos(mi,vu)).

[0081] After obtaining the target index, when using the first main model to determine the response information, the question to be answered, the target text block, the overview text, the entire content of the target table, and the target index can be input into the first main model according to a preset format to obtain the response information. Since the main model cannot effectively process tabular data, the entire content of the target table input into the first main model can be text that has undergone format conversion. For example, the content of the target table can be generated in Markdown format.

[0082] In some embodiments, the input content of the first major model can be determined according to the following template:

[0083] <Complete Table> Markdown format of the table;

[0084] <Match Text> A text block related to the user's question;

[0085] <Rules> are rules for generating responses, such as "Based on the above content, answer the following user questions";

[0086] <Index Enhancement> The most relevant table index, for example, "entry 2 is known to be more relevant";

[0087] <User Issues> User's request to look up a table.

[0088] Among them, <Overall Table> is the entire table content of the target table; <Matching Text> is the target text block; <Rule> is the prompt text, that is, the text used to prompt the first major model what processing is to be performed; <Index Enhancement> is the target index determined; and <User Question> is the question to be answered.

[0089] The following describes the large-model-based question-answering process with reference to one or more embodiments. Figure 3 is a schematic diagram of a large-model-based question-answering process provided by some embodiments of this application. As shown in Figure 3, the process includes the following steps:

[0090] S201: Identify the target text block that matches the question to be answered.

[0091] S202: Determine whether non-text data is stored for the target text block. If yes, execute S203; otherwise, execute S204.

[0092] S203: Obtain the summary text saved for this non-text data, and execute S205.

[0093] S204: Input the question to be answered and the target text block into the first large model to obtain the information to be answered.

[0094] S205: Determine whether the non-text data includes the target table. If yes, proceed to S206; otherwise, proceed to S208.

[0095] S206: Determine the target index based on the enhanced labels of the target table and the questions to be answered, and then proceed to S207.

[0096] S207: Input the question to be answered, the target text block, the overview text, the complete table content of the target table, and the target index into the first large model to obtain the answer information.

[0097] S208: Input the question to be answered, the target text block, and the overview text into the first main model to obtain the answer information.

[0098] To improve the efficiency of enhanced tag determination, based on the above embodiments, in some embodiments of this application, the process of determining the enhanced tag corresponding to each item in any table includes: converting the format of the table to obtain all the table content included in the table; inputting the all table content and a first prompt text into a second large model to obtain the enhanced tag corresponding to each item in the table, wherein the first prompt text is used to prompt the second large model to analyze each item in the all table content and output a text summarizing the content of the corresponding item.

[0099] To improve the efficiency of augmented tag determination, when determining the augmented tag corresponding to each entry in any table, the table can be formatted to obtain all the table content. For example, the content in the table can be converted to Markdown format. After obtaining all the table content, a large model can be used to generate the augmented tag corresponding to each entry in the table. In some embodiments of this application, the entire table content and the first prompt text can be input into a second large model to obtain the augmented tag corresponding to each entry in the table. The second large model can be any large model with language analysis capabilities; it can be the same as or different from the first large model. The first prompt text is input into the second large model to prompt it to analyze each entry in the received entire table content and output a summary text of the corresponding entry content.

[0100] For ease of understanding, Figure 4 is a schematic diagram of a table format conversion provided by some embodiments of this application. As shown in Figure 4, after converting the table-formatted text (as shown on the left side of Figure 4) according to a fixed format, the complete table content (as shown on the right side of Figure 4) can be obtained.

[0101] In some embodiments, during the process of generating enhanced tags for each entry based on the second major model, prompt words can be input according to the following template:

[0102] <Tag Enhancement Rules> Provides a description of the steps or rules for tag generation.

[0103] <Example> Given the input and output of the sample table

[0104] Table field names and descriptions

[0105] For example, to determine the augmentation label for each entry in Table 2 below, you can input the following into the second-largest model.

[0106] Table 2

[0107] The input to the second major model can include the following:

[0108] #Based on the input fields, find the most relevant tags, with the following requirements:

[0109] - Separate output tags with commas.

[0110] - The output cannot contain company names, and should not include numerical results.

[0111] - Tags must be representative and express the overall meaning of the corresponding entry.

[0112] #enter:{

[0113] Item 1: Scenario: Vehicle and Driver Management; Indicator: Number of Vehicles; Indicator Value: 10

[0114] Item 2: Scenario: Security situation; Indicator: Traffic accident volume; Indicator value: 0

[0115] }

[0116] After receiving the above input, the second model can output the following:

[0117] Output: {

[0118] Item 1: Automobiles, vehicle ownership, traffic management, vehicle statistics, vehicle data, number of vehicles, traffic data

[0119] Item 2: Traffic safety, traffic accidents, zero accidents, accident statistics, road safety

[0120] }

[0121] Based on the output of the second major model, the enhanced labels corresponding to item 1 in Table 2 can include automobiles, vehicle ownership, traffic management, vehicle statistics, vehicle data, number of vehicles, and traffic volume; the enhanced labels corresponding to item 2 can include traffic safety, traffic accidents, zero accidents, accident statistics, and road safety.

[0122] After determining the enhancement tags, the corresponding table for each enhancement tag can be stored as shown in Table 3. The database stores the table's overview text, which is a table summary of the content recorded in the table, as well as the feature vector of each overview text. It also stores all the table content and the enhancement tag corresponding to each item.

[0123] Table 3

[0124] To further enhance the accuracy of label determination, when using the second-largest model to generate enhanced labels, a reference example can be input into the second-largest model so that the second-largest model learns the determination method of the reference example and generates enhanced labels for the table.

[0125] For example, suppose we need to generate the enhanced tags shown in Table 4 below.

[0126] Table 4

[0127] When generating the augmented labels for each entry in Table 4 above, the following can be input into the second-largest model:

[0128] #Based on the input fields, find the most relevant tags, with the following requirements:

[0129] - Separate output tags with commas.

[0130] - The output cannot contain company names, and should not include numerical results.

[0131] - Tags must be representative and express the overall meaning of the corresponding entry.

[0132] #Please refer to the example:

[0133] -enter:{

[0134] Item 1: Scenario: Vehicle and Driver Management; Indicator: Number of Vehicles; Indicator Value: 10

[0135] Item 2: Scenario: Security situation; Indicator: Traffic accident volume; Indicator value: 0

[0136] }

[0137] Output: {

[0138] Item 1: Automobiles, vehicle ownership, traffic management, vehicle statistics, vehicle data, number of vehicles, traffic data

[0139] Item 2: Traffic safety, traffic accidents, zero accidents, accident statistics, road safety

[0140] }

[0141] #enter:{

[0142] Item 1: Date: April 2024, Region: Qingdao, Traffic Congestion Index: 1.790, Year-on-Year: 10.76% Decrease Item 2: Date: April 2024, Region: Jinan, Traffic Congestion Index: 1.769, Year-on-Year: 13.62% Increase Item 3: Date: April 2024, ...

[0143] }

[0144] After receiving the above input, the second model can determine the augmented label for each item. For example, the output could be:

[0145] Item 1: Coastal city, nestled between mountains and water, seafood, pleasant climate, easy traffic, Laoshan scenery...

[0146] Item 2: Shandong provincial capital, Jinan (Spring City), Daming Lake, hospitable, traffic congestion...

[0147] Item 3: ...

[0148] To further improve the efficiency of enhancing tag determination, based on the above embodiments, in some embodiments of this application, the method further includes: for each enhancing tag corresponding to each entry, determining whether the semantics of the enhancing tag are repeated with any other enhancing tag corresponding to the entry; if so, deleting any enhancing tag with repeated semantics, and continuing to execute the step of determining whether the semantics of the enhancing tag are repeated with any other enhancing tag corresponding to the entry for each enhancing tag, until the semantics of the enhancing tags corresponding to each entry are no longer repeated.

[0149] Since the augmented tags generated by the second model may contain semantically repetitive content, these augmented tags can be filtered after obtaining the augmented tags corresponding to each entry. In some embodiments of this application, for each augmented tag corresponding to each entry, it can be determined whether the semantics of the augmented tag are repetitive with any other augmented tag corresponding to that entry. If so, the semantically repetitive augmented tag can be deleted, and for each of the remaining augmented tags corresponding to each entry, it can be determined whether there are semantically repetitive tags among the augmented tags corresponding to that entry, until the semantics of the augmented tags corresponding to each entry are no longer repetitive.

[0150] In some embodiments of this application, determining whether the semantics of the enhanced tag are repeated with any other enhanced tag corresponding to the entry includes: determining a second similarity between the enhanced tag and the other enhanced tags; if the second similarity is greater than a first preset threshold, then determining that the enhanced tag and the other enhanced tags are semantically repeated; if the second similarity is not greater than the first preset threshold, then determining that the enhanced tag and the other enhanced tags are not semantically repeated.

[0151] When determining whether an enhanced tag has semantic overlap with other enhanced tags, a second similarity can be determined between the enhanced tag and the other enhanced tags. Subsequently, based on the second similarity and a first preset threshold, it is determined whether semantic overlap exists. If the second similarity is greater than the first preset threshold, it is determined that the enhanced tag has semantic overlap with the other enhanced tags. Otherwise, it is determined that the enhanced tag does not have semantic overlap with the other enhanced tags.

[0152] In some embodiments, after obtaining each augmented tag, all augmented tags can be encoded using an embedding model to generate word vectors corresponding to each augmented tag. For example, if the augmented tags include "coastal city," "surrounded by mountains and water," etc., then the generated word vectors are as follows:

[0153] Coastal city: [0.1211, -1.231, ..., 1.546, -0.1234]

[0154] Nestled against mountains and beside water: [1.1782, 0.1232, ..., -1.784, 0.1232]

[0155] ...

[0156] Calculate the similarity between all pairwise word vectors, and set a first preset threshold p. The similarity calculation can be based on the following formula:

[0157] Where A and B represent two word vectors; similarity(A, B) represents the second similarity between word vectors A and B; a represents an element in word vector A; b represents an element in word vector B; and d represents the inner product of elements.

[0158] If the second similarity is determined to be greater than the first preset threshold p, it indicates that the two corresponding enhanced labels have semantic overlap, and only one of the enhanced labels can be retained. For example, if the enhanced labels are "coastal city" and "surrounded by mountains and water," since the second similarity of these two enhanced labels is greater than the first preset threshold p, one of the enhanced labels can be randomly deleted, such as deleting the enhanced label "surrounded by mountains and water."

[0159] If the second similarity is determined to be no greater than the first preset threshold p, it means that the two corresponding enhanced labels do not have semantic overlap, and both enhanced labels can be retained.

[0160] The following describes the enhanced tag filtering process with reference to one or more embodiments. Figure 5 is a schematic diagram of an enhanced tag filtering process provided by some embodiments of this application. As shown in Figure 5, an entry includes enhanced tag A and enhanced tag B. The second similarity between enhanced tag A and enhanced tag B can be determined in the aforementioned manner (S501). If the second similarity between enhanced tag A and enhanced tag B is determined to be greater than a preset threshold (S502), the semantic repetition between enhanced tag A and enhanced tag B can be determined (S503). Then, only enhanced tag A is retained as the enhanced tag of the entry (S504).

[0161] To facilitate understanding, the enhancement tag selection process is explained below with reference to Figure 6. Figure 6 is a schematic diagram of an enhancement tag selection process provided by some embodiments of this application. As shown in Figure 6, enhancement tags 1-6 are obtained. The similarity between each pair of enhancement tags is calculated. Enhancement tag 1 and enhancement tag 2 have semantic overlap, so one of them is retained. Assuming enhancement tag 2 is retained, enhancement tag 1 is deleted. Next, the similarity between enhancement tag 2 and enhancement tag 3 is calculated. Based on this similarity, it is determined that enhancement tag 2 and enhancement tag 3 do not have semantic overlap, so both enhancement tags are retained. Furthermore, enhancement tag 5 and enhancement tag 6 have semantic overlap, so one of them is retained. Assuming enhancement tag 5 is retained, enhancement tag 6 is deleted. Subsequently, it is determined that enhancement tag 4 and enhancement tag 5 have semantic overlap, so both enhancement tags 4 and 5 can be retained. Finally, through selection, enhancement tags 2, 3, 4, and 5 are retained.

[0162] To further improve the efficiency of question answering, based on the above embodiments, in some embodiments of this application, the method further includes: inputting the enhanced tag and second prompt text corresponding to each item into a third large model to obtain a summary tag for each item, wherein the second prompt text is used to prompt the third large model to summarize the enhanced tags corresponding to each item to generate representative tags; and updating the enhanced tags of the corresponding items using the summary tags corresponding to each item.

[0163] To further improve the accuracy of question answering, in some embodiments of this application, after obtaining the enhanced tags, the enhanced tags can be summarized to obtain representative summary tags. In some embodiments of this application, the enhanced tags and second prompt text corresponding to each item can be input into a third large model to obtain the summary tags for each item. Inputting the second prompt text into the third large model is to prompt the third large model to summarize the enhanced tags corresponding to each item and generate representative tags. The third large model can be any large model with language processing capabilities, and it can be the same as or different from the first and second large models.

[0164] After obtaining the summary tags for each entry, the enhanced tags for the corresponding entries can be updated using the summary tags for each entry.

[0165] In some embodiments, the input content of the third major model can be determined according to the following template:

[0166] <Tag Generation Rules> Provides a description of the steps or rules for generating tags.

[0167] <Example> Given the input and output of the sample table

[0168] Field names and descriptions in the <Labels to be generated> table

[0169] In some embodiments, the following can be input into the third major model:

[0170] #Based on the input fields, summarize them into representative labels, with the following requirements:

[0171] - Output 1-2 tags, separated by commas.

[0172] - The output tags can cover the meaning of multiple fields.

[0173] #Please refer to the example:

[0174] -enter:{

[0175] Item 1: Automobiles, Traffic Management, Vehicle Statistics, Traffic Data

[0176] }

[0177] Outputs: Vehicle statistics, traffic management

[0178] #enter:

[0179] To facilitate understanding, the process of determining enhanced tags will be described below with reference to one or more embodiments. Figure 7 is a schematic diagram of an enhanced tag determination process provided by some embodiments of this application. As shown in Figure 7, the entire content of the table and the first prompt text are input into the second large model (S701) to obtain the enhanced tags corresponding to each item (S702). Enhanced tags with semantic repetition are removed, and the remaining enhanced tags are semantically distinct (S703). The remaining enhanced tags and the second prompt text corresponding to each item are input into the third large model (S704) to obtain the summary tags for each item (S705), and these summary tags are used as the final enhanced tags (S706).

[0180] To further improve the accuracy of question and answer, based on the above embodiments, in some embodiments of this application, the process of determining the summary text of the image includes:

[0181] The images to be stored are processed based on a multimodal large model to obtain an overview text of the images to be stored.

[0182] In some embodiments of this application, a multimodal large model can be used to determine the summary text of an image. That is, the image to be stored can be processed based on the multimodal large model to obtain the summary text of the image to be stored. The image to be stored can be an image from a document.

[0183] In some embodiments, Figure 8 is an image to be stored according to some embodiments of this application. As shown in Figure 8, there are a large number of cars traveling on the road, and the distance between the cars is relatively short. Most of the cars have their brake lights on, indicating that there is a traffic jam on this road. Based on the multimodal large model, the summary text obtained by processing Figure 8 can be: Traffic congestion has occurred on xxx road.

[0184] After obtaining the image's summary text, the image to be stored and its corresponding summary text can be stored in a database. In some embodiments, the storage can be performed according to the contents shown in Table 5.

[0185] Table 5

[0186] As shown in Table 5, after determining the image's summary text, the image identifier, the image's summary text, the feature vector of the summary text, and the image's address / url can be stored. The feature vector of the summary text can be generated by an embedding model. Storing the feature vector of the summary text facilitates subsequent similarity determination, eliminating the need to determine the feature vector of the summary text each time similarity is calculated.

[0187] To further improve the accuracy of question and answer, based on the above embodiments, in some embodiments of this application, the process of determining the summary text of the table includes: converting the format of the table to be stored to obtain all table content; and processing the all table content based on the fourth major model to obtain the summary text of the table to be stored.

[0188] In some embodiments of this application, the determination of the summary text of a table can be based on a large model. Since a large model cannot process tabular data, the table to be stored can be formatted to obtain all its content. This table to be stored can be a table within a document. In some embodiments, after obtaining a document, a table recognition tool can be used to locate the table. How to locate a table within an article is a known technique, and will not be elaborated upon in some embodiments of this application.

[0189] After obtaining the complete content of the table to be stored, a summary text of the table can be generated using a large model. In other words, the entire table content is processed based on the fourth large model to obtain the summary text of the table to be stored. After obtaining the summary text, the summary text, its feature vector (embedding), and the table content can be stored in Markdown format. The feature vector of the summary text can be generated by the embedding model.

[0190] In some embodiments, assuming the table to be stored is Table 6, it can be stored according to the example in Table 7 when storing it in the database.

[0191] Table 6

[0192] Table 7

[0193] To further improve the accuracy of question answering, based on the above embodiments, in some embodiments of this application, the process of determining whether non-text data is pre-saved for the target text block includes: obtaining a first feature vector of the overview text of each saved non-text data; determining a third similarity between the second feature vector of the target text block and each first feature vector; and determining whether there is corresponding non-text data in the pre-saved target text block based on each third similarity and a second preset threshold.

[0194] When determining whether non-text data is stored for a target text block, a first feature vector of the pre-stored overview text of each non-text data can be obtained, and a third similarity between the second feature vector of the target text block and each first feature vector can be determined. This third similarity can be a cosine similarity.

[0195] After determining each third similarity, it is possible to determine whether there is corresponding non-text data in the pre-saved target text block based on each third similarity and the second preset threshold.

[0196] In some embodiments, each non-text data with a third similarity greater than a second preset threshold can be identified as non-text data to be stored for the target text block.

[0197] To further improve the accuracy of question answering, based on the above embodiments, in some embodiments of this application, the process of determining the second feature vector of the target text block includes: dividing the target text block into multiple text sub-blocks according to a preset segmentation rule; and determining the third feature vector of each text sub-block as the second feature vector. The step of determining the third similarity between the second feature vector of the target text block and each first feature vector includes: for each text sub-block, determining the third similarity between the second feature vector of that text sub-block and each first feature vector.

[0198] Target text blocks typically contain many sentences, and the dimension of the feature vector of a target text block is fixed. The feature vector of an entire text segment may mask subtle features of a single sentence, resulting in low accuracy in similarity comparison. Therefore, to further improve the accuracy of question answering, in some embodiments of this application, when determining the second feature vector of a target text block, the target text block can be divided into multiple sub-blocks according to a preset segmentation rule. In some embodiments of this application, each sentence in the target text block can be considered as a sub-block; alternatively, the target text block can be divided according to a set length to obtain multiple sub-blocks, meaning that each sub-block has the same length.

[0199] After obtaining each text sub-block, the third feature vector of each text sub-block can be determined as the second feature vector.

[0200] When determining the third similarity, the third similarity between the second feature vector of each text sub-block and each first feature vector can be determined separately.

[0201] For example, suppose the target text block N contains three text sub-blocks, n1, n2, and n3, where the third feature vector of n1 is v1: [0,1,0,1,1]; the third feature vector of n2 is v2: [1,1,0,1,1]; and the third feature vector of n3 is v4: [1,1,1,1,1]. Assume the first feature vectors of the pre-saved summary text of non-textual data are vf1, vf2, vf3, ... . The second preset threshold is pf, and the value range of this second preset threshold is [0,1].

[0202] Therefore, when determining the third similarity, the third feature vector of each text sub-block can be sequentially cosine similarity calculated with vfi (i is an integer representing the non-text data number), resulting in cos1, cos2, cos3, ... . Here, cos1 = v1@vf1, cos2 = v2@vf1, cos3 = v3@vf1, ..., where v1@vf1 represents the similarity between v1 and vf1.

[0203] If any third similarity score is greater than or equal to the second preset threshold pf, it can be assumed that non-text data has been pre-stored for the target text block, and the address of the corresponding non-text data can be obtained. It should be noted that the non-text data stored for the target text block can be a set; that is, multiple non-text data can be stored for the target text block. For example, the target text block may contain three images, a table, and an audio file.

[0204] Based on the above embodiments, some embodiments of this application also provide an electronic device. Figure 45 is a schematic diagram of the structure of an electronic device provided by some embodiments of this application. As shown in Figure 45, it includes: a processor 4501, a communication interface 4502, and a memory 4503. The memory 4503 is configured to store a computer program. The processor 4501 is connected to the communication interface 4502 and the memory 4503 and is configured to execute the computer program to cause the electronic device to perform the following: determining a target text block that matches the question to be answered based on each pre-saved text block; if non-text data is pre-saved for the target text block, obtaining the summary text of the saved non-text data, wherein the non-text data includes at least one of images and tables, and the summary text is a text summarizing all the content in the non-text data; and inputting the question to be answered, the target text block, and the summary text into a first large model according to a preset format to obtain the answer information.

[0205] In one possible implementation, if the non-text data pre-saved for the target text block includes a target table, after obtaining the summary text of the saved non-text data and before inputting the question to be answered, the target text block, and the summary text into the first large model according to a preset format, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform: obtaining enhanced tags pre-saved for each entry in the target table, the enhanced tags being key information obtained by summarizing the content of the corresponding entry; determining a first similarity between the question to be answered and the enhanced tags corresponding to each entry; determining a target entry matching the question to be answered based on each first similarity, and obtaining the target index of the target entry; when the processor 4501 performs the step of inputting the question to be answered, the target text block, and the summary text into the first large model according to a preset format to obtain response information, it is specifically configured to execute the computer program to cause the electronic device to perform: inputting the question to be answered, the target text block, the summary text, the entire table content of the target table, and the target index into the first large model according to a preset format to obtain response information.

[0206] In one possible implementation, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform: format conversion of the table to obtain all table content included in the table; inputting the all table content and a first prompt text into a second large model to obtain an enhanced label corresponding to each item in the table, wherein the first prompt text is used to prompt the second large model to analyze each item in the all table content and output a text summarizing the content of the corresponding item.

[0207] In one possible implementation, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform: for each enhancement tag corresponding to each entry, determine whether the semantics of the enhancement tag are repeated with any other enhancement tag corresponding to the entry; if so, delete any enhancement tag with repeated semantics, and continue to perform the step of determining whether the semantics of the enhancement tag are repeated with any other enhancement tag corresponding to the entry for each enhancement tag, until the semantics between enhancement tags corresponding to each entry are no longer repeated.

[0208] In one possible implementation, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform: determining a second similarity between the enhanced tag and the other enhanced tags; if the second similarity is greater than a first preset threshold, determining that the enhanced tag is semantically repetitive with the other enhanced tags; if the second similarity is not greater than the first preset threshold, determining that the enhanced tag is semantically non-repetitive with the other enhanced tags.

[0209] In one possible implementation, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform: inputting the enhanced tag and second prompt text corresponding to each entry into a third large model to obtain a summary tag for each entry, wherein the second prompt text is used to prompt the third large model to summarize the enhanced tags corresponding to each entry to generate representative tags; and updating the enhanced tags of the corresponding entries using the summary tags corresponding to each entry.

[0210] In one possible implementation, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform: processing the image to be stored based on a multimodal large model to obtain an overview text of the image to be stored.

[0211] In one possible implementation, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform: format conversion of the table to be stored to obtain all table content; and processing of the all table content based on a fourth major model to obtain an overview text of the table to be stored.

[0212] In one possible implementation, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform: acquiring a first feature vector of an overview text for each saved non-text data; determining a third similarity between a second feature vector of the target text block and each of the first feature vectors; and determining, based on each third similarity and a second preset threshold, whether corresponding non-text data exists in the pre-saved target text block.

[0213] In one possible implementation, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform: dividing the target text block into multiple text sub-blocks according to a preset segmentation rule; and determining the third feature vector of each text sub-block as the second feature vector.

[0214] When the processor 4501 executes the process of determining the third similarity between the second feature vector of the target text block and each first feature vector, it is specifically configured to execute the computer program to cause the electronic device to perform: for each text sub-block, determining the third similarity between the second feature vector of the text sub-block and each first feature vector.

[0215] It should be noted that this application does not limit the specific connection medium between the memory 4503 and the processor 4501. In some embodiments of this application, the memory 4503 and the processor 4501 communicate with each other via a communication bus 4504. The connection methods between other components are only illustrative and are not intended to be limiting. For ease of illustration, only one thick line is used in Figure 45, but this does not indicate that there is only one bus or one type of bus.

[0216] In some embodiments, the communication bus mentioned in the above-described electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used to represent it in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface 4502 is also configured for communication between the above-described electronic device and other devices. The memory 4503 may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. In some embodiments, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0217] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0218] Some embodiments of this application also provide another question-answering method based on a large model. In this method, a fifth large model, pre-trained for the target domain, is used to standardize and rewrite the question to be answered, obtaining a standard question corresponding to the question to be answered. Target candidate text matching the standard question is searched in the knowledge base. The target candidate text, the standard question, and the prompt text are concatenated to obtain input data, which is then input into a sixth large model to obtain response information. This response information is the text obtained by the sixth large model after analyzing the target candidate text and the standard question according to the requirements of the prompt text. Figure 9 is a flowchart illustrating another question-answering process based on a large model provided in some embodiments of this application. As shown in Figure 9, this process includes the following steps:

[0219] S901: Based on the fifth model pre-trained for the target domain, the question to be answered is standardized and rewritten to obtain the standard question corresponding to the question to be answered.

[0220] User questions may be too colloquial, which can reduce search accuracy. To address this, some embodiments of this application generalize user questions, making them more professionally worded. For example, a user's question, "My TV suddenly stopped displaying pictures, what should I do to fix it?" can be standardized as "What should I do if my TV screen is black?" or "What should I do if my TV screen isn't displaying anything?" Similarly, a user's question, "My TV and computer suddenly stopped displaying pictures, what should I do to fix them?" can be standardized as "What should I do if my TV screen is black?", "What should I do if my computer screen is black?", "What should I do if my TV screen isn't displaying anything?", or "What should I do if my computer screen isn't displaying anything?".

[0221] When standardizing and rewriting questions to be answered, the simplest method is to maintain a thesaurus and perform synonym replacement based on this thesaurus. Generally, the search engine can identify common synonyms by calculating vector values. However, when a user's question contains specialized terminology and its synonyms from specific fields, the search engine may fail to accurately find relevant content from the thesaurus. For example, a synonym for "screen flickering" is "LCD screen malfunction." When a user's question is "How do I fix my TV's screen flickering?", the expected match during knowledge base matching is text related to "LCD screen malfunction." However, in actual matching, because "screen flickering" and "LCD screen malfunction" are domain-specific terms, the search engine may not understand their relevance when analyzing similarity, potentially resulting in a mismatch. Therefore, when a question contains synonyms that cannot be semantically identified, maintaining a thesaurus can ensure the search engine can accurately retrieve relevant knowledge content.

[0222] While the aforementioned synonym matching scheme can quickly resolve the issue, user questions to be answered are typically diverse, making it impossible to exhaustively enumerate synonyms when constructing a synonym list. Therefore, in some embodiments of this application, the standardized rewriting of the questions to be answered can be based on the fifth major model. However, currently available open-source large models are trained on broad-domain datasets, lacking domain-specific knowledge and thus failing to achieve good results in specific domains. Therefore, in some embodiments of this application, a general model can be pre-trained for the target domain to obtain the fifth major model. Then, in the question-answering process based on the large model, the questions to be answered are standardized and rewritten based on this fifth major model to obtain the standard questions corresponding to the questions to be answered.

[0223] In some embodiments of this application, the standardized rewriting of the question to be answered may include part or all of the following: reference resolution, subquery decomposition, and query rewriting. Subquery decomposition can be understood as breaking down a question to be answered into multiple sub-questions, and query rewriting can be understood as standardizing the rewriting of the question to be answered.

[0224] S902: Search the knowledge base for target candidate texts that match the standard question.

[0225] After obtaining the standard question corresponding to the question to be answered, target candidate text matching the standard question can be searched in the knowledge base. This target candidate text can be understood as text related to the content included in the standard question. The knowledge base can be a pre-stored database for this target domain or an online web database. The knowledge base can include documents, news, and other information.

[0226] S903: The target candidate text, the standard question, and the prompt text are concatenated to obtain input data, and the input data is input into the sixth model to obtain response information. The response information is the text obtained by the sixth model after analyzing the target candidate text and the standard question according to the requirements of the prompt text.

[0227] After obtaining the target candidate text, the target candidate text, standard question, and prompt text can be concatenated to obtain input data. This input data is then fed into the sixth model, which analyzes the target candidate text and standard question according to the requirements of the prompt text to determine the response information.

[0228] In some embodiments of this application, the concatenated input data may be in the following format:

[0229] '<Instruction>You are a Q&A assistant, and your current task is to answer questions based on the known information using your expertise. If you cannot obtain an answer from it, please say "The question cannot be answered based on the known information",'

[0230] 'It is not allowed to add fabricated content to the answer, and the answer should be in Chinese.< / Instruction>\n'

[0231] '<Known Information>{{top-k_context}}< / Known Information>\n'

[0232] '<Question>{{question}}< / Question>\n.'

[0233] In the above input data, "<Instruction>...< / Instruction>\n" represents the prompt text, "<Known Information>{{top-k_context}}< / Known Information>\n" represents the target candidate text, where {{top-k_context}} represents the specific content included in the target candidate text, and "<Question>{{question}}< / Question>\n" represents the standard question, where {{question}} represents the specific content included in the standard question.

[0234] During the experiment, it was learned that the attention characteristics of the large model for sentences are as follows: the large model pays more attention to the content at the head of the input to the model, and the model has a better understanding ability for the starting part of the paragraph. Therefore, in some embodiments of this application, the input data can also be in the following format:

[0235] '<Known Information>{{top-k_context}}< / Known Information>\n'

[0236] '<Question>{{question}}< / Question>\n'

[0237] '<Instruction>You are a Q&A assistant, and your current task is to answer questions based on the known information using your expertise. If you cannot obtain an answer from it, please say "The question cannot be answered based on the known information",'

[0238] 'It is not allowed to add fabricated content to the answer, and the answer should be in Chinese.< / Instruction>\n.'

[0239] According to the above content, in some embodiments of this application, when performing text splicing, the splicing is carried out in the order of the candidate text, the standard question, and the prompt text to ensure that important information is located at the head of the data.

[0240] In some embodiments of this application, a fifth model pre-trained for the target domain is used to standardize and rewrite the question to be answered, obtaining a standard question corresponding to the question to be answered. The target candidate text matching the standard question is then searched in the knowledge base. The target candidate text, the standard question, and the prompt text are then concatenated to obtain input data, which is then input into a sixth model. This allows the sixth model to acquire knowledge of the target domain and analyze the target candidate text and the standard question according to the requirements of the prompt text to obtain the answer information, thereby improving the accuracy of the large model in question answering in a specific domain.

[0241] The fine-tuning training process of the fifth major model trained for the target domain is described below with reference to one or more embodiments. In order to obtain the fifth major model trained for the target domain, in some embodiments of this application, the general large model can be fine-tuned, the large model after fine-tuning can be evaluated using a reward model, and finally reinforcement learning can be performed for fine-tuning.

[0242] In some embodiments of this application, supervised fine-tuning (SFT) training of open-source large models can be performed based on LoRA, such as the Qianwen large model. Figure 10 is a schematic diagram of a LoRA fine-tuning process in some embodiments of this application. In the LoRA fine-tuning process, the original pretrained weights parameter W is frozen, and an additional ΔW is trained. ΔW consists of two matrices A and B. Initially, A is a random matrix, and B is a matrix of all zeros. The LoRA fine-tuning process is essentially training ΔW, which can be expressed as: Y = (W + ΔW)X = (W + AB)X

[0243] According to the above formula, the parameters for LoRA fine-tuning training are matrix A and matrix B, while the original weight parameter W remains unchanged.

[0244] After the large model completes SFT training, the next stage is to build a reward model to score and evaluate question-answer pairs. In this stage, the dataset can be constructed as shown in the following example:

[0245] For example: What should I do if the TV screen suddenly goes black and stops displaying anything?

[0246] Response 1: What are the reasons and solutions for a TV screen that turns black instead of displaying a picture?

[0247] Response 2: What are the solutions for when the TV cannot play content?

[0248] The task of the reward model is to train a model using a pair of (prompt, response) and a reward score, which can be viewed as a classification or regression task. The reward model scores each text pair to evaluate the performance of the fine-tuned model. Compared to the model trained by SFT, the Reward Model adds a value head, which can be viewed as a linear layer with an input dimension equal to the model's hidden_dim and an output dimension of 1, representing the score the model predicts for each character. In other words, the reward model takes a question and its corresponding answer as input and outputs a scalar score.

[0249] Figure 11 is a schematic diagram of the structure of a reward model according to some embodiments of this application. As shown in Figure 11, the question-answer pair is input into the reward model, which adds a Value head to the original Model, and the final output is a score.

[0250] For example, x: Question to be answered (What should I do if the TV screen suddenly goes black?).

[0251] y best The standard problem of getting high rewards for output (the reasons why the TV screen turns black and cannot display a picture, and the solutions).

[0252] y worse The problem of low output rewards (a solution for when TV cannot play content).

[0253] For each training sample (x, y) best ,y worse You can get the score. best and score worse Among them, score best =r θ (x,y best This can be understood as the reward model for y. best The score worse =r θ (x,y worse This can be understood as the reward model for y. worse The score.

[0254] In some embodiments of this application, maximizing the top-ranked and bottom-ranked outputs is sufficient for the model to distinguish between good and bad outputs as much as possible. Training the Reward Model is essentially a ranking task, with the loss function being: loss = -log(σ(r) θ (x,y best )-r θ (x,yworse )))

[0255] Where loss represents the loss value, σ represents the parameter to be optimized, and r θ (x,y best ) indicates that the reward model for y best The rating, r θ (x,y worse ) indicates that the reward model for y worse The score.

[0256] It should be noted that in some embodiments of this application, the SFT training in the first step can be completed first, and then the training of the reward model in this step can be completed.

[0257] For example, assuming the reward model has a maximum score of 5 points, when the reward model scores, y best =5, y worse =1, after normalization, becomes 0.8, and the loss is -log(0.8); when y best =5, y worse =4, after normalization, becomes 0.2, and the loss is -log(0.2).

[0258] In some embodiments of this application, the purpose of the loss function is to enable the model to distinguish between top-ranked and bottom-ranked outputs as much as possible. In the third step of reinforcement learning, the reward model will act as the environment, assigning reward scores to the characters predicted by the model.

[0259] In the reinforcement learning fine-tuning phase, reinforcement learning helps train a large model, enabling it to generate more appropriate responses based on user prompts. To prevent the model from going astray during training updates, a reference model is set up for correction. During reinforcement learning fine-tuning, the question X to be answered can be input into the initial large model and the currently fine-tuned large model, yielding output texts y1 and y2 respectively. The text from the current policy is then passed to the reward model to obtain a scalar score rθ. A penalty term is calculated to compare the generated texts of the two models, typically designed as a scaling of the relative entropy (Kullback-Leibler, KL) between the output word distribution sequences. The reinforcement phase continuously optimizes the parameters iteratively, maximizing the scoring function and minimizing the data distributions of the two models. In some embodiments, the loss function is as follows:

[0260] in, Here, x represents the model input, y represents the model output, RL represents the reference model (it has no actual parameter meaning and is only used as a label), SFT represents the fine-tuned model (again, it only serves to distinguish the models), β is the data distribution modulation parameter between the two models (it can be set empirically), r represents the reward model, and θ is the parameter that the reward model needs to train. E(x,y) represents the maximum expected value. π represents the probability that the answer y is obtained from the reference model for question x; SFT (y|x) represents the probability that the answer y is obtained by fine-tuning the model for question x.

[0261] It should be noted that the fine-tuning training process for large models described above in some embodiments of this application is only for ease of understanding. How to fine-tune training large models is a known technique, and some embodiments of this application will not elaborate on it further.

[0262] To improve the efficiency of question answering, based on the above embodiments, in some embodiments of this application, after obtaining the standard question corresponding to the question to be answered, and before searching for target candidate text matching the standard question in the knowledge base, the method further includes: determining the target intent corresponding to the standard question based on a classification model. The step of searching for target candidate text matching the standard question in the knowledge base includes: searching for target candidate text matching the standard question in the target knowledge base corresponding to the target intent.

[0263] Since knowledge bases typically store a large amount of data, in some embodiments of this application, different knowledge bases can be pre-saved for different intents. After determining the standard question to be answered, before searching for the corresponding target candidate text in the knowledge base, the target intent corresponding to the standard question can be determined based on a classification model. When the target candidate text is determined, the search is performed only based on the target knowledge base corresponding to that target intent. In some embodiments of this application, the classification model can be any model with classification capabilities, and this application does not impose any restrictions on it. The target intent can be inquiring about the weather, policies, vehicles, etc., and this application does not impose any restrictions on it. Furthermore, how to determine the target intent based on a trained classification model is a known technique, and this application will not elaborate on this aspect.

[0264] To obtain a classification model with higher accuracy, based on the above embodiments, in some embodiments of this application, the classification model includes a feature extraction layer and a linear layer. The training process of the classification model includes: obtaining word vector codes and labels corresponding to sample text, wherein the labels are used to identify the standard intent of the sample text, and the word vector codes include part-of-speech identifiers for each character; inputting the word vector codes and the labels into the feature extraction layer, wherein the feature extraction layer determines the focus of feature extraction based on the part-of-speech identifiers when extracting semantic features, and obtains predicted semantic features; inputting the predicted semantic features into the linear layer for intent prediction, and obtaining the prediction probability corresponding to each preset intent; determining the target prediction probability corresponding to the standard intent based on the prediction probability corresponding to each preset intent; determining a first loss value based on the target prediction probability and preset control parameters, and adjusting the parameters of the classification model based on the first loss value.

[0265] In some embodiments of this application, a sample set is pre-configured to obtain a classification model with higher accuracy. The classification model can be trained based on the sample text included in the sample set. To facilitate subsequent training of the classification model, the sample set contains not only each sample but also the corresponding word vector codes and labels. The word vector codes are used to convert the text content into a vector representation that is easy for electronic devices to recognize. The labels are used to represent the standard intent of the corresponding sample text.

[0266] In classification models, the text category is often determined by meaningful content words, while ineffective words such as adjectives and stop words are often not decisive. Therefore, in some embodiments of this application, word vector encoding also includes part-of-speech tagging for each character. To improve the model's attention to content words and classification accuracy, some embodiments of this application pre-record characters that may appear in the sample set. Characters that are recorded and are not stop words or adjectives are marked with a part-of-speech tag of 1, while those not recorded are marked with a part-of-speech tag of 0, preventing the classification model from paying too much attention to some useless words. Figure 12 is a schematic diagram of word vector encoding provided by some embodiments of this application. As shown in Figure 12, the word vector encoding includes position encoding, context encoding, character encoding, and prior importance encoding. Among them, position encoding is used to identify the position of the corresponding character in the text. Context encoding is used to identify which text the corresponding character belongs to. Prior importance encoding is the part-of-speech tagging of the corresponding character. As shown in Figure 12, the part-of-speech tags for "TV black screen" and "processing" are both 1, indicating that these two words are relatively important and need to be focused on.

[0267] After obtaining the word vector codes and labels corresponding to the sample text, these codes and labels can be input into the feature extraction layer of the classification model. This layer then performs feature extraction on the received word vector codes. Since the word vector codes include part-of-speech tags for each character, the feature extraction layer can determine the focus of feature extraction based on these tags when extracting semantic features. For example, during feature extraction, based on an attention mechanism, it can pay more attention to the semantics of characters with a part-of-speech tag of 1, thereby obtaining predicted semantic features. In some embodiments of this application, this feature extraction layer can be viewed as a Bidirectional Encoder Representations from Transformers (BERT) model; however, it can also be other models, which can be configured as needed by those skilled in the art.

[0268] After obtaining the predicted semantic features of the sample text, the predicted semantic features can be input into a linear layer for intent prediction. Since the classification includes several known preset intents, in some embodiments of this application, the predicted probability corresponding to each preset intent can be obtained based on the linear layer.

[0269] To facilitate the determination of the loss value and thus the adjustment of the classification model parameters, in some embodiments of this application, the target prediction probability corresponding to the standard intent in the label of the sample text can be determined based on the prediction probability corresponding to each preset intent. In other words, it determines what the classification model's prediction probability of the standard intent corresponding to the sample text should be during the intent recognition process.

[0270] After determining the target prediction probability, the first loss value can be determined based on the target prediction probability and the preset control parameters, and the parameters of the classification model can be adjusted based on the first loss value.

[0271] In some embodiments, imbalanced samples often exist in private datasets, and user expressions are diverse, potentially including difficult examples such as Chinese inverted sentences. Therefore, this application controls positive and negative samples and easy and difficult classification samples in the loss function of this part, and the first loss value can be calculated based on the following formula: L obj =-a t ·(1-p t ) γ log(p t )

[0272] Among them, L obj p represents the first loss value. t This represents the predicted probability of the target, i.e., the probability value corresponding to the standard classification; a tα represents the control parameter for positive and negative samples, and its contribution to the loss is controlled by setting α; γ is the modulation coefficient, (1-p t ) γ This is used to control how easily each sample can be distinguished.

[0273] Where, p t The calculation method is as follows:

[0274] p represents the probability value of a positive sample, and y = 1 represents the label value of a positive sample.

[0275] In some embodiments of this application, the predicted intent of the sample text can be determined based on the predicted probability corresponding to each preset intent. If the predicted intent is consistent with the standard intent, it can be determined as a positive sample, i.e., y = 1. Then, P... t This represents the target prediction probability corresponding to the standard intent. If the predicted intent differs from the standard intent, it can be identified as a negative sample, i.e., belonging to the "otherwise" case, in which case P... t This is the target prediction probability corresponding to 1-standard intention, i.e., 1-p.

[0276] Among them, a t The calculation method is as follows:

[0277] When the label y = 1, αt = α; when the label y = otherwise, αt = 1 - α. The range of α is 0 to 1. At this time, we can control the contribution of positive and negative samples to the first loss value by setting α.

[0278] To further improve the accuracy of question answering, based on the above embodiments, in some embodiments of this application, the step of searching for target candidate texts matching the standard question in the knowledge base includes: determining the similarity between the standard question and each text based on the first feature vector of the standard question and the second feature vector of each text in the knowledge base; determining the relevance score between the standard question and each text in the knowledge base based on a keyword retrieval algorithm; and determining the target candidate texts matching the standard question in the knowledge base based on each similarity and each relevance score.

[0279] To further improve the accuracy of question answering, in some embodiments of this application, a hybrid search, such as vector search and keyword search, can be performed when retrieving target candidate text.

[0280] The advantages of vector retrieval include: high accuracy in searching complex semantic texts, high accuracy in understanding similar semantics, multilingual understanding (e.g., high accuracy in matching Chinese with English), and high fault tolerance (able to handle spelling errors and vague descriptions). While vector retrieval has significant advantages in the above scenarios, it is less effective in certain situations. For example, the accuracy of searching for a person's or object's name (e.g., searching for "Zhang San"), searching for abbreviations or phrases, and searching for IDs (e.g., searching for "gpt-3.5-turbo" or "titan-xlarge-v1.01"). These disadvantages are precisely the strengths of traditional keyword search. For instance, the traditional keyword retrieval algorithm BM25 (Best Matching 25) can achieve exact matching, such as matching product names, personal names, or product numbers; it can also match with a small number of characters. For example, vector retrieval with a small number of characters is very ineffective, but many users are accustomed to entering only a few keywords.

[0281] Based on the advantages of vector retrieval and keyword search, in some embodiments of this application, when matching target candidate texts, the similarity between the standard question and each text is determined according to the first feature vector of the standard question and the second feature vector of each text in the knowledge base; that is, vector retrieval is performed in the knowledge base. Simultaneously, based on the keyword retrieval algorithm, a relevance score is determined between the standard question and each text in the knowledge base; that is, keyword retrieval is performed in the knowledge base.

[0282] After obtaining the similarity and relevance scores for each text in the knowledge base, target candidate texts that match the standard question can be determined. In some embodiments of this application, the text with the highest similarity and the text with the highest relevance score can be determined as target candidate texts that match the standard question. Alternatively, the top-k texts with similarity scores higher than threshold 1 and the top-k texts with relevance scores higher than threshold 2 can be determined as target candidate texts.

[0283] Figure 13 is a schematic diagram illustrating the principle of a hybrid retrieval process provided in some embodiments of this application. As shown in Figure 13, when determining target candidate texts, after obtaining the user question, the feature vector of the user question can be determined based on a pre-trained embedding model, and the similarity between this feature vector and each feature vector in the vector database corresponding to the knowledge base can be determined, thereby obtaining the top-K texts with a value higher than a threshold. The vector database stores the feature vectors corresponding to each text in the knowledge base. Simultaneously, keyword retrieval is performed based on the user question and the text in the knowledge base. During keyword retrieval, the relevance score corresponding to each text can be determined based on the BM25 algorithm, thereby obtaining the top-K texts with a value higher than a threshold. The obtained top-K*2 texts are determined as target candidate texts.

[0284] It should be noted that since this stage is a coarse-sorting process, the threshold set in some embodiments of this application can be a low threshold.

[0285] To further improve the accuracy of question answering, based on the above embodiments, in some embodiments of this application, the step of determining the target candidate text matching the standard question in the knowledge base according to each similarity score and each relevance score includes: for each text in the knowledge base, determining the target score corresponding to the text according to the similarity score, relevance score and preset weight of the text; and determining the target candidate text according to each target score.

[0286] When determining target candidate texts that match the standard question in the knowledge base based on each similarity and relevance score, a target score can be determined for each text in the knowledge base based on its similarity, relevance score, and preset weight. Target candidate texts are then determined based on each obtained target score.

[0287] In some embodiments, the target score for any text can be determined based on the following formula: score result =α*score(BM25)+b*score(Embedding)

[0288] Among them, score result represents the target score; score(BM25) represents the relevance score, which is calculated based on the BM25 algorithm; score(Embedding) represents the similarity; α and b represent the preset weights, where α+b=1.

[0289] After determining the target score for each text in the knowledge base, all target scores can be sorted, and the top_k texts with the highest target scores can be selected as target candidate texts. This can be expressed as the following formula: res = top_k(score) result )

[0290] Where res represents the set of determined target candidate texts; top_k(score result This means selecting the k texts with the highest scores from all target scores.

[0291] To further improve the accuracy of question answering, based on the above embodiments, in some embodiments of this application, determining the target candidate text matching the standard question in the knowledge base according to each similarity score and each relevance score includes: determining a first preset number of first candidate texts according to each similarity score; determining a second preset number of second candidate texts according to each relevance score; and performing deduplication processing on the first candidate texts and the second candidate texts to obtain the target candidate text.

[0292] When determining target candidate texts that match the standard question in the knowledge base based on each similarity score and each relevance score, in some embodiments of this application, a first preset number of first candidate texts can be determined based on each similarity score. In some embodiments, based on each similarity score, the top-K texts with values ​​higher than a threshold θ are obtained as first candidate texts, where K is the first preset number.

[0293] Based on each relevance score, a second preset number of second candidate texts are determined. In some embodiments, the top-K texts with relevance scores above a threshold λ can be selected as second candidate texts, where K is the second preset number.

[0294] In some embodiments of this application, the first preset quantity and the second preset quantity may be the same or different.

[0295] After obtaining the first and second candidate texts, these texts can be identified as the target candidate texts.

[0296] However, since the first and second candidate texts often contain identical text, in some embodiments of this application, the first and second candidate texts can be deduplicated to obtain the target candidate text. To improve the efficiency of deduplication, in some embodiments of this application, when storing each text in the knowledge base, an identifier can be added to each text. This identifier is used to uniquely identify the corresponding text. During deduplication, texts with identical identifiers only need to be deduplicated; there is no need to recalculate the text similarity, thus improving the efficiency of deduplication.

[0297] In some embodiments, the target candidate text can be determined based on the following formula: res = set(topk(score(BM25)) + topk(score(Embedding)))

[0298] Where res represents the set of target candidate texts; topk(score(BM25)) represents the first candidate text; topk(score(Embedding)) represents the second candidate text; set() performs deduplication on the first and second candidate texts.

[0299] To further improve the accuracy of question and answer, based on the above embodiments, in some embodiments of this application, after searching for target candidate text matching the standard question in the knowledge base and before concatenating the target candidate text, the standard question, and the prompt text, the method further includes: searching for target question and target answer information matching the standard question in a pre-collected error accumulation knowledge base; determining the candidate answer information corresponding to the target question and the target answer information in the error accumulation knowledge base as third candidate text; performing deduplication processing on the target candidate text and the third candidate text to obtain a fourth candidate text, and updating the target candidate text using the fourth candidate text.

[0300] Research has found that for some specific questions, even vector retrieval and keyword retrieval methods may fail to find matching candidate texts. Therefore, in some embodiments of this application, an error accumulation knowledge base can be maintained in advance, which stores frequently incorrect question-answer pairs. After identifying the target candidate text, the target question and target answer information that match the standard question can be searched in the error accumulation knowledge base. That is, the standard question is compared with each question and answer in the error accumulation knowledge base to find matching target question and target answer information. In some embodiments of this application, matching can be based on vector retrieval, keyword retrieval, or a combination of both.

[0301] Since the purpose of some embodiments of this application is to find candidate texts related to the standard question that can be used for reference, and the useful knowledge included in the question is usually limited, in some embodiments of this application, after determining the target question, candidate answer information corresponding to the target question can be obtained from the error accumulation knowledge base. The content included in the candidate answer information may be questions related to the standard question. In some embodiments of this application, the candidate answer information, along with the target answer information found in the error accumulation knowledge base, can be determined as the third candidate text.

[0302] After determining the third candidate text, it can also be designated as the target candidate text. However, since there may be duplicate text between the existing target candidate text and the third candidate text, in some embodiments of this application, the target candidate text and the third candidate text can be deduplicated to obtain a fourth candidate text, and the target candidate text can be updated using the fourth candidate text.

[0303] To further improve the accuracy of question answering, based on the above embodiments, in some embodiments of this application, if there are multiple target candidate texts, after searching for target candidate texts that match the standard question in the knowledge base, and before concatenating the target candidate texts, the standard question, and the prompt text, the method further includes: determining a similarity score between the standard question and each target candidate text; and filtering the target candidate texts based on the similarity score.

[0304] When multiple target candidate texts are identified, these texts may be determined based on different retrieval methods, and the retrieval scoring criteria for different target candidate texts may differ. Therefore, it is not possible to directly combine and rank all target candidate texts based on the similarity score at the time of retrieval. In some embodiments of this application, it is necessary to combine and re-rank all the obtained target candidate texts based on a new ranking method, and to perform another round of screening during the re-ranking process to obtain higher-quality target candidate texts. In other words, after determining the target candidate texts, a fine-grained screening is performed again before determining the input data. This method can reduce the number of characters in the input data, thereby improving the accuracy of the sixth model's response.

[0305] When performing refined screening, the similarity score between the standard question and each target candidate text can be determined, and the target candidate text can be screened based on each similarity score.

[0306] In one possible implementation, a score can be calculated based on the re-ranking model to determine whether the standard question matches the target candidate text. All target candidate texts are then ranked, and those ranked lower are deleted, retaining only the top K target candidate texts. Optionally, the re-ranking model can be any one or any combination of BAAI General Embedding (BGE) Re-Ranker, BGE Re-Ranker v2-M3, BGE Re-Ranker v2-Gemma-2B, etc., and can be configured as needed by those skilled in the art.

[0307] In one possible implementation, the similarity between the standard question and each target candidate text can be determined based on a pre-trained classification model, thereby ranking the target candidate texts according to the degree of similarity. In some embodiments of this application, semantic features of each text can be extracted based on the BERT model, thereby predicting text relevance based on the extracted semantic features. Figure 14 is a schematic diagram of a classification model structure provided by some embodiments of this application. As shown in Figure 14, when training the classification model, the standard question and any target candidate text can be used as inputs to the classification model. In Figure 14, w1 and w2 represent the first text, such as the word feature vector corresponding to the standard question, and w3, w4, and w5 represent the second text, such as the word feature vector corresponding to the target candidate text. "[CLS]" and "[SEP]" are used to identify and distinguish different texts. In the BERT model, CLS is an abbreviation for Classification, and "[CLS]" is a special marker used for text classification tasks. SEP stands for separator, and "[SEP]" is used to segment different sentences or documents. Figure 15 is a schematic diagram of word vector encoding of input data for a classification model provided in some embodiments of this application. As shown in Figure 15, the input data of the classification model includes position encoding, context encoding, and character encoding. As shown in Figure 15, the character encoding corresponds to the character encoding of two sentences, namely sentence 1 and sentence 2. The subsequent classification model can then determine whether sentence 1 and sentence 2 are similar. In order for the classification model to know which character encodings belong to the same sentence, the input data also includes context encoding. As shown in Figure 15, the context encoding corresponding to the characters belonging to sentence 1 is 1, and the context encoding corresponding to the characters belonging to sentence 2 is 2.

[0308] The above input data is fed into the BERT model to obtain the full-text semantic features output by the BERT model. These full-text semantic features are then fed into a linear layer, which performs a binary classification judgment of "similar" or "dissimilar" based on the full-text semantic features. To further determine the similarity between two corresponding texts, a Sigmoid activation function is applied after the linear layer to map the similarity between the two texts, with the output result ranging from 0 to 1.

[0309] In some embodiments, the loss value can be calculated based on the following formula:

[0310] Where NLL(x,y) and J CE represents the loss value; yi represents the label value, where yi = 1 when the two sentences are similar and yi = 0 when they are dissimilar; This represents the predicted value of the classification model when two sentences are determined to be similar. When dissimilar

[0311] The parameters of the classification model can then be adjusted based on the calculated loss value.

[0312] The process of determining target candidate text is described below with reference to one or more embodiments. Figure 16 is a schematic diagram of a target candidate text determination process provided by some embodiments of this application. As shown in Figure 16, when determining the target candidate text corresponding to the standard question, a candidate text set 1 matching the standard question can be queried in the knowledge base using a hybrid retrieval method (S1601), and a candidate text set 2 matching the standard question can be queried in the error accumulation knowledge base using a hybrid retrieval method (S1602). After obtaining candidate text set 1 and candidate text set 2, the texts included in the sets can be deduplicated (S1603), and the remaining texts can be further filtered (S1604) to obtain K high-quality texts as target candidate texts (S1605).

[0313] The process of determining the response information is described below with reference to one or more other embodiments. Figure 17 is a schematic diagram of a response information determination process provided by some embodiments of this application. As shown in Figure 17, after determining the target candidate text (S1701), the text can be filtered according to the similarity score between the standard question and each target candidate text (S1702) to obtain the K target candidate texts with the highest similarity scores (S1703). The K target candidate texts, the standard question, and the designed prompt text are input into the LLM (S1704) to obtain the response information (S1705).

[0314] To further improve the accuracy of question and answer, based on the above embodiments, in some embodiments of this application, the method further includes: determining whether the response information meets the quality requirements based on the seventh model; if yes, determining that the response information has passed the verification and outputting it; if no, determining that the response information has failed the verification and performing manual evaluation; the quality requirements include at least one of the following: the standard question matches the response information, the response information comes from the target candidate text, and the wording of the response information is appropriate.

[0315] To further improve the accuracy of question-and-answer sessions, the quality of the received responses can be verified. In some embodiments of this application, the seventh model can be used to determine whether the received responses meet quality requirements. If so, the responses are deemed to have passed verification and are output; otherwise, they are deemed to have failed verification and require manual evaluation. In some embodiments of this application, quality requirements may include one or more of the following: matching the standard question with the response; the response originating from the target candidate text; and appropriate wording in the response. That is, the seventh model is used to determine whether the user's standard question matches the response; the seventh model is used to determine whether the response output by the sixth model originates from the target candidate text; and the seventh model is used to determine whether the wording of the response output by the sixth model is reasonable, such as whether the response is harmless and positive, or whether the response is fluent.

[0316] In some embodiments, response information can be obtained and analyzed using asynchronous or multi-threaded calls. If all large models deem the verification passed, manual evaluation is no longer required; otherwise, manual evaluation will proceed. Figure 18 is a schematic diagram illustrating the principle of response information quality evaluation provided in some embodiments of this application. As shown in Figure 18, after obtaining the response information (Out), the quality of the response information can be evaluated based on three large models, such as Generative Pre-Trained Transformer (GPT)1, GPT2, and GPT3. In some embodiments, the response information and target candidate text can be input into GPT1 to determine whether the response information originates from the target candidate text. In other embodiments, the response information and a standard question can be input into GPT2 to determine whether the response information matches the standard question. In still other embodiments, the response information can be input into GPT3 to determine whether the wording in the response information is appropriate. As shown in Figure 18, assuming the verification results of GPT1, GPT2, and GPT3 are all 1, indicating that the verification passed, the response information can be directly output.

[0317] Figure 19 is a schematic diagram illustrating the principle of response information quality evaluation provided in some embodiments of this application. As shown in Figure 19, the quality of the response information Out is evaluated based on three GPT models. The input data of the three GPT models are consistent with those shown in Figure 18, and will not be repeated in this embodiment. As shown in Figure 19, assuming that the verification results of GPT1 and GPT3 are both 1, indicating that the verification passed, while the verification result of GPT2 is 0, indicating that the verification failed, then it can be determined that the final verification result of the response information is failed, and manual intervention is required for evaluation.

[0318] The following describes the question-answering process based on a large model in conjunction with one or more embodiments. Figure 20 is a schematic diagram of a question-answering process based on a large model provided by some embodiments of this application. As shown in Figure 20, the process may include pre-processing, intent classification, hybrid retrieval and post-processing.

[0319] After obtaining the user's question (S2001), that is, after obtaining the user's question to be answered, pre-processing can be performed (S2002). In the pre-processing stage, some or all of the processing of the question to be answered, such as substitution resolution, decomposition of subqueries, and query rewriting, can be performed based on the large model to obtain the standard question.

[0320] After obtaining the standard question, the intention of the standard question can be classified using a classification model (S2003) to obtain the target intention.

[0321] After obtaining the target intent, a hybrid search is performed (S2004). During the hybrid search process, some or all of the vector search and keyword search can be performed in the target knowledge base corresponding to the target intent to obtain the target candidate text.

[0322] After obtaining the target candidate texts, these target candidate texts are rearranged, i.e., post-processing (S2005), so as to perform secondary screening according to the rearranged order to obtain the remaining target candidate texts.

[0323] After the final target candidate text is determined, the corresponding response information can be generated based on the large model (S2006).

[0324] After receiving the response, we can use the seventh model to determine whether the response meets the quality requirements (S2007).

[0325] Some embodiments of this application employ various methods such as question rewriting, multi-path recall, and hybrid retrieval to improve the accuracy of the RAG model, which can bring a 10% improvement in the accuracy of query results.

[0326] Based on the above embodiments, still referring to Figure 45, the processor 4501 included in the electronic device is further configured to execute a computer program stored in the memory 4503 to cause the electronic device to perform: standardizing and rewriting the question to be answered based on a fifth model pre-trained for the target domain to obtain a standard question corresponding to the question to be answered; searching for target candidate text matching the standard question in the knowledge base; concatenating the target candidate text, the standard question, and the prompt text to obtain input data, and inputting the input data into a sixth model to obtain response information, wherein the response information is text obtained by the sixth model after analyzing the target candidate text and the standard question according to the requirements of the prompt text.

[0327] In one possible implementation, after obtaining the standard question corresponding to the question to be answered, and before searching for target candidate text matching the standard question in the knowledge base, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform: determining the target intent corresponding to the standard question based on a classification model; when the processor 4501 performs the step of searching for target candidate text matching the standard question in the knowledge base, it is specifically configured to execute the computer program to cause the electronic device to perform: searching for target candidate text matching the standard question in the target knowledge base corresponding to the target intent.

[0328] In one possible implementation, the classification model includes a feature extraction layer and a linear layer. The processor 4501 executes the computer program to enable the electronic device to perform the training process of the classification model, which includes: acquiring word vector codes and labels corresponding to sample text, wherein the labels are used to identify the standard intent of the sample text, and the word vector codes include part-of-speech identifiers for each character; inputting the word vector codes and the labels into the feature extraction layer, wherein the feature extraction layer determines the focus of feature extraction based on the part-of-speech identifiers when extracting semantic features to obtain predicted semantic features; inputting the predicted semantic features into the linear layer for intent prediction to obtain the prediction probability corresponding to each preset intent; determining the target prediction probability corresponding to the standard intent based on the prediction probability corresponding to each preset intent; determining a first loss value based on the target prediction probability and preset control parameters, and adjusting the parameters of the classification model based on the first loss value.

[0329] In one possible implementation, the processor 4501 is specifically configured to execute the computer program to cause the electronic device to perform: determining the similarity between the standard question and each text based on a first feature vector of the standard question and a second feature vector of each text in the knowledge base; determining a relevance score between the standard question and each text in the knowledge base based on a keyword retrieval algorithm; and determining target candidate texts in the knowledge base that match the standard question based on each similarity and each relevance score.

[0330] In one possible implementation, the processor 4501 is specifically configured to execute the computer program to cause the electronic device to perform: for each text in the knowledge base, determining a target score corresponding to the text based on the similarity, relevance score and preset weight of the text; and determining target candidate texts based on each target score.

[0331] In one possible implementation, the processor 4501 is specifically configured to execute the computer program to cause the electronic device to perform: determining a first preset number of first candidate texts based on each similarity score; determining a second preset number of second candidate texts based on each relevance score; and performing deduplication processing on the first candidate texts and the second candidate texts to obtain the target candidate texts.

[0332] In one possible implementation, after searching for target candidate text matching the standard question in the knowledge base, and before concatenating the target candidate text, the standard question, and the prompt text, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform: searching for target question and target answer information matching the standard question in a pre-collected error accumulation knowledge base; determining the candidate answer information corresponding to the target question in the error accumulation knowledge base, as well as the target answer information, as third candidate text; performing deduplication processing on the target candidate text and the third candidate text to obtain a fourth candidate text, and updating the target candidate text using the fourth candidate text.

[0333] In one possible implementation, if there are multiple target candidate texts, after searching for target candidate texts that match the standard question in the knowledge base, and before concatenating the target candidate texts, the standard question, and the prompt text, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform: determining a similarity score between the standard question and each of the target candidate texts; and filtering the target candidate texts based on the similarity score.

[0334] In one possible implementation, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform: determining whether the response information meets the quality requirements based on the seventh model; if yes, determining that the response information has passed the verification and outputting it; if no, determining that the response information has failed the verification and performing manual evaluation; the quality requirements include at least one of the following: the standard question matches the response information, the response information comes from the target candidate text, and the wording of the response information is appropriate.

[0335] Some embodiments of this application also provide another question-answering method based on a large model for training an embedding model. In some embodiments of this application, when training the embedding model, sample text from a specific domain is input into the encoder of the initial embedding model for encoding. The encoder includes multiple encoding sub-layers. During encoding, the first feature vector output by each encoding sub-layer in the encoder is obtained, and the predicted feature vector of the sample text is determined based on the weights corresponding to each encoding sub-layer and the first feature vector. Since the predicted feature vector is determined by fusing the first feature vectors of different encoding sub-layers, the predicted feature vector incorporates semantic features from different levels. Compared to related technologies, where Retrieval Enhancement Generation (RAG) models often use open-source embedding models, which rely on the size and quality of the training corpus, this method improves encoding accuracy. A first loss value is determined based on the predicted feature vector and the standard feature vector of the sample text. The embedding model obtained by training the initial embedding model based on this first loss value has higher accuracy during encoding. Figure 21 is a flowchart illustrating the training process of another embedding model based on a large model provided in some embodiments of this application. As shown in Figure 21, the process includes the following steps:

[0336] S2101: Obtain a training set that has been pre-saved for a specific domain, the training set including sample text.

[0337] To obtain an embedding model with higher accuracy, in some embodiments of this application, a training set is pre-configured, and the initial embedding model is trained using the sample texts included in the training set to obtain the embedding model. To ensure that the trained embedding model has better encoding capabilities in a specific domain, in some embodiments of this application, the pre-configured training set may be pre-configured for that specific domain. That is, the sample texts stored in the training set all belong to the same domain. In some embodiments, the specific domain may be education, transportation, mobile phones, computers, etc., and those skilled in the art can configure it as needed; this application does not impose any limitations on this.

[0338] S2102: Input the sample text into the encoder for encoding, obtain the first feature vector output by each encoding sub-layer in the encoder, and determine the predicted feature vector of the sample text based on the weights corresponding to each encoding sub-layer and the first feature vector.

[0339] To train the initial embedding model, in some embodiments of this application, after obtaining the training set, the sample text can be input into an encoder for encoding. The encoder generates feature vectors from the sample text. In some embodiments, the encoder can be a BERT (Bidirectional Encoder Representations from Transformers) model. During encoding, the encoder can apply a random mask to the sample text, such as a 30% mask, and then predict the masked portion to obtain the vector representation of the sample text.

[0340] In some embodiments of this application, the encoder may include multiple encoding sub-layers, each of which can be regarded as an encoder. For example, the encoder of the embedded model consists of 12 layers of Transformer Encoders.

[0341] The feature vectors obtained by conventional BERT models are often 768-dimensional feature vectors determined by CLS (Classification Encoding) or by averaging the encoding of each character. However, analysis shows that in the field of image feature extraction, the feature information obtained by fusing deep and shallow layers of a neural network is more conducive to improving the model's detection accuracy, as it can better focus on shallow image information, such as lines and shapes. Similarly, in the field of text semantic feature extraction, the understanding of each layer of text in the model should also be different. Therefore, in some embodiments of this application, during the encoding process, the first feature vector output by each encoding sub-layer in the encoder can be obtained.

[0342] For a model, the deeper the model, the stronger the semantic information it extracts, and the more important it is for the semantic feature representation of text. However, shallower networks may focus more on detailed information such as the part-of-speech tag of each word. Therefore, while the design should focus more on deep features, it should also combine shallow detailed information, especially for text feature representation. Based on this, after obtaining the first feature vector output by each encoding sub-layer, the predicted feature vector of the sample text can be determined according to the weights corresponding to each sub-layer and the first feature vector. In some embodiments of this application, the weights corresponding to each encoding sub-layer can be the same or different.

[0343] Figure 22 is a schematic diagram of an encoder structure provided in some embodiments of this application. As shown in Figure 22, the encoder may include six encoding sub-layers. Each DECODER_2 shown on the left side of Figure 22 (left and right in the illustration) represents each encoding sub-layer. First, the sample text (Input) is input into the first encoding sub-layer to obtain the first feature vector output by the first encoding sub-layer. Then, this feature vector is input into the second encoding sub-layer to obtain the first feature vector output by the second encoding sub-layer. This first feature vector is then input into the third encoding sub-layer to obtain the first feature vector output by the third encoding sub-layer, and so on, until the first feature vector output by the last encoding sub-layer is obtained. The graphic composed of five circles in Figure 22 represents the first feature vector. After obtaining each first feature vector, the predicted feature vector of the sample text can be determined based on the weights corresponding to each encoding sub-layer and the first feature vectors. The graphic composed of circles shown on the right side of Figure 22 (left and right in the illustration) represents the predicted feature vector. In some embodiments of this application, shallower encoding sub-layers have smaller weights, while deeper encoding sub-layers have larger weights. As shown in Figure 22, the weights corresponding to the first, second, third, and fourth encoding sub-layers are 0.00001, 0.0001, 0.001, 0.01, 0.1, and 1, respectively. For ease of understanding, the predicted feature vector of the sample text can be determined based on the following formula: x = r n-2 x1+r n-3 x2 + ... + rx n-1 +x n

[0344] Where x represents the predicted feature vector, n represents the number of coding sub-layers included in the encoder, x1 represents the first coding sub-layer, x2 represents the second coding sub-layer, and x... n This indicates the nth coding sublayer, and r represents the weight coefficient. Those skilled in the art can set this value as needed, for example, r = 0.1.

[0345] S2103: Determine a first loss value based on the predicted feature vector and the standard feature vector of the sample text, and train the initial embedding model based on the first loss value to obtain the embedding model.

[0346] After determining the predicted feature vector of the sample text, a first loss value can be determined to train the initial embedding model based on this first loss value. The first loss value can be determined based on the predicted feature vector and the standard feature vector of the sample text. To facilitate model understanding, when inputting the sample text into the encoder, the sample text can undergo character encoding conversion. Figure 23 is a schematic diagram of a model input provided by some embodiments of this application. As shown in Figure 23, the model input can include three parts: character encoding, sentence encoding, and position encoding. Character encoding is determined based on each character in the sample text; sentence encoding is used to identify whether corresponding characters belong to the same sentence; and position encoding is used to identify the position of the corresponding character in the sample text. The feature vector obtained from the encoding conversion can be used as the standard feature vector. Then, the standard feature vector is masked, and the encoder processes the masked standard feature vector to obtain the predicted feature vector. The first loss value is then determined based on the deviation between the predicted feature vector and the standard feature vector. It should be noted that how to determine the predicted feature vector and standard feature vector of the sample text, and how to determine the first loss value, are known techniques and will not be elaborated upon here.

[0347] After determining the first loss value, an initial embedding model can be trained based on the first loss value to obtain the embedding model, which can be used to determine any text feature vector.

[0348] In some embodiments of this application, when training the embedding model, sample text from a specific domain is input into the encoder of the initial embedding model for encoding. The encoder includes multiple encoding sub-layers. When the encoder encodes, the first feature vector output by each encoding sub-layer is obtained, and the predicted feature vector of the sample text is determined based on the weights corresponding to each encoding sub-layer and the first feature vector. Since the predicted feature vector is determined by fusing the first feature vectors of different encoding sub-layers, the predicted feature vector incorporates semantic features from different levels, thus improving the encoding accuracy. A first loss value is determined based on the predicted feature vector and the standard feature vector of the sample text. The embedding model obtained by training the initial embedding model based on the first loss value has higher accuracy during encoding.

[0349] To further improve the accuracy of the trained embedding model, based on the above embodiments, in some embodiments of this application, the initial embedding model further includes a decoder. After determining the predicted feature vector of the sample text and before training the initial embedding model based on the first loss value, the method further includes: inputting the predicted feature vector into the decoder for decoding to obtain the predicted text output by the decoder; and determining a second loss value based on the predicted text and the sample text. Training the initial embedding model based on the first loss value to obtain the embedding model includes: training the initial embedding model based on the first loss value and the second loss value to obtain the embedding model.

[0350] To further improve the accuracy of the trained embedding model, the initial embedding model can also include a decoder during training. This decoder is responsible for reconstructing the original sample text using vectors. This decoder consists of only one Transformer layer. In the decoder part, the sentence is further annoyed, with a mask ratio of up to 70%, increasing the complexity of the task by increasing the mask ratio. The decoder structure is simple, with only a single Transformer layer, and the high mask ratio of the input sentence makes decoding challenging, forcing the encoder to generate high-quality predicted feature vectors.

[0351] In some embodiments of this application, the predicted feature vector determined by the decoder can be input into the decoder for decoding, thereby obtaining the predicted text output by the decoder. A second loss value is determined based on the predicted text and the sample text. After determining the second loss value, an initial embedding model can be trained based on the sum of the first and second loss values, thereby obtaining the embedding model.

[0352] In some embodiments, the sum of the first loss value and the second loss value can be determined as the target loss value, and the initial embedding model can be trained based on the target loss value. Of course, the product, quotient, etc., of the first loss value and the second loss value can also be determined as the target loss value, and those skilled in the art can set it as needed.

[0353] In some embodiments of this application, the training task of the entire model is as follows: in the Encoder stage, the masked portion is reconstructed, which is Masked Language Modeling (MLM); in the Decoder stage, the entire sample text is reconstructed. The overall loss function is: Loss = L encode +L decode

[0354] Among them, Loss represents the target loss value, and the initial embedding model can be trained based on this target loss value; L encode represents the first loss value, that is, the loss value of the encoder; L decode represents the second loss value, that is, the loss value of the decoder.

[0355] Some embodiments of this application are improved based on the RetroMAE structure. The starting point of the improvement is to enrich the vector information of the text more. The hidden state vector of the CLS token as the semantic representation can also be enriched based on different layers, taking into account the encoding information of each layer and the token information of the last layer at the same time. Among them, the RetroMAE structure is a new retrieval-oriented pre-training paradigm based on the Masked Auto-Encoder (MAE). The training process of the embedding model will be described below in conjunction with one or more embodiments. FIG. 24 is a schematic structural diagram of an embedding model provided by some embodiments of this application. As shown in FIG. 24, this embedding model includes two parts. The left side (left and right in the figure) in FIG. 24 is the encoder, and the right side (left and right in the figure) is the decoder. The encoder is used to generate the predicted feature vector, and the decoder is responsible for using the vector to restore the original sample text. In some embodiments of this application, assume that the sample text is "I am in Tiananmen, Beijing". In the Encoder stage, 30% of the sample text is masked, and the predicted feature vector is obtained through the Encoder. This predicted feature vector is determined according to the word encoding (token) of each word output by the Encoder and the first feature vector output by each encoding sublayer. That is to say, this predicted feature vector is determined according to the deeply fused CLS and word token vectors. After obtaining the predicted feature vector, this predicted feature vector is input into the Decoder part. In the Decoder part, the sample text is further increased in noise, and the masking ratio is 70%. By increasing the masking ratio, the complexity of the task is improved. As shown in FIG. 24, the masked sample text is [M][M]North[M]Tianan[M].

[0356] The training process of the embedding model will be described below in conjunction with another or more embodiments. The storage format of the training set can be unlabeled text segments, such as:

[0357] [{

[0358] "text": "segment"

[0359] }]

[0360] During the training phase, the model can be trained based on a stored text dataset, enabling the resulting embedding model to possess powerful representation capabilities. Figure 25 is a schematic diagram of the training of an embedding model provided by some embodiments of this application. As shown in Figure 25, the text in the training set is input into the initial embedding model to obtain the model's output data. The structure of this model can be the structure shown in Figure 24.

[0361] To further improve the accuracy of the embedding model encoding, based on the above embodiments, in some embodiments of this application, the method further includes: obtaining any sample text in the training set as a first fine-tuned text, a first similar text corresponding to the first fine-tuned text, and a first dissimilar text corresponding to the first fine-tuned text; inputting the first fine-tuned text, the first similar text, and the first dissimilar text into the embedding model respectively to obtain each second feature vector; determining a third loss value based on a fourth similarity between the second feature vectors corresponding to the first fine-tuned text and the first similar text, and a fifth similarity between the second feature vectors corresponding to the first fine-tuned text and the first dissimilar text; and fine-tuning the embedding model based on the third loss value.

[0362] To further improve the accuracy of the embedding model encoding, in some embodiments of this application, the trained embedding model can be fine-tuned. During fine-tuning, any sample text in the training set can be used as the first fine-tuned text, the first similar text corresponding to the first fine-tuned text, and the first dissimilar text corresponding to the first fine-tuned text. That is, to enhance the generalization of the model, some similar texts exist in the text stored in the training set. Any fine-tuned text can have one or more similar texts, and similarly, dissimilar texts can have one or more. The similar texts can be understood as positive samples of the fine-tuned text, and the dissimilar texts can be understood as negative samples of the fine-tuned text.

[0363] Subsequent training can be performed based on the first fine-tuned text, the first similar text, and the first dissimilar text. During this comparative learning process, a larger batch size means more negative samples are encountered within a batch, leading to more accurate model learning. Therefore, the batch size can be increased during this training phase. Batch size is a crucial parameter that determines the amount of data used in each training iteration.

[0364] The first fine-tuned text, the first similar text, and the first dissimilar text are respectively input into the embedding model to obtain each second feature vector. In some embodiments of this application, the embedding model undergoing fine-tuning training only includes the encoder. Therefore, the obtained second feature vector is a feature vector determined comprehensively based on the weights corresponding to each encoding sublayer and the output feature vector.

[0365] After obtaining each second feature vector, a fourth similarity can be determined between the second feature vectors corresponding to the first fine-tuned text and the first similar text; that is, the fourth similarity between the first fine-tuned sample and the positive sample. Simultaneously, a fifth similarity is determined between the second feature vectors corresponding to the first fine-tuned sample and the first dissimilar text; that is, the fifth similarity between the first fine-tuned sample and the negative sample. Having determined the fourth and fifth similarities, a third loss value can be determined based on these values, and then the obtained embedding model can be fine-tuned and trained based on this third loss value.

[0366] In some embodiments, the third loss value can be calculated based on the following formula:

[0367] Where p represents the first fine-tuned text; q represents the first similar text; q' represents the first dissimilar text; and T represents the temperature. This represents the similarity between the second feature vector corresponding to the first fine-tuned text and the first similar text; denoted by , represents the similarity between the second feature vectors corresponding to the first fine-tuned text and the first dissimilar text; min·· represents the desired third loss value to be as small as possible; Q' represents the set of text pairs formed by the first fine-tuned text and each of the first dissimilar texts.

[0368] As shown in the formula above, the numerator represents the similarity between the first fine-tuned text and the positive sample, while the denominator represents the similarity between the first fine-tuned text and the negative sample. Therefore, it can be seen that the greater the similarity within the same category and the smaller the similarity between different categories, the smaller the loss value will be. Thus, the goal of the above formula is to make the numerator as large as possible and the denominator as small as possible.

[0369] The construction of positive and negative sample pairs will be explained below with reference to Figure 26. Figure 26 is a similarity diagram provided by some embodiments of this application. As shown in Figure 26, T1, T2, T3, ..., T N I1, I2, I3, ..., I NThe vectors I1 and T1 represent the second feature vectors obtained after different texts have been processed by the embedding model. I1T1 represents the similarity between the second feature vectors I1 and T1, I2T1 represents the similarity between the second feature vectors I2 and T1, and so on. The similarity can be cosine similarity. In Figure 26, the cosine similarity along the main diagonal is maximized, while the cosine similarity elsewhere is minimized. The diagonal lines represent two matching sentence pairs, forming positive sample pairs. For example, the first fine-tuned text and the first similar text can be considered a positive sample pair. The off-diagonal lines form negative sample pairs, representing two dissimilar sentence pairs. For example, the first fine-tuned text and the first dissimilar text can be considered a negative sample pair.

[0370] The fine-tuning training process of the embedding model is described below with reference to one or more embodiments. Figure 27 is a schematic diagram of a fine-tuning training architecture provided by some embodiments of this application. As shown in Figure 27, in order to improve the efficiency of model training during the fine-tuning training process, two embedding models can be used to process the text. The two models shown in Figure 27 can have the same structure and parameters. The model on the left (left and right in the illustration) in Figure 27 is used to determine the second feature vector of the Text, which can be understood as the first fine-tuning sample described in the above embodiments. The model on the right (left and right in the illustration) in Figure 27 is used to determine the second feature vector of the Page, which can be understood as the first similar text and the first dissimilar text in the above embodiments. After obtaining the second feature vector output by each model, comparative learning can be performed based on each obtained second feature vector.

[0371] To further improve the accuracy of the embedding model encoding, based on the above embodiments, in some embodiments of this application, the method further includes: obtaining any sample in the training set as the second fine-tuned text, the second similar text corresponding to the second fine-tuned sample, and the second dissimilar text corresponding to all the second fine-tuned texts; inputting the second fine-tuned text, the second similar text, and the second dissimilar text into the embedding model respectively to obtain each third feature vector; determining the sixth similarity between the third feature vector of the second fine-tuned sample and the second similar text, and the seventh similarity between the third feature vector of the second fine-tuned sample and the second dissimilar text; determining a fourth loss value based on the difference between the sixth similarity and the seventh similarity, and the activation function; and fine-tuning the embedding model based on the fourth loss value.

[0372] To further improve the accuracy of the embedding model encoding, in some embodiments of this application, the trained embedding model can be fine-tuned for a specific task. During the fine-tuning process, any sample text in the training set can be obtained as the second fine-tuned text, the second similar text corresponding to the second fine-tuned text, and the second dissimilar text corresponding to the second fine-tuned text. Subsequently, the obtained embedding model can be trained based on the obtained second fine-tuned text, second similar text, and second dissimilar text. In some embodiments of this application, the trained embedding model can be a pre-trained model or a model obtained by performing general fine-tuning training on a pre-trained model.

[0373] During model training, the second fine-tuned text, the second similar text, and the second dissimilar text can be input into the embedding model to obtain each third feature vector. In some embodiments of this application, the embedding model undergoing fine-tuning training only includes the encoder. Therefore, the obtained third feature vector is a feature vector determined by combining the weights and output feature vectors of each encoding sublayer.

[0374] After obtaining each third feature vector, the sixth similarity between the third feature vectors corresponding to the second fine-tuned text and the second similar text can be determined, that is, the sixth similarity between the second fine-tuned sample and the positive sample can be determined. At the same time, the seventh similarity between the third feature vectors corresponding to the second fine-tuned sample and the second dissimilar text can be determined, that is, the seventh similarity between the second fine-tuned sample and the negative sample can be determined.

[0375] After determining the sixth and seventh similarities, the fourth loss value can be determined by the difference between the sixth and seventh similarities and the activation function. Based on this fourth loss value, the obtained embedding model can be fine-tuned.

[0376] In some embodiments, the fourth loss value can be calculated based on the following formulas: sim_pos = cosine(text, positive) sim_neg = cosine(text, negative_text) sim_diff = sigmoid(sim_pos - neg_pos) loss = log(sim_diff)

[0377] Where sim_pos represents the sixth similarity; text represents the third feature vector of the second fine-tuned text; positive represents the third feature vector of the second similar text; cosine(text,positive) represents the cosine similarity between the third feature vector of the second fine-tuned text and the third feature vector of the second similar text; sim_neg represents the seventh similarity; negative_text represents the third feature vector of the second dissimilar text; cosine(text,negative_text) represents the cosine similarity between the third feature vector of the second fine-tuned sample and the third feature vector of the second dissimilar text; sigmoid() represents the activation function; sim_diff represents the mapped value; and loss represents the fourth loss value.

[0378] The loss functions provided in some embodiments of this application can make the model converge more easily and improve the model performance more significantly.

[0379] In addition, since positive and negative samples between texts can be determined in this application, the training of this part belongs to labeled training, which can be regarded as a common classification task. In some embodiments of this application, cross-entropy loss can also be used for training.

[0380] In some embodiments of this application, a sixth or seventh similarity score of the third feature vector between the second fine-tuned sample and the second similar or dissimilar text can be determined. The probability values ​​corresponding to the "similar" category and the "dissimilar" category are then determined based on this sixth or seventh similarity score. Since the similarity of the corresponding texts is known in advance, that is, the standard classification of the corresponding texts is known, a fourth loss value can be determined based on the probability values ​​corresponding to the standard classification in some embodiments of this application. In some embodiments, the fourth loss value can be calculated with reference to the following formula: L model =-p t log(p t )

[0381] Among them, L model p represents the fourth loss value. t p represents the probability value corresponding to the standard classification. t The calculation method is as described in the formula above, and will not be repeated here.

[0382] The fine-tuning training process of the embedding model is described below with reference to one or more embodiments. Figure 28 is a schematic diagram of a fine-tuning training architecture provided by some embodiments of this application. As shown in Figure 28, in order to improve the efficiency of model training during the fine-tuning training process, two embedding models can be used to process the text. The two models shown in Figure 28 can have the same structure and parameters. The model on the left (left and right in the illustration) in Figure 28 is used to determine the third feature vector of the text, which can be understood as the second fine-tuning sample described in the above embodiments. The model on the right (left and right in the illustration) in Figure 28 is used to determine the second feature vectors of Pos and Neg, where Pos can be understood as the second similar text in the above embodiments, and Neg can be understood as the second dissimilar text in the above embodiments. After obtaining the third feature vector output by each model, the fourth loss value can be calculated based on each obtained third feature vector and the calculation formula of the fourth loss value provided in the above embodiments, thereby performing fine-tuning training based on the fourth loss value.

[0383] To further improve the accuracy of the embedding model encoding, based on the above embodiments, in some embodiments of this application, the process of determining similar texts corresponding to any fine-tuned text includes: obtaining similar text pairs generated based on a large model, and determining similar text pairs containing the fine-tuned text as target text pairs; determining the fourth feature vector of each text in the target text pair based on other pre-trained embedding models; and determining whether the non-fine-tuned text in the target text pair is a similar text to the fine-tuned text based on the eighth similarity between each fourth feature vector and a preset threshold.

[0384] In some embodiments of this application, when determining similar texts corresponding to any fine-tuned text, similar text pairs generated based on a large model can be obtained, and the similar text pairs containing the fine-tuned text can be determined as target text pairs. The non-fine-tuned texts included in the target text pairs are all texts related to the fine-tuned text.

[0385] After identifying the target text pair, the fourth feature vector of each text in the target text pair can be determined based on other pre-trained embedding models. Then, based on the eighth similarity between each fourth feature vector and a preset threshold, it can be determined whether the non-fine-tuned text in the target text pair is similar to the fine-tuned text. In some embodiments of this application, the fourth feature vector of each text can be determined multiple times based on multiple other embedding models, thereby performing multiple calculations of the eighth similarity, and thus determining multiple times whether the non-fine-tuned text in the target text pair is similar to the fine-tuned text. In other words, multiple judgments are made using multiple other embedding models.

[0386] In some embodiments, when the fine-tuned text is Question 1, the determined target text pair can be Question 1-Summary, Question 1-Answer, Question 1-Fragment, etc. The fourth feature vector of each text in the target text pair can be determined based on M3E (Moka Massive Mixed Embedding), BGE, or Text2Vec models. Then, the eighth similarity between the corresponding fourth feature vectors in each target text pair is calculated. Based on the eighth similarity and a preset threshold, it is determined whether the non-fine-tuned text in the target text pair is similar to the fine-tuned text. The preset threshold can be any decimal, such as 0.7.

[0387] Figure 29 is a schematic diagram illustrating the principle of similar text determination provided in some embodiments of this application. As shown in Figure 29, for any text pair, the fourth feature vector of each text included in the text pair is determined based on three other embedding models: M3E, BGE, and Text2Vec. After determining the fourth feature vector, hard voting is performed. In the hard voting stage, for each other embedding model, the eighth similarity between the fourth feature vectors output by that other embedding model can be determined, and it can be determined whether the eighth similarity is greater than a preset threshold. If so, the texts in the text pair are determined to be similar texts.

[0388] When any two of the three other models determine that the text in the corresponding text pair is similar to each other, then it can be determined that the text in that text is indeed similar.

[0389] To further improve the accuracy of the embedding model encoding, based on the above embodiments, in some embodiments of this application, after determining the similar text pairs containing the fine-tuned text as target text pairs, and before determining each fourth feature vector of the target text pair based on other pre-trained embedding models, the method further includes: determining whether the texts in the target text pair are similar based on the eighth model; if so, continuing to execute the subsequent step of determining each fourth feature vector of the target text pair based on other pre-trained embedding models.

[0390] As the model training progresses towards the final stage, the quality requirements for the training set data become increasingly stringent. Therefore, in some embodiments of this application, a more rigorous approach can be adopted to determine the sample text. After determining the target text pair, before determining each fourth feature vector, it is possible to determine whether the texts in the target text pair are similar based on the eighth model. If so, the subsequent steps of determining each fourth feature vector of the target text pair based on other pre-trained embedding models are then executed. In other words, for any target text pair, the eighth model is first used to determine whether the two are similar texts using a prompt text method. Only after the eighth model passes the verification is further determined based on other embedding models.

[0391] Figure 30 is a schematic diagram illustrating the principle of similar text determination provided in some embodiments of this application. In some embodiments of this application, text pairs may include question-summary; question-answer; question-fragment; summary-answer; summary-fragment; answer-fragment. For any text pair, a large-scale model judgment is performed based on three other embedding models, namely M3E, BGE, and Text2Vec models, and an eighth model, to determine the fourth feature vector of each text included in the text pair. After determining the fourth feature vector, hard voting is performed. In the hard voting stage, for each other embedding model, the eighth similarity between the fourth feature vectors output by that other embedding model can be determined, and it can be determined whether the eighth similarity is greater than a preset threshold. If so, the texts in the text pair are determined to be similar. At the same time, the result of the large-scale model judgment is obtained. If the verification results of the above four models all pass, it can be determined that the texts in the text pair are indeed similar.

[0392] Furthermore, in some embodiments of this application, when determining dissimilar texts for any fine-tuned text, the basic models bge, m3e, and text2vec can be used to vectorize all texts and calculate the similarity between the fine-tuned text and each text. Texts with similarity scores in the range of 0.3 to 0.6 are identified as dissimilar texts, i.e., negative samples. Samples with similarity scores in the range of 0.6 to 0.7, or texts that the large model considers relevant but have a similarity score less than 0.5, are identified as hard samples, which can also be used as dissimilar texts.

[0393] To further improve the accuracy of the embedding model encoding, based on the above embodiments, in some embodiments of this application, the process of determining similar text pairs generated based on the large model includes: generating associated texts corresponding to the obtained text segments based on the large model, wherein the associated texts include at least one of the question of the text segment, the overview text of the text segment, and the answer information corresponding to the question of the text segment; and randomly selecting any two texts from the text segment and the associated texts as similar text pairs.

[0394] In some embodiments of the present application, when generating similar text pairs, associated text corresponding to the obtained text segments can be generated based on a large model. The associated text includes at least one of the problem of the text segment, the summary text of the text segment, and the response information corresponding to the problem of the text segment. Among them, the text segment can be a text segment obtained by segmenting a pre-collected document. In some embodiments, PDF and txt can be segmented based on the langchain framework, and the markDown document can be segmented by title. The langchain framework is a programming framework that helps to use large language models (LLMs) in applications.

[0395] In some embodiments, when generating the problem of the text segment based on a large model, the segmented text segment and a pre-designed prompt text can be input into the large model, and the large model can generate a corresponding problem according to the text segment. The large model can be the Chatgpt (Chat Generative Pre-trained Transformer) large model. Among them, the input data of the large model can be:

[0396] '<Instruction>You are a Q&A assistant. Your current task is to generate corresponding questions based on the known information. If no question can be extracted from it, say "No corresponding question can be generated based on the known information",'

[0397] 'Do not fabricate information based on the known segments. The questions should be generated in Chinese.< / Instruction>\n'

[0398] '<Known information>{{context}}< / Known information>\n'

[0399] '<Question>{{question}}< / Question>\n'.

[0400] When generating the summary text of the text segment based on a large model, the input data of the large model can be:

[0401] '<Instruction>You are a Q&A assistant. Your current task is to generate a summary based on the known information using your expertise. If no summary can be obtained from it, say "No corresponding summary can be generated based on the known information",'

[0402] 'Do not fabricate information based on the segments. The answers should be in Chinese.< / Instruction>\n'

[0403] '<Known information>{{context}}< / Known information>\n'

[0404] '<Abstract>{{abstrast}}< / Abstract>\n'”。

[0405] Figure 31 is a schematic diagram of the generation principle of a problem and an overview text provided by some embodiments of the present application. As shown in Figure 31, when generating the problem and abstract of an article segment, that is, the segmented text segment, the pre-designed prompt words (i.e., prompt text) and the article segment can be input into the GPT large model to obtain the problem and abstract of the article segment. Among them, the abstract is the overview text described in the above embodiments.

[0406] After generating the problem of the text segment, the reply information corresponding to the problem of the text segment can also be generated based on the large model. At this time, the input data of the large model can be:

[0407] '<Instruction>You are a Q&A assistant. Your current task is to use your expertise to answer questions based on the known information. If you cannot obtain the answer from it, please say "The question cannot be answered based on the known information",'

[0408] 'Do not allow adding fabricated content to the answer, and the answer should be in Chinese.< / Instruction>\n'

[0409] '<Known Information>{{top-k_context}}< / Known Information>\n'

[0410] '<Answer>{{answer}}< / Answer>\n'

[0411] The large model will reply to the corpus question based on the above input data.

[0412] Figure 32 is a schematic diagram of the generation principle of a reply information provided by some embodiments of the present application. As shown in Figure 32, when generating the reply information of an article segment, that is, the problem of the segmented text segment, the problem and the article segment can be input into the GPT large model to obtain the reply information of the problem of the article segment.

[0413] After determining the associated text corresponding to the text segment, any two texts can be randomly selected from the text segment and the associated text as a similar text pair. For example, the similar text pair can be problem - overview text; problem - reply information; problem - text segment; overview text - reply information; overview text - text segment; reply information - text segment.

[0414] Based on the above embodiments, referring to Figure 45, the processor 4501 included in the electronic device is further configured to execute a computer program stored in the memory 4503 to cause the electronic device to perform: acquiring a training set pre-saved for a specific domain, the training set including sample text; inputting the sample text into the encoder for encoding, acquiring a first feature vector output by each encoding sub-layer in the encoder; determining a predicted feature vector of the sample text based on the weights corresponding to each encoding sub-layer and the first feature vector; determining a first loss value based on the predicted feature vector and the standard feature vector of the sample text, and training the initial embedding model based on the first loss value to obtain the embedding model.

[0415] In one possible implementation, the initial embedding model further includes a decoder. After determining the predicted feature vector of the sample text and before training the initial embedding model based on the first loss value, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform: inputting the predicted feature vector into the decoder for decoding to obtain the predicted text output by the decoder; determining a second loss value based on the predicted text and the sample text; when the processor 4501 performs the step of training the initial embedding model based on the first loss value to obtain the embedding model, it is specifically configured to execute the computer program to cause the electronic device to perform: training the initial embedding model based on the first loss value and the second loss value to obtain the embedding model.

[0416] In one possible implementation, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform: acquiring any sample text in the training set as a first fine-tuned text, a first similar text corresponding to the first fine-tuned text, and a first dissimilar text corresponding to the first fine-tuned text; inputting the first fine-tuned text, the first similar text, and the first dissimilar text into the embedding model respectively to obtain each second feature vector; determining a third loss value based on a fourth similarity between the second feature vectors corresponding to the first fine-tuned text and the first similar text, and a fifth similarity between the second feature vectors corresponding to the first fine-tuned text and the first dissimilar text; and fine-tuning the embedding model based on the third loss value.

[0417] In one possible implementation, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform: acquiring any sample in the training set as a second fine-tuned text, a second similar text corresponding to the second fine-tuned sample, and a second dissimilar text corresponding to the second fine-tuned text; inputting the second fine-tuned text, the second similar text, and the second dissimilar text into the embedding model respectively to obtain each third feature vector; determining a sixth similarity between the third feature vectors of the second fine-tuned sample and the second similar text, and a seventh similarity between the third feature vectors of the second fine-tuned sample and the second dissimilar text; determining a fourth loss value based on the difference between the sixth similarity and the seventh similarity, and an activation function; and fine-tuning the embedding model based on the fourth loss value.

[0418] In one possible implementation, the processor 4501 executes the computer instructions to cause the electronic device to perform a process for determining similar texts corresponding to any fine-tuned text, including: acquiring similar text pairs generated based on a large model, determining similar text pairs containing the fine-tuned text as target text pairs; determining a fourth feature vector for each text in the target text pair based on other pre-trained embedding models; and determining whether non-fine-tuned texts in the target text pair are similar texts to the fine-tuned text based on an eighth similarity between each fourth feature vector and a preset threshold.

[0419] In one possible implementation, after identifying similar text pairs containing the fine-tuned text as target text pairs, and before determining each fourth feature vector of the target text pairs based on other pre-trained embedding models, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform: determining whether the texts in the target text pairs are similar based on an eighth model, and if so, continuing to execute the subsequent step of determining each fourth feature vector of the target text pairs based on other pre-trained embedding models.

[0420] In one possible implementation, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform a process for determining similar text pairs generated based on a large model, including: generating associated text corresponding to the obtained text segment based on the large model, the associated text including at least one of the question of the text segment, the overview text of the text segment, and the answer information corresponding to the question of the text segment; and randomly selecting any two texts from the text segment and the associated text as similar text pairs.

[0421] Some embodiments of this application also provide a fourth question-and-answer method based on a large model for road hazard investigation. This method includes: if an investigation initiation request is received, displaying a list of locations to be investigated on a monitor; if an investigation instruction is received to investigate any target location in the list, obtaining on-site investigation records; based on each question to be investigated, parsing the content included in the on-site investigation records to determine the investigation result for each question; and displaying each question and its corresponding result on a monitor. Here, the question to be investigated is the aforementioned question to be answered, and the investigation result is the aforementioned response information. This method eliminates the need for investigators to arrive at the scene based on the location information of the locations to be investigated after receiving the investigation task, and to determine the required operations and investigation priorities based on their own understanding and process requirements according to the task description. For example, in the scenario of a long downhill section, the current road hazard inspection process generally requires determining the downhill gradient, identifying the presence of small access intersections, assessing the adequacy of speed reduction measures, speed limit signs, and monitoring facilities, identifying any poor visibility, determining if there are curves combined with slopes, and supplementing the observed issues based on the specific on-site conditions. After completing the inspection according to the above steps, inspectors can fill out the report on the road hazard inspection terminal. If any unclear issues are encountered during the inspection, they need to manually refer to electronic or paper materials, or consult other inspectors. After the inspection is completed, an inspection report is submitted and sent to relevant personnel for review and workflow processing. It is evident that the aforementioned method simplifies the road hazard inspection process, reduces the professional skills required of inspectors, and improves the efficiency of road hazard inspection.

[0422] Figure 33 is a flowchart illustrating a fourth large-model-based question-answering process provided in some embodiments of this application. The process includes the following steps:

[0423] S3301: If a request to initiate an investigation is received, display the list of locations to be investigated.

[0424] S3302: If an investigation instruction is received to investigate any target location in the list of locations to be investigated, obtain the on-site investigation record, which includes at least one of images, audio, and text.

[0425] S3303: Based on each problem to be investigated, the contents included in the on-site investigation record are parsed to obtain and display the investigation results corresponding to each problem to be answered.

[0426] In one possible implementation, the step of parsing the content included in the on-site investigation record based on each problem to be investigated to obtain the investigation result corresponding to each problem to be answered includes: analyzing the on-site investigation record and the list of problems to be investigated based on a large model to determine the investigation result corresponding to each problem to be investigated in the list of problems to be investigated.

[0427] In one possible implementation, acquiring the on-site investigation record includes: if an image capturing instruction is received, capturing an image of the target location to be investigated; parsing the content included in the on-site investigation record based on each question to be investigated to obtain the investigation result corresponding to each question to be answered includes: identifying the target object corresponding to each object to be investigated in the image, and whether the target object is abnormal; analyzing whether the target object is abnormal and each question to be investigated based on a large model to determine the investigation result corresponding to each question to be investigated.

[0428] In one possible implementation, acquiring the on-site investigation record includes: if an audio input command is received, acquiring audio; performing speech recognition on the audio to obtain and display the recognized text corresponding to the audio; if a text confirmation command is received, identifying the recognized text as the on-site investigation record.

[0429] In one possible implementation, the step of parsing the content included in the on-site investigation record based on each question to be investigated to obtain the investigation result corresponding to each question to be answered includes: if the on-site investigation record is text, inputting the on-site investigation record, each question to be investigated, and the prompt text into the large model, so that the large model analyzes the on-site investigation record and each question to be investigated based on the prompt text, and determines the investigation result corresponding to each question to be investigated.

[0430] In one possible implementation, the step of parsing the content included in the on-site investigation record based on each question to be investigated, and obtaining the investigation result corresponding to each question to be answered, includes: if the on-site investigation record is multiple images of the same target object, determining the overlapping areas in all images based on the content included in each image; generating a complete target object image based on the target objects contained in all the overlapping areas; determining whether there is an anomaly in the target object in the complete target object image based on a large model, and determining the investigation result of the question to be investigated corresponding to the target object.

[0431] In one possible implementation, the step of parsing the content included in the on-site investigation record based on each question to be investigated to obtain the investigation result corresponding to each question to be answered includes: if the on-site investigation record is multiple images of the same target object, for each image, analyzing the image and each question to be investigated based on a large model to determine the candidate investigation result corresponding to each question to be investigated for the target object in the image; for each question to be investigated, fusing the candidate investigation results corresponding to each question to be investigated for each image to obtain the investigation result corresponding to the question to be investigated.

[0432] In one possible implementation, the method further includes: acquiring the current first location information of the road hazard investigation terminal and the second location information of the location to be investigated corresponding to the problem for which no investigation result has been obtained; determining a target supplementary location to be investigated within the distance threshold based on the first location information, the second location information, and a distance threshold, and sending it to the road hazard investigation terminal, so that the road hazard investigation terminal displays the target supplementary location to be investigated, so that the user of the road hazard investigation terminal can go to the target supplementary location to investigate the problem for which no investigation result has been obtained.

[0433] In one possible implementation, after determining the target supplementary location to be investigated within the distance threshold, the method further includes: determining the current movement speed of the road hazard investigation terminal; determining the target delay value corresponding to the movement speed according to the pre-saved correspondence between different speeds and delay values; determining the supplementary location to be investigated that is moving away from the target supplementary locations to be investigated according to the current time and the target delay value; and deleting the supplementary location to be investigated that is moving away from the target supplementary locations to be investigated.

[0434] In some embodiments of this application, a road hazard investigation terminal is also provided, which includes a display and a processor. In some embodiments, the road hazard investigation terminal used in the road hazard investigation process can be a mobile phone with an application installed on it, which can be used for road hazard investigation. When performing corresponding operations, the road hazard investigation terminal hardware can support functions such as sound acquisition, image acquisition, sound playback, and location information acquisition. Figure 34 is a schematic diagram of a road hazard investigation terminal provided in some embodiments of this application. As shown in Figure 34, the terminal can be configured with a microphone, a display, and a speaker. The microphone is used for sound acquisition, the display is used for information display, and the speaker is used for sound playback. Simultaneously, the terminal can also have a Global Positioning System (GPS) positioning function to achieve location information acquisition.

[0435] It should be noted that the above examples are provided for ease of understanding only. Road hazard detection terminals are not limited to mobile phones; they can also be other devices such as servers or recorders. Of course, even if the road hazard detection terminal is a mobile phone, the functions of a mobile phone are not limited to the examples above.

[0436] The process of road hazard investigation based on a road hazard investigation terminal will be described below with reference to various embodiments. Figure 35 is a schematic diagram of the structure of a road hazard investigation terminal provided in some embodiments of this application. The terminal 350 includes a display 3501 and a processor 3502. The processor 3502 is configured to: if a troubleshooting start request is received, display a list of locations to be investigated on the display 3501; if a troubleshooting instruction is received to investigate any target location in the list of locations to be investigated, obtain on-site investigation records and send them to the server. The on-site investigation records include at least one of images, audio, and text; if each problem to be investigated and the corresponding investigation result are received, display them on the display 3501. The investigation result is determined by the server based on each problem to be investigated by parsing the content included in the on-site investigation record.

[0437] To identify potential road hazards, inspectors can use the Road Hazard Inspection Terminal 350 (hereinafter referred to as Terminal 350) to perform road hazard inspection tasks. When a road hazard inspection is required, the inspector can send an inspection start request to Terminal 350. For example, launching an application on the terminal's display page can be considered as the inspector sending an inspection start request. Of course, the inspector can also send an inspection start request by clicking the start button in an application.

[0438] In some embodiments of this application, if the processor 3502 receives a troubleshooting initiation request, it indicates that the troubleshooter wishes to perform a hazard investigation task. The processor 3502 can control the display 3501 to display the acquired list of locations to be investigated. This list of locations to be investigated can be obtained by the terminal 350 from a preset storage location, or it can be sent by other electronic devices connected to the terminal 350.

[0439] For example, terminal 350 can establish a connection with the server via WebSocket and periodically pull a list of locations to be investigated from the server in a heartbeat manner. After receiving the request from terminal 350, the server can read all the locations to be investigated from the database and return all the locations to be investigated to terminal 350 according to the set format.

[0440] Figure 36 is a schematic diagram of a list of locations to be investigated provided in some embodiments of this application. As shown in Figure 36, the display 3501 of the terminal 350 displays the acquired list of locations to be investigated, and each location to be investigated has a corresponding "Investigate" button. To facilitate investigators to go to the corresponding locations for investigation, the specific location and name of each location to be investigated can be displayed. After the investigator sees the list of locations to be investigated on the terminal 350, they can go to the investigation site according to the description of the specific location corresponding to each location. After arriving at the site, the investigator can click the corresponding "Investigate" button on the display page to enter the investigation page of the corresponding location to be investigated and start the investigation task. The list of locations to be investigated can include locations that have been investigated or locations that have not been investigated. Investigators can click the "To be investigated" or "Investigated" buttons to filter locations. Of course, investigators can also click the "My" button to view the investigation tasks they need to perform.

[0441] After viewing the list of locations to be investigated, the inspector can select any location to investigate. For ease of description, the selected location will be referred to as the target location to be investigated. If the processor 3502 receives an instruction to investigate the target location, it can assume that the inspector is about to investigate the potential hazards at that location. On-site investigation records can be acquired so that the server can determine whether the corresponding target object has a safety hazard based on these records. These on-site investigation records can be images, audio, or text, or a combination of both. The records can be directly input by the inspector; for example, after arriving at the scene, the inspector can take a picture of the target object and input the image into their terminal 350. The records can also be sent by other electronic devices connected to the terminal 350. Furthermore, the records can be retrieved from a preset storage location. In other words, the acquired on-site investigation records can be real-time data or historical data. This also illustrates that the road hazard investigation terminal provided in some embodiments of this application can perform both real-time online investigation and offline investigation. The embodiments of this application do not limit the method of obtaining on-site investigation records; those skilled in the art can configure it as needed.

[0442] After obtaining the on-site investigation record, the record can be sent to the server, which then determines the investigation result for each problem to be investigated based on the record. The server can retrieve each problem when determining it. In some embodiments of this application, there can be one or more problems to be investigated. For example, multiple problems can be stored in a problem list. That is, each location to be investigated corresponds to a list of problems that need to be investigated, and these problems constitute the problem list. The number of problems corresponding to the same location can be one or more. Problems corresponding to different locations can be the same or different. These problems can be pre-stored in a preset storage location, such as in a database; or they can be obtained from an external source along with the on-site investigation record and sent by the road hazard investigation terminal, for example, when the investigator enters the on-site investigation record, they also input the corresponding problem list into the road hazard investigation terminal.

[0443] After obtaining the on-site investigation records and the list of issues to be investigated, the content included in the on-site investigation records can be parsed based on each issue in the list to determine the investigation result for each issue. In some embodiments, the server uses a pre-trained image recognition model to recognize images and determine the investigation result for each issue. For example, any issue to be investigated could be "whether there is a warning sign on the roadside," and the determined investigation result could be "yes."

[0444] After determining the investigation results for each problem to be investigated, the server can send each problem to be investigated and its corresponding results to the road hazard investigation terminal so that the investigators can view the results. The road hazard investigation terminal will then display each problem to be investigated and its corresponding results.

[0445] Figure 37 is a schematic diagram of the inspection results display provided by some embodiments of this application. As shown in Figure 37, the target inspection location is "Inspection Location 1". The inspection results for the following issues are: "Is there road surface damage?", "Is there missing or damaged roadside guardrails?", "Is there damaged or unclear traffic signs?", "Is there missing or damaged traffic lights?", and "Is there damaged or missing road lighting facilities?". After viewing the inspection results for each issue, the inspector can check the results and click the "Save" button after confirming that everything is correct.

[0446] In some embodiments, the investigation result may be that an anomaly exists, no anomaly exists, or no detection is found. If the investigation result is "no detection found," it means that the problem was not investigated during this investigation and can wait for subsequent investigations by other investigators. In other words, when conducting road hazard investigations, investigators may investigate all problems corresponding to a specific location, or they may investigate only some of the problems.

[0447] In some embodiments of this application, if a troubleshooting initiation request is received, the obtained list of locations to be investigated is displayed. If a troubleshooting instruction is received to investigate any target location in the list of locations to be investigated, the on-site investigation record is obtained. Then, the service parses the content included in the on-site investigation record based on each problem to be investigated, determines the investigation result of each problem to be investigated, and displays it on the display for the inspector to view. The entire process only requires obtaining the on-site investigation record. Subsequently, the content included in the on-site investigation record is parsed based on each problem to be investigated to determine the investigation result. There is no need for the inspector to conduct in-depth analysis of the on-site investigation record on-site, which reduces the technical requirements for the inspector and improves the efficiency of hazard investigation.

[0448] To improve the accuracy of road hazard investigation, based on the above embodiments, in some embodiments of this application, the server can analyze the on-site investigation records and the list of problems to be investigated based on a large model to determine the investigation result corresponding to each problem in the list of problems to be investigated.

[0449] In some embodiments of this application, when the server parses the content included in the on-site investigation record based on each problem to be investigated and determines the investigation result of each problem, it can analyze the on-site investigation record and the list of problems to be investigated based on a large model to determine the investigation result corresponding to each problem. In some embodiments of this application, the large model can be a large language model or a multimodal large model. Those skilled in the art can choose a suitable large model according to the type of on-site investigation record, and this application does not impose any limitations on this. After receiving the on-site investigation record and each problem to be investigated, the large model can analyze each problem sequentially and determine the corresponding investigation result. In some embodiments, any problem to be investigated can be "whether there is a warning sign on the roadside", and the investigation result determined by the large model can be "yes".

[0450] It should be noted that how the large model analyzes the on-site investigation records and each problem to be investigated is part of the large model's internal processing mechanism. In some embodiments, the server can use any of the aforementioned question-and-answer methods based on the large model to determine the corresponding investigation result based on each problem in the list of problems to be investigated. Here, the problem to be investigated is the aforementioned problem to be answered, and the investigation result is the aforementioned response information. The processing flow inside the large model has been described in detail above and will not be repeated here.

[0451] To further improve the accuracy of road hazard investigation, based on the above embodiments, the server is specifically configured to perform the following: if the on-site investigation record is an image, based on the object to be investigated corresponding to each problem to be investigated, identify the target object corresponding to each object to be investigated in the image, and whether the target object has any abnormalities; analyze whether the target object has any abnormalities and each problem to be investigated based on a large model, and determine the investigation result corresponding to each problem to be investigated.

[0452] In some embodiments of this application, terminal 350 may also include a camera, as shown in FIG38. FIG38 is a schematic diagram of the back of a terminal provided in some embodiments of this application. The upper left (left and right in the figure) corner of the terminal shown in FIG38 is the rear camera of terminal 350.

[0453] When entering on-site investigation records, investigators can click the "Add Image" button to upload at least one image. As shown in Figure 39, which is a schematic diagram of on-site investigation record input provided by some embodiments of this application, investigators can click the "+" in Figure 38 to add images, and the added images can be used as on-site investigation records. After receiving the image addition instruction, terminal 350 can jump to display the image addition page shown in Figure 40, which is a schematic diagram of the image upload page provided by some embodiments of this application. Investigators can select existing images in terminal 350 or take images immediately. After the image selection is completed, they can click the "Finish" button to complete the upload of the on-site investigation record.

[0454] If the processor 3502 receives an image capture command, it can control the image acquisition unit to capture an image of the target location to be investigated. This image capture command can be triggered by the investigator clicking the "capture" button. In other words, the investigator can use the handheld terminal 350 to capture an image of the target location to be investigated through the image acquisition unit.

[0455] After obtaining the image of the target location to be investigated, the image can be sent to the server as an on-site investigation record.

[0456] If the server receives the on-site investigation record as an image, it can first determine the object to be investigated corresponding to each problem to be investigated. For example, the object to be investigated for the question "Are the zebra crossings clear?" is "zebra crossings"; the object to be investigated for the question "Are the buildings tilted?" is "buildings"; the object to be investigated for the question "Are there any missing or damaged warning signs?" is "warning signs"; the object to be investigated for the question "Are there any missing or damaged roadside guardrails?" is "roadside guardrails"; the object to be investigated for the question "Are there any missing or damaged central median strips?" is "central median strips"; the object to be investigated for the question "Are there any missing or damaged speed limit signs?" is "speed limit signs"; the object to be investigated for the question "Are there any roadside obstacles?" is "roadside obstacles"; the object to be investigated for the question "Are there any issues with excessive height on curves?" is "road surface"; and the object to be investigated for the question "Are there any issues with the road surface's anti-skid coefficient?" is "road surface". The object to be investigated for each problem can be determined based on the analysis of the problem using a pre-trained model; or it can be saved by relevant personnel when writing the problem to be investigated.

[0457] After identifying the object to be investigated for each problem, the corresponding target object in the image can be identified based on that object. In other words, the presence and location of the target object in the image are determined to facilitate subsequent assessment of any anomalies. Simultaneously, the anomaly of each target object can be determined, such as whether it is damaged. It should be noted that both small and large models can be used for target object identification and detection; those skilled in the art can configure the model as needed.

[0458] After identifying the target object and determining whether it exhibits any anomalies, a descriptive text indicating whether the target object has anomalies, along with each problem to be investigated, can be input into the large model. The large model then analyzes the descriptive text and each problem to determine the investigation result for each problem. In some embodiments, the data format output by the large model can be as shown in Table 8 below:

[0459] Table 8

[0460] In one possible implementation, the identification of target objects in an image can also be based on a large model. In some embodiments of this application, the image and corresponding prompts can be input into the large model, enabling the large model to identify the region where the target object is located in the image and determine whether the target object is abnormal, thereby determining the investigation result corresponding to each problem to be investigated.

[0461] In one possible implementation, after the image is acquired, it can be preprocessed, such as by noise reduction, to further ensure that subsequent road hazard investigations are accurate based on the image.

[0462] In some embodiments of this application, a training set can be pre-constructed, and the original model can be trained based on the sample data included in the training set to obtain a model for recognizing target objects and detecting whether there are anomalies in the target objects.

[0463] When constructing the training set, a predetermined number of sample images can be collected in advance, and the regions where the target objects are located can be marked in these sample images, for example, by using bounding boxes to mark the target objects. A corresponding label can be saved for each sample image, describing the target object present in the image and whether the target object exhibits any anomalies. These sample images and their corresponding labels constitute the sample data.

[0464] After obtaining the training set, each sample data in the training set can be input into the original model for training until the model converges, resulting in a trained model. The trained model can be used to identify whether a target object exists in an image, and whether the target object exhibits any anomalies. Table 9 below shows an example of a recognition result provided by some embodiments of this application:

[0465] Table 9

[0466] In Table 8 above, "-" indicates that no corresponding result was identified. According to Table 8, the trained model can identify warning signs, and these warning signs are abnormal; it cannot identify roadside guardrails; it can identify median strips, and these median strips are not abnormal; it can identify speed limit signs, and these speed limit signs are not abnormal; it can identify roadside obstacles, but there is no issue of whether the obstacles are abnormal. Roadside obstacles need to be cleared promptly. Therefore, the identification result for whether roadside obstacles are abnormal in Table 8 is "-".

[0467] To further improve the accuracy of road hazard investigation, based on the above embodiments, in some embodiments of this application, if the on-site investigation record is audio, the server is further configured to: perform speech recognition on the audio to obtain the recognized text corresponding to the audio and send it to the road hazard investigation terminal; if a text confirmation instruction is received, update the on-site investigation record using the recognized text.

[0468] To facilitate investigators in entering on-site investigation records, the terminal 350 may also include a microphone. Investigators can enter on-site investigation records via voice input. Figure 41 is a schematic diagram of on-site investigation record input provided by some embodiments of this application. As shown in Figure 41, investigators can click the "Voice Input" button to speak the information they plan to input in natural language, which is then collected by the audio acquisition unit to obtain audio.

[0469] If the processor 3502 receives an audio input command, it can control the audio acquisition unit to acquire audio through the microphone. This audio input command can be triggered when an investigator clicks the "voice input" button.

[0470] After receiving the collected audio, processor 3502 can send the audio to the server. If the server receives the on-site investigation record as audio, it can perform speech recognition on the audio to obtain the corresponding recognized text. Since speech recognition may result in misrecognition or typos, the recognized text can be sent to the road hazard investigation terminal after it is obtained. If the road hazard investigation terminal receives the recognized text sent by the server, it can display the recognized text on display 3501 so that the investigators can check it.

[0471] Figure 42 is a schematic diagram of on-site inspection record input provided by some embodiments of this application. As shown in Figure 42, the terminal 350 displays the recognized text "User-input description of potential hazards: This road has many problems. The road surface is full of potholes, making it easy for vehicles to lose control. The roadside guardrails are also broken and cannot stop vehicles from rushing out. Road signs and traffic lights are unclear, which can easily lead to accidents. The lighting at night is insufficient, making it difficult for drivers to see the road, increasing the risk. It is recommended to repair the road, repair the guardrails, clarify the road signs and traffic lights, and improve nighttime lighting." in the dialog box. After confirming that the information is correct, the inspector can click the "Upload" button to complete the on-site inspection record entry. If the processor 3502 receives a text confirmation instruction, it can send the recognized text to the server. After receiving the text confirmation instruction, the server can use the recognized text to update the on-site inspection record. This text confirmation instruction can be triggered by the inspector clicking the "Upload" button on the road hazard inspection terminal.

[0472] In some embodiments of this application, if the on-site investigation record is audio, the processor of the road hazard investigation terminal can also recognize the audio to obtain the corresponding recognized text, and display the recognized text on the display; if a text confirmation command is received, the recognized text can be used to update the on-site investigation record. That is, in some embodiments of this application, when performing speech recognition on audio, the terminal 350 can perform recognition based on a built-in speech-to-text software development kit (SDK), meaning the audio is processed locally. If the terminal 350 does not have a built-in speech-to-text SDK, the audio can be uploaded to a server, speech recognition can be completed, and then the server can send the recognized text to the terminal 350.

[0473] To further improve the accuracy of road hazard investigation, based on the above embodiments, in some embodiments of this application, the server is specifically configured to perform the following: if the on-site investigation record is text, input the on-site investigation record, each problem to be investigated, and the prompt text into the large model, so that the large model analyzes the on-site investigation record and each problem to be investigated based on the prompt text, and determines the investigation result corresponding to each problem to be investigated.

[0474] In some embodiments of this application, if the on-site investigation record obtained by the server is text, the text classification, text extraction, and text generation techniques of the large-scale model can be invoked to match the on-site investigation record with the list of issues to be investigated. If the on-site investigation record contains a part related to the issues to be investigated, an answer can be given based on the content of the on-site investigation record. Issues not covered by the on-site investigation record do not need to be answered. The text can be input by the investigator via audio or manually entered by the investigator into the input box. In some embodiments of this application, the on-site investigation record, the list of issues to be investigated, and the prompt text can be input into the large-scale model, so that the large-scale model can analyze the on-site investigation record and the list of issues to be investigated based on the prompt text, and determine the investigation result corresponding to each issue in the list of issues to be investigated.

[0475] In some embodiments, the on-site investigation record and the issues to be investigated may be the contents shown in Table 10 below:

[0476] Table 10

[0477] Based on the above on-site investigation records and issues to be investigated, the investigation results are shown in Table 11 below:

[0478] Table 11

[0479] To further improve the accuracy of road hazard investigation, based on the above embodiments, in some embodiments of this application, the server is specifically configured to perform the following: if the on-site investigation record consists of multiple images of the same target object, determine the overlapping areas in all images based on the content included in each image; generate a complete target object image based on the target object contained in all the overlapping areas; determine whether there is an anomaly in the target object in the complete target object image based on a large model, and determine the investigation result of the problem to be investigated corresponding to the target object.

[0480] In some embodiments of this application, inspectors can upload or capture multiple images of the same target object, such as multiple images from different shooting positions and angles. Since different shooting positions and angles may result in a single image containing only a portion of the target object, to further improve the accuracy of road hazard investigation, in some embodiments of this application, if the acquired on-site investigation record consists of multiple images of the same target object, overlapping regions are determined based on the content included in each image. These overlapping regions are image regions containing the target object. After determining all overlapping regions, a complete image of the target object can be generated based on the target object contained in all overlapping regions. In some embodiments of this application, these overlapping region images can undergo contrast enhancement, grayscale enhancement, and stretching processing, and then be reconstructed to obtain a complete image of the target object. How to generate a complete image of the same object based on multiple images is a known technique, and this application will not elaborate on it further.

[0481] In some embodiments of this application, in order to further improve the accuracy of road hazard investigation, the neighborhood averaging method and median filtering method can also be used to perform noise reduction processing on the complete target object image.

[0482] After obtaining a complete image of the target object, a large model can be used to determine whether there are any anomalies in the target object within the complete image, and to determine the investigation results for the corresponding problems to be investigated. How the large model determines whether there are any anomalies in the target object and the corresponding investigation results have been described in the above embodiments, and will not be repeated in this application.

[0483] To further improve the accuracy of road hazard investigation, based on the above embodiments, in some embodiments of this application, the server is specifically configured to perform the following: if the on-site investigation record consists of multiple images of the same target object, for each image, analyze the image and each problem to be investigated based on a large model to determine the candidate investigation results corresponding to each problem to be investigated for the target object in the image; for each problem to be investigated, fuse the candidate investigation results corresponding to each problem to be investigated for each image to obtain the investigation result corresponding to the problem to be investigated.

[0484] Because the observed state of the same object varies depending on the viewing angle—for example, the top of a cube might be damaged, but this damage wouldn't be visible in a frontal view—to further improve the accuracy of road hazard identification, in some embodiments of this application, if the acquired on-site investigation records consist of multiple images of the same target object, the target object in each image can be analyzed to determine the candidate investigation results for each image. Then, based on these candidate investigation results, the final investigation result is determined. In some embodiments of this application, for each image, a large model can be used to analyze the image and the list of problems to be investigated, determining the candidate investigation results for each problem corresponding to the target object in the image. After obtaining the candidate investigation results for each image and each problem, the candidate investigation results for each image can be fused to obtain the investigation result corresponding to that problem. When fusing the candidate investigation results, the candidate investigation result that appears most frequently can be determined as the investigation result corresponding to the problem. Alternatively, each candidate investigation result can be input into a large model, which can then integrate all the candidate investigation results and summarize the investigation results corresponding to the problem to be investigated.

[0485] To further improve the efficiency of road hazard investigation, based on the above embodiments, in some embodiments of this application, the server is further configured to: obtain the current first location information of the road hazard investigation terminal, and the second location information of the location corresponding to the problem to be investigated for which no investigation result has been obtained; determine the target supplementary location to be investigated within the distance threshold according to the first location information, the second location information, and a distance threshold, and send it to the road hazard investigation terminal, so that the road hazard investigation terminal displays the target supplementary location to be investigated, so that the user of the road hazard investigation terminal can go to the target supplementary location to investigate the problem for which no investigation result has been obtained.

[0486] To further improve the efficiency of road hazard investigation, in some embodiments of this application, the server can also obtain the terminal's current first location information. This first location information can be obtained by the terminal via GPS and sent to the server. After obtaining the first location information, the terminal 350 can upload it to the server at a fixed frequency, so that the server can obtain nearby locations to be investigated, allowing investigators to conduct road hazard investigations nearby.

[0487] In some embodiments of this application, the server can also identify all issues for which no investigation results have been obtained. It also determines the second location information of the location corresponding to the list of issues for investigation for each issue. The first and second location information can be street names, house numbers, or latitude and longitude information; this application does not impose any limitations on these.

[0488] After obtaining the first and second location information, supplementary locations to be investigated within the terminal's distance threshold can be determined based on the first and second location information and the distance threshold. In other words, based on the first and second location information, it is determined which locations within the terminal's distance threshold have not yet been investigated, and these locations are designated as supplementary locations to be investigated. The distance threshold can be any value, such as 100 meters, 1000 meters, 58.5 meters, etc., and can be configured as needed by those skilled in the art.

[0489] After determining the target location to be investigated, the server can send the target location to the terminal 350. After receiving the target location, the terminal 350 can display the second location information corresponding to the target location on the display 3501, so that the user can go to the target location to investigate the problem for which no investigation results have been obtained.

[0490] Figure 43 is a schematic diagram illustrating the determination of target supplementary investigation locations according to some embodiments of this application. As shown in Figure 43, the investigation locations corresponding to investigation problems for which investigation results have not been obtained include: investigation location 1, investigation location 2, investigation location 3, investigation location 4, and investigation location 5. Based on the first location information, each second location information, and the distance threshold r, the target supplementary investigation locations located within the distance threshold r of the terminal 10 can be determined as investigation location 3, investigation location 4, and investigation location 5, respectively. Investigators can go to any of the target supplementary investigation locations to conduct road hazard investigations.

[0491] To further improve the efficiency of road hazard investigation, based on the above embodiments, in some embodiments of this application, the server is further configured to perform: determining the current movement speed of the road hazard investigation terminal; determining the target latency value corresponding to the movement speed according to the pre-saved correspondence between different speeds and latency values; determining supplementary locations to be investigated that are moving away from the target supplementary locations to be investigated according to the current time and the target latency value; and deleting the supplementary locations to be investigated that are moving away from the target supplementary locations to be investigated.

[0492] Since inspectors carrying terminal 350 may be standing, walking, or in a moving vehicle, to further improve the efficiency of road hazard inspection, it can be determined whether terminal 350 is approaching or moving away from the target location to be inspected. In some embodiments of this application, the current movement speed of terminal 350 can be determined. Based on the pre-saved correspondence between different speeds and delay values, the target delay value corresponding to the movement speed is determined. Since the movement speed of a vehicle is relatively fast, while the walking speed is relatively slow, in some embodiments of this application, the delay value corresponding to high-speed movement can be determined as 0.2 seconds, the delay value corresponding to low-speed movement as 10 seconds, and the delay value corresponding to relative stillness as 45 seconds. In some embodiments of this application, after determining the current movement speed, it can be first determined whether the movement speed is high-speed, low-speed, or relatively still.

[0493] After determining the target latency value, the location being moved away from can be determined based on the current time and the target latency value. In other words, the location of the terminal is determined at the moment the current time plus the target latency value is calculated. Then, the distance between this location and the location corresponding to the first location information relative to each target location being investigated is compared to determine whether the terminal 350 is moving away from or approaching a target location being investigated.

[0494] To ensure that investigators do not backtrack or take detours, once a supplementary location that is moving away from the target supplementary location has been identified, it can be deleted from the list of target supplementary locations.

[0495] For one or more target supplementary locations to be investigated that the terminal 350 is approaching, the nearest target supplementary location to be investigated can be determined as the recommended investigation location. Once the investigator arrives at the target supplementary location to be investigated, an investigation command can be sent to investigate that target supplementary location. At this time, the pending issues in the list of pending issues obtained by the processor 3502 can be unanswered issues, that is, pending issues for which no investigation results have been obtained.

[0496] In one possible implementation, for relatively simple questions, investigators can be guided to answer them. In some embodiments of this application, investigators can be guided to open the application deployed on the investigation terminal via notification reminders, and the text information of the questions to be answered can be displayed on the screen, guiding the user to answer each question. Alternatively, the terminal's speaker can be used to announce the questions to be answered in sequence via voice broadcast. After the broadcast, the microphone is turned on to collect the investigator's response, and the investigator's voice response is converted into text and then fuzzy matched with the standard answer. The relevant answer is selected based on the matching result. For example, the terminal can announce the question to be investigated, "Is there a roadside obstacle?" The investigator can answer by typing "No" or by voice input. Based on the user's answer, the terminal 350 can determine the investigation result as "Is there a roadside obstacle? The answer is: No".

[0497] To facilitate subsequent review, after obtaining the investigation results, terminal 350 can send the corresponding questions to be answered and the investigation results to the server for storage.

[0498] It should be noted that the display of the road hazard investigation results in this application can be in any form, not limited to the examples above. Those skilled in the art can configure the display method of the investigation results as needed. Figure 44 is a schematic diagram of the investigation result display provided by some embodiments of this application. As shown in Figure 44, when displaying each problem to be investigated and the corresponding investigation result, all optional options corresponding to each problem to be investigated can be displayed, and the investigation result finally obtained from the server is highlighted on the display. For example, the border of the option box corresponding to the determined investigation result is highlighted. As shown in Figure 44, the investigation result for problem 1 "Is there any road surface damage?" is yes, so the border of the "Yes" option box is highlighted when displayed. The investigation result for problem 2 "Is there any missing or damaged roadside guardrail?" is no, so the border of the "No" option box is highlighted when displayed.

[0499] In some embodiments of this application, after obtaining data or a request, the terminal can send the corresponding data or request to the server, such as sending a troubleshooting initiation request, a troubleshooting instruction, or on-site troubleshooting records to the server. The server then retrieves the relevant data from the database according to the request and returns it to the terminal. For example, after retrieving the issues to be investigated from the database, the server sends each issue to the terminal. The server can also store the data obtained from the terminal, such as the troubleshooting results and on-site troubleshooting records corresponding to each issue, in the database.

[0500] Based on the above embodiments, and still referring to FIG45, the electronic device further includes a display (not shown in FIG45), and the processor 4501 is further configured to execute a computer program stored in the memory 4503 to cause the electronic device to perform: if a screening start request is received, display the obtained list of locations to be screened; if a screening instruction is received to screen any target location in the list of locations to be screened, obtain on-site screening records, the on-site screening records including at least one of images, audio, and text; based on each question to be screened, parse the content included in the on-site screening records to obtain the screening result corresponding to each question to be answered, and display it on the display.

[0501] In one possible implementation, the processor 4501 is specifically configured to execute the computer program to cause the electronic device to perform: analyzing the on-site investigation records and the list of problems to be investigated based on a large model, and determining the investigation result corresponding to each problem in the list of problems to be investigated.

[0502] In one possible implementation, the processor 4501 is specifically configured to execute the computer program to cause the electronic device to perform: if the on-site investigation record is an image, then based on the object to be investigated corresponding to each problem to be investigated, identify the target object corresponding to each object to be investigated in the image, and whether the target object is abnormal; analyze whether the target object is abnormal and each problem to be investigated based on a large model, and determine the investigation result corresponding to each problem to be investigated.

[0503] In one possible implementation, the processor 4501 is specifically configured to execute a computer program to cause the electronic device to perform: if the on-site investigation record is audio, perform speech recognition on the audio to obtain the recognized text corresponding to the audio and display it; if a text confirmation instruction is received, confirm the recognized text as the on-site investigation record.

[0504] In one possible implementation, the processor 4501 is specifically configured to execute the computer program to cause the electronic device to perform: if the on-site investigation record is text, input the on-site investigation record, each problem to be investigated, and the prompt text into the large model, so that the large model analyzes the on-site investigation record and each problem to be investigated based on the prompt text, and determines the investigation result corresponding to each problem to be investigated.

[0505] In one possible implementation, the processor 4501 is specifically configured to execute the computer program to cause the electronic device to perform: if the on-site investigation record consists of multiple images of the same target object, determine the overlapping regions in all images based on the content included in each image; generate a complete target object image based on the target object contained in all the overlapping regions; determine whether there is an anomaly in the target object in the complete target object image based on a large model, and determine the investigation result of the problem to be investigated corresponding to the target object.

[0506] In one possible implementation, the processor 4501 is specifically configured to execute the computer program to cause the electronic device to perform: if the on-site investigation record consists of multiple images of the same target object, for each image, analyze the image and each problem to be investigated based on a large model to determine the candidate investigation results corresponding to each problem to be investigated for the target object in the image; for each problem to be investigated, fuse the candidate investigation results corresponding to each problem to be investigated to obtain the investigation result corresponding to the problem to be investigated.

[0507] In one possible implementation, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform: acquiring the current first location information of the road hazard investigation terminal and the second location information of the location to be investigated corresponding to the problem for which no investigation result has been obtained; determining a target supplementary location to be investigated within the distance threshold based on the first location information, the second location information, and a distance threshold, and sending it to the road hazard investigation terminal, so that the road hazard investigation terminal displays the target supplementary location to be investigated, so that the user of the road hazard investigation terminal can go to the target supplementary location to investigate the problem for which no investigation result has been obtained.

[0508] In one possible implementation, after determining the target supplementary location to be investigated within the distance threshold, the processor 4501 is further configured to execute the computer program to cause the electronic device to perform: determining the current movement speed of the road hazard investigation terminal; determining the target delay value corresponding to the movement speed according to a pre-saved correspondence between different speeds and delay values; determining the supplementary location to be investigated that is moving away from the target supplementary location to be investigated based on the current time and the target delay value; and deleting the supplementary location to be investigated that is moving away from the target supplementary location to be investigated.

[0509] Based on the above embodiments, any of the above-mentioned question-answering methods based on large models in some embodiments of this application can also be implemented through the collaboration of various modules. The specific methods of each module have been described in detail in the foregoing embodiments and will not be elaborated here.

[0510] Based on the above embodiments, some embodiments of this application provide a computer-readable and non-volatile storage medium storing a computer program executable by a processor. When the program runs on the processor, it causes the processor to implement any of the aforementioned question-answering method steps based on a large model.

[0511] Finally, it should be noted that the above embodiments are only used to illustrate the solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding solutions to deviate from the scope of the solutions of the embodiments of this application.

Claims

1. An electronic device comprising: a communication interface; a memory configured to store a computer program; a processor connected with the memory and the communication interface, configured to execute the computer program to cause the electronic device to perform: determining a target text block matching a question to be replied according to each text block saved in advance; if non-text data is saved in advance for the target text block, obtaining a summary text of the saved non-text data, the non-text data including at least one of a picture and a table, the summary text being a text summarizing all contents in the non-text data; inputting the question to be replied, the target text block and the summary text into a first large model in a preset format to obtain reply information.

2. The electronic device of claim 1, if the non-text data saved in advance for the target text block includes a target table, after the obtaining a summary text of the saved non-text data, and before the inputting the question to be replied, the target text block and the summary text into a first large model in a preset format, the processor is further configured to execute the computer program to cause the electronic device to perform: obtaining an enhanced label saved in advance for each entry in the target table, the enhanced label being key information obtained after summarizing the content of the corresponding entry; determining a first similarity between the question to be replied and the enhanced label corresponding to each entry; determining a target entry matching the question to be replied according to each of the first similarities, and obtaining a target index of the target entry; the inputting the question to be replied, the target text block and the summary text into a first large model in a preset format to obtain reply information, comprising: inputting the question to be replied, the target text block, the summary text, all table contents of the target table and the target index into the first large model in a preset format to obtain reply information.

3. The electronic device of claim 2, the determination process of the enhanced label corresponding to each entry in any table comprising: format converting the table to obtain all table contents included in the table; inputting the all table contents and a first prompt text into a second large model to obtain the enhanced label corresponding to each entry in the table, the first prompt text being used to prompt the second large model to analyze each entry in the all table contents and output a text summarizing the content of the corresponding entry.

4. The electronic device of claim 3, the processor is further configured to execute the computer program to cause the electronic device to perform: for each enhanced label corresponding to each entry, determining whether the semantic of the enhanced label is repeated with any other enhanced label corresponding to the entry; if yes, deleting any enhanced label with repeated semantics, and continuing to perform the step of determining whether the semantic of each enhanced label corresponding to each entry is repeated with any other enhanced label corresponding to the entry until the semantics between the enhanced labels corresponding to each entry are not repeated.

5. The electronic device of claim 4, wherein the processor is configured to execute the computer program to cause the electronic device to perform: determining a second similarity between the augmented label and the other augmented labels; determining that the augmented label is semantically duplicated with the other augmented labels if the second similarity is greater than a first preset threshold; and determining that the augmented label is not semantically duplicated with the other augmented labels if the second similarity is not greater than the first preset threshold.

6. The electronic device of claim 3 or 4, wherein the processor is further configured to execute the computer program to cause the electronic device to perform: inputting the augmented label corresponding to each entry and a second prompt text to a third large model to obtain a summary label of each entry, the second prompt text being used to prompt the third large model to summarize the augmented label corresponding to each entry to generate a representative label; and updating the augmented label of each entry using the summary label corresponding to each entry.

7. The electronic device of claim 1, wherein the process of determining the summary text of the table comprises: performing format conversion on the table to be stored to obtain all table contents; and processing the all table contents based on a fourth large model to obtain the summary text of the table to be stored.

8. The electronic device of claim 1, wherein the process of determining whether the non-text data corresponding to the target text block is pre-stored comprises: obtaining a first feature vector of the summary text of each pre-stored non-text data; determining a third similarity between a second feature vector of the target text block and each first feature vector, respectively; and determining whether the non-text data corresponding to the target text block is pre-stored according to each third similarity and a second preset threshold.

9. The electronic device of claim 8, wherein the process of determining the second feature vector of the target text block comprises: dividing the target text block into multiple text sub-blocks according to a preset division rule; determining a third feature vector of each text sub-block as the second feature vector, respectively; and the determining the third similarity between the second feature vector of the target text block and each first feature vector, respectively, comprises: determining the third similarity between the second feature vector of each text sub-block and each first feature vector, respectively.

10. The electronic device of claim 1, wherein the processor is further configured to execute the computer program to cause the electronic device to perform: performing standardized rewriting on the question to be replied based on a fifth large model pre-trained for a target field to obtain a standard question corresponding to the question to be replied; searching for a target candidate text matching the standard question in a knowledge base; performing text splicing on the target candidate text, the standard question, and a prompt text to obtain input data, and inputting the input data into a sixth large model to obtain reply information, the reply information being a text obtained by the sixth large model from analyzing the target candidate text and the standard question according to a requirement of the prompt text. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 11.The electronic device of claim 10, after the obtaining the standard question corresponding to the question to be replied, and before the searching the target candidate text matching the standard question in the knowledge base, the processor is further configured to execute the computer instructions to cause the electronic device to perform: determining a target intent corresponding to the standard question based on a classification model; wherein the searching the target candidate text matching the standard question in the knowledge base comprises: searching the target candidate text matching the standard question in a target knowledge base corresponding to the target intent. 12.The electronic device of claim 11, wherein the classification model comprises a feature extraction layer and a linear layer, and a training process of the classification model comprises: obtaining a word vector encoding and a label corresponding to a sample text, the label being used to identify a standard intent of the sample text, the word vector encoding comprising a part-of-speech identification of each character; inputting the word vector encoding and the label into the feature extraction layer, the feature extraction layer determining a feature extraction focus according to the part-of-speech identification when extracting semantic features to obtain predicted semantic features; inputting the predicted semantic features into the linear layer for intent prediction to obtain a predicted probability corresponding to each preset intent; determining a target predicted probability corresponding to the standard intent according to the predicted probability corresponding to each preset intent; determining a first loss value according to the target predicted probability and a preset control parameter, and adjusting parameters of the classification model according to the first loss value. 13.The electronic device of claim 10 or 11, wherein the processor is specifically configured to execute the computer instructions to cause the electronic device to perform: determining a similarity between the standard question and each text in the knowledge base according to a first feature vector of the standard question and a second feature vector of each text; determining a relevance score between the standard question and each text in the knowledge base based on a keyword retrieval algorithm; determining a target candidate text matching the standard question in the knowledge base according to each similarity and each relevance score. 14.The electronic device of claim 13, wherein the processor is specifically configured to execute the computer instructions to cause the electronic device to perform: for each text in the knowledge base, determining a target score corresponding to the text according to a similarity, a relevance score and a preset weight corresponding to the text; determining a target candidate text according to each target score. 15.The electronic device of claim 13, wherein the processor is specifically configured to execute the computer instructions to cause the electronic device to perform: determining a first preset number of first candidate texts according to each similarity; determining a second preset number of second candidate texts according to each relevance score; performing a deduplication process on the first candidate texts and the second candidate texts to obtain the target candidate text.

16. The electronic device of claim 10, after the processor is configured to execute the computer instructions to cause the electronic device to perform: searching, in a pre-collected error accumulation knowledge base, for a target question and target reply information that match the standard question; determining candidate reply information corresponding to the target question in the error accumulation knowledge base and the target reply information as third candidate text; and performing deduplication processing on the target candidate text and the third candidate text to obtain fourth candidate text, and using the fourth candidate text to update the target candidate text, before the processor is configured to execute the computer instructions to cause the electronic device to perform: text splicing the target candidate text, the standard question, and prompt text.

17. The electronic device of claim 10, if the target candidate text is multiple, after the processor is configured to execute the computer instructions to cause the electronic device to perform: determining a similarity score between the standard question and each of the target candidate text; and screening the target candidate text according to the similarity score, before the processor is configured to execute the computer instructions to cause the electronic device to perform: text splicing the target candidate text, the standard question, and prompt text.

18. The electronic device of claim 10, the processor is further configured to execute the computer instructions to cause the electronic device to perform: determining, based on a seventh model, whether the reply information meets a quality requirement, if yes, determining that the reply information passes the verification and outputting; if not, determining that the reply information fails the verification, and performing manual evaluation; and the quality requirement includes at least one of the following: the standard question matches the reply information, the reply information is from the target candidate text, and the words of the reply information are appropriate.

19. The electronic device of claim 1, the initial embedding model includes an encoder, the encoder includes a plurality of encoding sub-layers, and the processor is further configured to execute the computer instructions to cause the electronic device to perform: obtaining a pre-saved training set for a specific field, the training set including sample text; inputting the sample text into the encoder for encoding to obtain a first feature vector output by each encoding sub-layer of the encoder; determining a predicted feature vector of the sample text according to a weight corresponding to each encoding sub-layer and the first feature vector; and determining a first loss value according to the predicted feature vector and a standard feature vector of the sample text, and training the initial embedding model according to the first loss value to obtain an embedding model.

20. The electronic device of claim 19, the initial embedding model further includes a decoder, after the processor is configured to execute the computer instructions to cause the electronic device to perform: determining the predicted feature vector of the sample text, before the processor is configured to execute the computer instructions to cause the electronic device to perform: training the initial embedding model according to the first loss value to obtain an embedding model, inputting the predicted feature vector into the decoder for decoding to obtain predicted text output by the decoder; ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ determine a second loss value according to the predicted text and the sample text; In the training of the initial embedding model according to the first loss value to obtain the embedding model, the processor is specifically configured to execute the computer instructions to enable the electronic device to perform: training the initial embedding model according to the first loss value and the second loss value to obtain the embedding model.

21. The electronic device of claim 19, wherein the processor is further configured to execute the computer instructions to enable the electronic device to perform: obtaining any sample text in the training set as a first fine-tuning text, a first similar text corresponding to the first fine-tuning text, and a first dissimilar text corresponding to the first fine-tuning text; inputting the first fine-tuning text, the first similar text, and the first dissimilar text into the embedding model respectively to obtain each second feature vector; determining a third loss value according to a fourth similarity between the second feature vectors corresponding to the first fine-tuning text and the first similar text, and a fifth similarity between the second feature vectors corresponding to the first fine-tuning sample and the first dissimilar text; fine-tuning training the embedding model according to the third loss value.

22. The electronic device of claim 19, wherein the processor is further configured to execute the computer instructions to enable the electronic device to perform: obtaining any sample in the training set as a second fine-tuning text, a second similar text corresponding to the second fine-tuning sample, and a second dissimilar text corresponding to the second fine-tuning text; inputting the second fine-tuning text, the second similar text, and the second dissimilar text into the embedding model respectively to obtain each third feature vector; determining a sixth similarity of the third feature vectors between the second fine-tuning sample and the second similar text, and a seventh similarity of the third feature vectors between the second fine-tuning sample and the second dissimilar text; determining a fourth loss value according to a difference between the sixth similarity and the seventh similarity, and an activation function; fine-tuning training the embedding model according to the fourth loss value.

23. The electronic device of claim 21 or 22, wherein the determination process of the similar text corresponding to any fine-tuning text comprises: obtaining a similar text pair generated based on a large model, and determining a target text pair containing the fine-tuning text; determining a fourth feature vector of each text of the target text pair based on a pre-trained other embedding model; determining whether the non-fine-tuning text in the target text pair is the similar text of the fine-tuning text according to an eighth similarity between each fourth feature vector and a preset threshold.

24. The electronic device of claim 23, wherein after the electronic device determines the target text pair containing the fine-tuning text, and before the electronic device determines the fourth feature vector of each text of the target text pair based on the pre-trained other embedding model, the processor is further configured to execute the computer instructions to enable the electronic device to perform: The eighth model is used to determine whether the texts in the target text pair are similar, and if so, subsequent steps of determining each fourth feature vector of the target text pair based on other pre-trained embedding models are performed.

25. The electronic device of claim 23, wherein the determination of the similar text pair generated based on the large model comprises: generating, based on the large model, associated texts corresponding to the obtained text segments, the associated texts including at least one of a question of the text segment, a summary text of the text segment, and a reply information corresponding to the question of the text segment; randomly selecting any two texts from the text segments and the associated texts as the similar text pair.

26. The electronic device of claim 1, further comprising a display. The processor is further configured to execute the computer instructions to cause the electronic device to perform: if a troubleshooting initiation request is received, displaying, on the display, a list of obtained locations to be investigated; if an investigation instruction for investigating any target location to be investigated in the list of locations to be investigated is received, obtaining an on-site investigation record, the on-site investigation record including at least one of an image, an audio, and a text; based on each investigation question, analyzing content included in the on-site investigation record to obtain an investigation result corresponding to each investigation question, and displaying the investigation result on the display.

27. The electronic device of claim 26, wherein the processor is further configured to execute the computer instructions to cause the electronic device to perform: if the on-site investigation record is an audio, performing speech recognition on the audio to obtain recognized text corresponding to the audio, and displaying the recognized text on the display; if a text confirmation instruction is received, updating the on-site investigation record using the recognized text.

28. The electronic device of claim 26, wherein the processor is specifically configured to execute the computer instructions to cause the electronic device to perform: if the on-site investigation record is an image, identifying, according to each investigation object corresponding to each investigation question, a target object corresponding to each investigation object in the image, and whether the target object has an abnormality; based on a large model, analyzing whether the target object has an abnormality and each investigation question to determine an investigation result corresponding to each investigation question.

29. The electronic device of claim 26 or 27, wherein the processor is specifically configured to execute the computer instructions to cause the electronic device to perform: if the on-site investigation record is a text, inputting the on-site investigation record, each investigation question, and a prompt text into the large model, causing the large model to analyze the on-site investigation record and each investigation question based on a prompt of the prompt text to determine an investigation result corresponding to each investigation question.

30. The electronic device of claim 26, wherein the processor is specifically configured to execute the computer instructions to cause the electronic device to perform: ​ If the on-site investigation record is multiple images of the same target object, determine the overlapping area in all images according to the content included in each image; Generate a complete target object image of the target object according to the target object contained in all the overlapping areas; Determine whether the target object in the complete target object image is abnormal based on a large model, and determine the investigation result of the target object corresponding to the problem to be investigated.

31. The electronic device of claim 26, wherein the processor is specifically configured to execute the computer instructions to cause the electronic device to perform: If the on-site investigation record is multiple images of the same target object, for each image, analyze the image and each problem to be investigated based on a large model to determine the candidate investigation result of the target object in the image corresponding to each problem to be investigated; For each problem to be investigated, fuse the candidate investigation result of each image corresponding to the problem to be investigated to obtain the investigation result corresponding to the problem to be investigated.

32. The electronic device of claim 26, wherein the processor is further configured to execute the computer instructions to cause the electronic device to perform: Obtain the first location information of the road hazard investigation terminal at present, and the second location information of the investigation site corresponding to the problem to be investigated for which the investigation result is not obtained; According to the first location information, the second location information and the distance threshold, determine the target supplementary investigation site within the distance threshold and display the target supplementary investigation site in the display, so that the user of the electronic device goes to the target supplementary investigation site to investigate the problem to be investigated for which the investigation result is not obtained.

33. The electronic device of claim 32, wherein the processor is further configured to execute the computer instructions to cause the electronic device to perform: Determine the current moving speed of the road hazard investigation terminal; According to the correspondence between the different speeds and the time delay values preserved in advance, determine the target time delay value corresponding to the moving speed; According to the current time and the target time delay value, determine the supplementary investigation site that is moving away, and delete the supplementary investigation site that is moving away in the target supplementary investigation site.

34. A large model-based question and answer method, the method comprising: According to each text block preserved in advance, determine the target text block matched with the question to be replied; If non-text data is previously saved for the target text block, obtain the summary text of the saved non-text data, the non-text data including at least one of pictures and tables, and the summary text being a text summarizing all the contents in the non-text data; Input the question to be replied, the target text block and the summary text into a first large model according to a preset format to obtain reply information. ​

Images