Sample construction method, relevancy model training method and question and answer enhancement processing method
By using semantic similarity matching and large language models to generate relevance scores, the problem of long sample construction time and low accuracy under manual scoring methods is solved, achieving more efficient and accurate sample construction and improved question-answering model answer quality.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING QIYI CENTURY SCI & TECH CO LTD
- Filing Date
- 2026-01-04
- Publication Date
- 2026-04-21
AI Technical Summary
In existing technologies, the manual scoring method for building question-answering models and training samples requires a lot of manpower and time, and the scoring results are easily affected by the level of expertise, resulting in low accuracy.
By obtaining query information samples and performing semantic similarity matching with content information in the knowledge base, model prompt words are generated. A large language model is used to determine the relevance score, and sample pairs are constructed. The relevance model is then used to rank and re-rank the candidate content information.
It reduced sample construction time, improved the accuracy of relevance scores, and enhanced the answer quality of the question-answering model.
Smart Images
Figure CN121901733A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of data processing technology, and in particular to a sample construction method, a relevance model training method, and a question-answering enhancement processing method. Background Technology
[0002] With the continuous development of data processing technology, retrieval-enhanced question answering models are widely used in various question answering scenarios.
[0003] In a knowledge-based question-answering scenario, upon receiving a user's query, the system first identifies matching knowledge from the knowledge base. This matching knowledge is then input into the question-answering model to generate the corresponding answer. To improve the quality of answers in knowledge-based question-answering scenarios, a pre-trained scoring model can be used to score the retrieved knowledge before inputting it into the question-answering model. Based on the scoring results, knowledge with higher scores is selected to generate the answer to the query.
[0004] Currently, training samples for the model are usually constructed by manual scoring, and the retrieved knowledge is scored based on the trained model. However, this manual scoring method not only requires a lot of manpower and time, but the scoring results are also easily affected by the professional level of the scorers, resulting in low accuracy. Summary of the Invention
[0005] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a sample construction method, a relevance model training method, and a question-answering enhancement processing method.
[0006] This disclosure provides a sample construction method, the method comprising: Obtain a sample of the query information; By performing semantic similarity matching between the query information sample and the content information in the knowledge base, the content information corresponding to the query information sample is determined and used as the content information sample corresponding to the query information sample. Based on the scoring criteria, corresponding query information samples and content information samples, model prompt words are generated. The model prompt words are input into the large language model. The model prompt words are used to drive the large language model to determine the relevance score between the query information sample and the content information sample according to the scoring criteria and the semantic similarity between the content information sample and the query information sample. The model input consists of query information samples and content information samples with corresponding relationships, and the corresponding relevance scores are used as the true values to construct sample pairs.
[0007] In some embodiments, obtaining the query information sample includes: The pre-collected user conversation content is input into the intent recognition model, which then extracts plot-related query information from the user conversation content. A query information sample is constructed based on the aforementioned plot-related query information.
[0008] In some embodiments, constructing a query information sample based on the plot-related query information includes: The plot-related query information is matched with entity objects in the target entity library. If the match is successful, the plot-related query information is determined as a query information sample. The entity objects in the target entity library are obtained by entity recognition of the content information in the knowledge base.
[0009] In some embodiments, the query information sample includes at least one of a basic plot query information sample, a plot comparison query information sample, and a plot location query information sample. The basic plot query information sample is used to query at least one of the plot introduction, plot characters, and plot evaluation of a video. The plot comparison query information sample is used to query the comparison between a target video and other videos. The plot location query information sample is used to query the video or video segment where the plot description text is located.
[0010] In some embodiments, the knowledge base includes a plot knowledge base, which includes plot information, and the step of obtaining query information samples includes: Extract plot descriptions from the plot information in the plot knowledge base based on the plot synopsis; The plot description is filled into a preset question template to construct a plot location query information sample.
[0011] This disclosure also provides a method for training a relevance model for ranking, the method comprising: Multiple sample pairs are obtained; wherein, the sample pair includes a query information sample, a content information sample and a relevance score that have a corresponding relationship, the relevance score is used to characterize the semantic relevance between the content information sample and the query information sample, and the sample pair is constructed using the above sample construction method; The query information sample and content information sample from the sample pair are input into the model, and the model outputs the predicted relevance score of the query information sample and the content information sample. The predicted relevance score is verified using the relevance score in the sample pair. The parameters of the model are adjusted according to the verification result until the model reaches the preset convergence condition. The model is then used as the relevance model.
[0012] This disclosure also provides a question-answering enhancement processing method, the method comprising: Receive content query information, and match multiple candidate content information from the knowledge base based on semantic similarity; The content query information and the multiple candidate content information are input into a relevance model. The relevance model determines the relevance score between the content query information and each candidate content information, and sorts the candidate content information from largest to smallest according to the relevance score. The relevance score is used to characterize the semantic relevance between the corresponding candidate content information and the content query information. The relevance model is trained using the relevance model training method described above for sorting. Based on the information of the top N candidate contents and the content query information, model prompt words are generated and input into the question answering model. The model prompt words drive the question answering model to generate the answer content corresponding to the content query information; N is a preset natural number.
[0013] In some embodiments, the content query information includes plot query information, and the knowledge base is a plot knowledge base, which includes plot information.
[0014] This disclosure also provides a sample construction apparatus, which includes: The first acquisition module is used to acquire a sample of query information. The first determining module is used to determine the content information corresponding to the query information sample by performing semantic similarity matching between the query information sample and the content information in the knowledge base, and use it as the content information sample corresponding to the query information sample. The second determining module is used to generate model prompt words based on the scoring criteria, corresponding query information samples and content information samples, input the model prompt words into the large language model, and use the model prompt words to drive the large language model to determine the relevance score between the query information sample and the content information sample according to the scoring criteria and the semantic similarity between the content information sample and the query information sample; The construction module is used to construct sample pairs by taking corresponding query information samples and content information samples as model inputs and the corresponding relevance scores as real values.
[0015] This disclosure also provides a relevance model training apparatus for ranking, the apparatus comprising: The second acquisition module is used to acquire multiple sample pairs; wherein, the sample pair includes a query information sample, a content information sample and a relevance score that have a corresponding relationship, the relevance score is used to characterize the semantic relevance between the content information sample and the query information sample, and the sample pair is constructed using the above-mentioned sample construction method; The training module is used to input the query information samples and content information samples from the sample pairs into the model, and the model outputs the predicted relevance scores of the query information samples and the content information samples. The predicted relevance scores are verified using the relevance scores in the sample pairs. The parameters of the model are adjusted according to the verification results until the model reaches the preset convergence condition, and the model is then used as the relevance model.
[0016] This disclosure also provides a question-answering enhancement processing apparatus, which includes: Receive content query information, and match multiple candidate content information from the knowledge base based on semantic similarity; A receiving module is configured to input the content query information and the plurality of candidate content information into a relevance model, wherein the relevance model determines the relevance score between the content query information and each candidate content information, and sorts the candidate content information in descending order of the relevance score; wherein the relevance score is used to characterize the semantic relevance between the corresponding candidate content information and the content query information, and the relevance model is trained using the aforementioned relevance model training method; The sorting module is used to generate model prompts based on the information of the top N candidate contents and the content query information, and input the model prompts into the question-answering model. The model prompts drive the question-answering model to generate the answer content corresponding to the content query information; N is a preset natural number.
[0017] This disclosure also provides an electronic device, which includes: Processor and memory; The processor executes the method described in any embodiment of this disclosure by calling programs or instructions stored in the memory.
[0018] This disclosure also provides a computer-readable storage medium that stores a program or instructions that cause a computer to perform the methods described in any embodiment of this disclosure.
[0019] The sample construction method provided in this embodiment can generate a relatively accurate relevance score using a large language model, and construct sample pairs using query information samples, content information samples and relevance scores with corresponding relationships. Compared with the method of manually constructing sample pairs, it can reduce the sample construction time and improve the accuracy of the relevance score. Attached Figure Description
[0020] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 A schematic flowchart illustrating a sample construction method provided in an embodiment of this disclosure; Figure 2 A schematic diagram illustrating the process of constructing a sample pair according to an embodiment of this disclosure; Figure 3 A flowchart illustrating a relevance model training method for ranking, provided in an embodiment of this disclosure; Figure 4 A flowchart of a question-answering enhancement processing method provided in this embodiment of the disclosure; Figure 5 This is a schematic diagram of a sample construction apparatus provided in an embodiment of the present disclosure; Figure 6 This is a schematic diagram of a relevance model training device for ranking provided in an embodiment of the present disclosure; Figure 7 A schematic diagram of the structure of a question-answering enhancement processing device provided in an embodiment of this disclosure; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation
[0023] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be described in further detail below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.
[0024] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.
[0025] In the context of knowledge-based question answering, when a query is received from a user, the system first identifies the knowledge that matches the query from the knowledge base, and then inputs the knowledge into the question answering model to generate the answer content corresponding to the query.
[0026] In related technologies, in order to improve the quality of answers in knowledge-based question-answering scenarios, a pre-trained scoring model can be used to score the retrieved knowledge before inputting it into the question-answering model, so as to select the knowledge with higher scores to generate the answer content corresponding to the query information based on the scoring results.
[0027] Currently, training samples for the model are usually constructed by manual scoring, and the retrieved knowledge is scored based on the trained model. However, this manual scoring method not only requires a lot of manpower and time, but the scoring results are also easily affected by the professional level of the scorers, resulting in low accuracy.
[0028] Based on this, this disclosure provides a sample construction method. First, a query information sample is obtained. Then, by performing semantic similarity matching between the query information sample and the content information in the knowledge base, the corresponding content information is determined and used as the content information sample corresponding to the query information sample. Next, model prompt words are generated based on the scoring criteria, the corresponding query information samples and content information samples, and the model prompt words are input into a large language model. The model prompt words drive the large language model to determine the relevance score between the query information sample and the content information sample based on the scoring criteria and the semantic similarity between the content information sample and the query information sample. Further, the corresponding query information samples and content information samples are used as model inputs, and the corresponding relevance scores are used as real values to construct sample pairs.
[0029] As can be seen, the embodiments of this disclosure can generate more accurate relevance scores using a large language model, and construct sample pairs using query information samples, content information samples and relevance scores with corresponding relationships. Compared with the method of manually constructing sample pairs, this can reduce the sample construction time and improve the accuracy of relevance scores.
[0030] Figure 1 This is a flowchart of a sample construction method provided in an embodiment of this disclosure. See also... Figure 1 The sample construction method includes: S110. Obtain a sample of the query information.
[0031] The query information samples include query information extracted from user session content, such as the main plot of TV series C, and in which episode the male and female leads first meet.
[0032] In some embodiments, the query information sample can be constructed based on plot-related query information. Specifically, the query information sample may include at least one of the following: basic plot query information sample, plot comparison query information sample, and plot location query information sample.
[0033] The basic plot query information samples are used to query at least one of the following: plot summary, characters, and plot evaluation. For example, what is the content of a drama? What is the growth experience of the male lead? Is the plot development reasonable? The plot comparison query information samples are used to query the comparison between a target video and other videos. For example, which drama, A or B, has a more reasonable plot development? The plot location query information samples are used to query the video or video clip containing the plot description text. For example, in which episode do the male and female leads meet?
[0034] In some embodiments, since there are many conversations unrelated to the plot, such as casual chat, in the pre-collected user conversation content, after obtaining a large amount of user conversation content from online records, the user conversation content can be input into a trained intent recognition model, which will then extract plot-related query information from the user conversation content and construct query information samples based on the plot-related query information.
[0035] Plot-related search information may include at least one of the following: basic plot search information, plot comparison search information, and plot location search information.
[0036] Intent recognition models are used to extract a user's true purpose or intent from user session content. The output of intent recognition models includes intent tags and related slot information. For example, for the user session content "Are there any good Sichuan restaurants nearby?", after processing by the intent recognition model, the output will be intent tags such as "restaurant search" and information such as "nearby" and "Sichuan cuisine".
[0037] As can be seen, the embodiments of this disclosure can use an intent recognition model to extract plot-related query information from user session content, so that the plot-related query information can be used to construct query information samples in the future.
[0038] In other embodiments, after extracting plot-related query information from user session content, entity matching can be used to filter out plot-related query information that cannot be retrieved from the knowledge base.
[0039] Specifically, the plot-related query information is matched with each entity object in the target entity library. If the plot-related query information is found to match any entity object in the target entity library, then the plot-related query information is determined as a query information sample.
[0040] The entity objects stored in the target entity library are obtained by entity recognition of the content information in the knowledge base. The entity objects may include video names, character names, actor names, and other objects related to the plot.
[0041] In practical applications, if it is determined that the plot-related query information does not match any entity object in the target entity library, it means that the plot-related query information cannot retrieve the corresponding plot information in the plot knowledge base. At this time, the plot-related query information is filtered out, and only the plot-related query information that matches any entity object in the target entity library is retained.
[0042] In this embodiment of the disclosure, query information unrelated to the plot is filtered out by matching each plot-related query with entity objects in the target entity library.
[0043] S120. By performing semantic similarity matching between the query information sample and the content information in the knowledge base, the content information corresponding to the query information sample is determined and used as the content information sample corresponding to the query information sample.
[0044] A knowledge base is a repository that uses vector or indexing techniques to store content information. It stores feature vectors of the content information. Vectorization refers to the process of converting unstructured text content (such as plot information) into numerical vectors. Subsequently, when receiving content query information input by the user, the corresponding content information can be determined from the knowledge base through similarity matching.
[0045] Content information samples can be content information retrieved from a knowledge base based on semantic similarity matching that can be used to answer query information samples.
[0046] In some embodiments, after obtaining the query information sample, the content information sample corresponding to the query information sample can be determined by performing similarity matching between the feature vector of the query information sample and the feature vector of the content information in the vector knowledge base.
[0047] Specifically, the cosine similarity between the feature vectors of the query information sample and the feature vectors of the content information sample can be calculated, and the content information with a similarity greater than a preset threshold can be identified as the content information sample corresponding to the query information sample. The preset threshold can be set to 0.5, meaning that results with a pre-defined similarity greater than 0.5 are retrieved from the vector knowledge base as content information samples.
[0048] S130. Based on the scoring criteria, corresponding query information samples and content information samples, generate model prompt words, input the model prompt words into the large language model, and use the model prompt words to drive the large language model to determine the relevance score between the query information sample and the content information sample according to the scoring criteria and the semantic similarity between the content information sample and the query information sample.
[0049] The scoring criteria are used to determine the relevance between the query information sample and the content information sample. The query information sample is the user question, and the content information sample is the knowledge determined from the knowledge base. The higher the relevance between the user question and the knowledge, the more useful the retrieved knowledge is in answering the user question.
[0050] For example, model suggestion words generated based on scoring representations, corresponding query information samples, and content information samples can be: {Sample query information}; {Content information in the knowledge base}; The reference list contains relevant information from your knowledge base that can be used to answer user questions. Please rate the usefulness of each item in the reference list to your answer to the query, on a scale of 0 to 1, according to the following criteria: The content information can be directly used to answer the query, and the corresponding relevance score is 1. The content information is of no help to the query, and the corresponding score is 0. The content information cannot be used to directly answer the query, but it is helpful in answering the query. Please rate it according to its contribution to answering the query.
[0051] In this embodiment of the disclosure, after generating model prompt words, the model prompt words are input into the large language model. The model prompt words are used to drive the large language model to determine the relevance score between the query information sample and the corresponding content information sample based on the scoring criteria and the semantic similarity between the content information sample and the query information sample.
[0052] The relevance score is used to characterize the degree of contribution of the content information sample to the response of the query information sample. The relevance score can range from [0,1]. The higher the relevance score, the greater the contribution of the content information sample to the response of the query information sample; the lower the relevance score, the lower the contribution of the content information sample to the response of the query information sample.
[0053] In practical applications, professionals can also determine the degree of contribution of the content information sample to the response content of the query information sample in order to obtain a more accurate relevance score.
[0054] S140. Use the query information samples and content information samples with corresponding relationships as model inputs, and construct sample pairs using the corresponding relevance scores as true values.
[0055] In this embodiment of the disclosure, after obtaining the query information sample, the corresponding content information sample, and the relevance score between the query information sample and the content information sample, sample pairs are constructed based on the corresponding query information sample and content information sample, and the relevance score between the query information sample and the content information sample. That is, each sample pair consists of three fields: query information sample, content information sample, and relevance score.
[0056] After constructing multiple sample pairs, the model is trained by taking query information samples and content information samples as inputs and the relevance scores of query information samples and content information samples as real values, resulting in a re-ranking model for outputting the relevance between knowledge and questions.
[0057] In the sample construction method provided in this embodiment, a query information sample is first obtained. Then, the content information corresponding to the query information sample is determined by semantic similarity matching between the query information sample and the content information in the knowledge base, and is used as the content information sample corresponding to the query information sample. Next, model prompt words are generated based on the scoring criteria, the corresponding query information samples and content information samples, and the model prompt words are input into the large language model. The model prompt words drive the large language model to determine the relevance score between the query information sample and the content information sample according to the scoring criteria and the semantic similarity between the content information sample and the query information sample. Further, the corresponding query information samples and content information samples are used as model inputs, and the corresponding relevance scores are used as real values to construct sample pairs.
[0058] After obtaining query information samples and their corresponding content information samples, this embodiment of the present disclosure can generate model suggestion words using scoring criteria and corresponding query information samples and content information samples. Based on these model suggestion words, it drives a large language model to generate relevance scores between the query information samples and the content information samples. Therefore, this embodiment of the present disclosure can generate relatively accurate relevance scores using a large language model and construct sample pairs based on query information samples, content information samples, and relevance scores. Compared to manually constructing sample pairs, this reduces sample construction time and improves the accuracy of relevance scores.
[0059] In some embodiments, in order to enrich the query information samples and avoid the question-answering model only being able to answer questions that users have asked but not being able to handle questions that users may ask, the embodiments of this disclosure may also adopt a reverse generation method to enrich the plot positioning query information samples in the query information samples.
[0060] In this embodiment of the disclosure, the knowledge base may further include a plot knowledge base, which is a knowledge base that uses vector or indexing technology to store plot information. The plot knowledge base is used to store plot information, which is information used to describe the plot content of the video, such as the plot summary of a TV series, the characters' experiences, and the evaluation information of a movie.
[0061] In this embodiment of the disclosure, the plot description content is first extracted from the plot information in the plot knowledge base according to the plot synopsis, and then the plot description content is filled into a preset question template to construct a plot location query information sample.
[0062] The plot description refers to a summary of key plot information, such as the description of the male and female leads' first date. Pre-set question templates could include: "In which episode does the plot description appear?", "In which drama is the plot description in?", "In which time slot of which drama is the plot description in?", etc.
[0063] As can be seen, this embodiment of the present disclosure fills the plot description content extracted from the plot knowledge base based on the plot synopsis into the preset question template to construct a plot location query information sample, thereby enriching the query information sample, solving the problem of scarce training data, and improving the accuracy of relevance score calculation.
[0064] In other embodiments, to further enrich the query information samples, a query information generation model can be invoked to generate plot query information related to the basic information of the video, and a basic plot query information sample can be constructed based on the plot query information related to the basic information of the video as a query information sample.
[0065] The plot query information related to the basic information of the video can include basic information such as plot summary, characters, plot reviews, and actors. The query information generation model can include, for example, a networked big data language model. This model simulates user questions to construct basic plot query information samples, with corresponding prompts such as: "Regarding {movie / TV show name}, give me some questions that viewers have asked online related to its basic information."
[0066] In other embodiments, a query information generation model can be invoked to generate comparison query information between the target video and other videos, and a plot comparison query information sample can be constructed based on the comparison query information as a query information sample.
[0067] For example, when searching for comparison information between a target video and other videos, the corresponding prompts could be: "Regarding {TV show title}, give me some questions that viewers have asked online comparing it with other similar shows."
[0068] As can be seen, the embodiments of this disclosure improve the diversity and coverage of the query information samples through the above-described method, and further improve the accuracy of the relevance model in calculating the relevance score.
[0069] Figure 2 This is a schematic diagram illustrating the process of constructing a sample pair according to an embodiment of the present disclosure.
[0070] First, the query information sample can be constructed using the following method: The user session content is input into the trained intent recognition model, which extracts plot-related query information from the user session content and outputs the plot-related query information. Then, the plot-related query information is matched with entity objects in the target entity library. When it is determined that the plot-related query information matches the entity objects in the target entity library, the plot-related query information is determined as the query information sample.
[0071] The plot location query information sample can be constructed by reverse engineering. Specifically, firstly, the plot description content is extracted from the plot information in the plot knowledge base based on the plot synopsis, and then the plot description content is filled into the preset question template to obtain the plot location query information sample.
[0072] For the basic plot query information samples and plot comparison query information samples, a networked large language model can be used to simulate user questions to obtain basic plot query information samples and plot comparison query information samples. The final query information samples can include basic plot query information samples, plot comparison query information samples, and plot location query information samples.
[0073] After constructing a query information sample containing basic plot query information samples, plot comparison query information samples, and plot location query information samples, the plot information sample corresponding to the query information sample is determined by semantic similarity matching between the query information sample and the plot information in the plot knowledge base.
[0074] Then, based on the scoring criteria, corresponding query information samples, and plot information samples, model prompt words are generated. These prompt words are then input into the large language model, which uses them to drive the model to determine the relevance score between the query information samples and the content information samples based on the semantic similarity between the scoring representation and the content information samples.
[0075] Finally, the corresponding query information samples and content information samples are used as model inputs, and the corresponding relevance scores are used as true values to construct sample pairs.
[0076] Figure 3 This is a flowchart illustrating a relevance model training method for ranking provided in this disclosure. See also... Figure 3The training method for this relevance model specifically includes: S310. Obtain multiple sample pairs.
[0077] The sample pairs include query information samples, content information samples, and relevance scores that have a corresponding relationship. The relevance scores are used to characterize the semantic relevance between the content information samples and the query information samples.
[0078] The query information samples can include plot-related query information extracted from user conversations collected from online records. Query information samples and content information samples within the same sample pair have a corresponding relationship. For example, if the query information sample in sample pair 1 is "What are the main plot points of TV series C?", then its corresponding content information sample could be a plot summary of TV series C. Similarly, if the query information sample in sample pair 2 is "What are the main plot points of TV series C?", then its corresponding content information sample could be the experiences of the male protagonist in TV series C, etc.
[0079] The relevance score is used to characterize the semantic relevance between the content information sample and the query information sample. The higher the relevance score, the closer the semantics of the content information sample and the query information sample are, meaning that the content information sample is essential reference information when answering the query information sample. For example, if a user asks about a character's information using an artist's name, the correspondence between the actor and the character is essential; if a user asks about the experiences of character A's husband, the information about who character A's husband is is essential.
[0080] S320. Input the query information sample and content information sample from the sample pair into the model. The model outputs the predicted relevance scores of the query information sample and content information sample. The predicted relevance scores are verified using the relevance scores in the sample pair. The model parameters are adjusted according to the verification results until the model reaches the preset convergence condition. The model is then used as the relevance model.
[0081] The predicted relevance score is the semantic relevance prediction result of the base model to be trained, based on the semantic relevance between the query information sample and the content information sample, after processing by the model's own algorithm. The predicted relevance score may have a large deviation from the true value (i.e., the relevance score in the sample pair) in the early stage of training.
[0082] The preset convergence condition is a quantitative standard for determining whether the model training is complete. It is a pre-set target. When the model reaches the preset convergence condition, it means that the model's predictive ability has reached a stable level and meets the standard, and there is no need to continue adjusting the parameters.
[0083] In this embodiment of the disclosure, a sample pair containing query information samples, content information samples, and relevance scores is input into the model to be trained, driving the model to output the predicted relevance scores of the query information samples and content information samples. Then, the actual relevance scores in the sample pairs are used to compare with the model's predicted relevance scores to determine the accuracy of the predicted relevance scores. Based on the verification results between the actual relevance scores and the predicted relevance scores, the internal parameters of the model are adjusted to reduce the prediction bias of the model.
[0084] Repeat the above steps of input, prediction, verification, and parameter tuning until the model's prediction results meet the preset convergence conditions, such as the prediction accuracy reaching the target or the error stabilizing within the preset threshold. At this point, the model can be used as the final relevance model.
[0085] The model to be trained may include a fine-ranking model implemented with a cross-encoder architecture. Its internal modules follow the Transformer encoder paradigm and include the following structure: input layer and encoding format, Transformer backbone network, output layer and score calibration, with a total of 279M parameters, a maximum input length of 512 tokens, a loss function of binary cross-entropy, and an output activation function of Sigmoid.
[0086] In practical applications, after training the relevance model using the above method, the relevance model is used to determine the target plot information from multiple candidate plot information for the plot query information input by the user, so as to determine the answer content corresponding to the plot query information based on the target plot information.
[0087] In the relevance model training method for ranking provided in this embodiment, multiple sample pairs are first obtained. Each sample pair includes a query information sample, a content information sample, and a relevance score that have a corresponding relationship. The relevance score is used to characterize the semantic relevance between the content information sample and the query information sample. Then, the query information sample and the content information sample in the sample pair are input into the model. The model outputs the predicted relevance scores of the query information sample and the content information sample. The predicted relevance scores are verified using the actual relevance scores in the sample pair. The parameters of the model are adjusted according to the verification results until the model reaches a preset convergence adjustment. Then, the model is used as a relevance model.
[0088] As can be seen, since the relevance model in this embodiment is trained based on the corresponding query information samples and content information samples, as well as the relevance score between the query information samples and content information samples, in subsequent applications, it is possible to reorder each candidate content information according to the semantic similarity between the query information and the candidate content information, so as to improve the recall rate of useful knowledge in the knowledge question answering scenario.
[0089] In a knowledge-based question-answering scenario, upon receiving a user's query, the system first identifies matching knowledge from the knowledge base. This matching knowledge is then input into the question-answering model to generate the corresponding answer. In related technologies, to improve the quality of answers in plot-based knowledge-based question-answering scenarios, the retrieved knowledge can be reordered based on semantic similarity between the query and the existing knowledge before being input into the question-answering model.
[0090] However, since the plot knowledge base stores plot descriptions of videos, it is difficult to make an accurate ranking of them based solely on semantic similarity, which affects the response quality of the question answering model.
[0091] Based on this, the present disclosure provides a question-answering enhancement processing method. Figure 4 This is a flowchart of a question-answering enhancement processing method provided in an embodiment of this disclosure. See also... Figure 4 The question-answering enhancement processing method specifically includes: S410. Receive content query information and match multiple candidate content information from the knowledge base based on semantic similarity.
[0092] Among them, content query information refers to the natural language text entered by the user to query information, such as the text content "What is the main content of TV series A?" After receiving the plot query information entered by the user, the intelligent assistant in video software will generate corresponding answer content based on the plot query information.
[0093] Specifically, upon receiving content query information, the content query information is first vectorized to extract its feature vector. Then, the feature vector of the content query information is matched with the feature vector of the content information in the knowledge base to determine multiple candidate content information corresponding to the content query information.
[0094] In some embodiments, the content query information can also be input into a vector generation model. After processing by the vector generation model, a feature vector of the content query information is obtained. After determining the feature vector of the content query information, the content query information is retrieved through vector similarity matching.
[0095] In other embodiments, during the vector similarity matching process, feature vectors with a similarity of not less than a preset threshold to the feature vectors of the content query information can be determined from the knowledge base, and the content information corresponding to the feature vectors can be determined as multiple candidate content information corresponding to the content query information.
[0096] As can be seen, when receiving content query information, this embodiment of the present disclosure can determine multiple candidate content information for the content query information by searching the knowledge base, so that the corresponding answer content can be determined for the content query information based on the multiple candidate content information.
[0097] S420. Input the content query information and multiple candidate content information into the relevance model. The relevance model determines the relevance score between the content query information and each candidate content information, and sorts the candidate content information from largest to smallest according to the relevance score.
[0098] The relevance score is used to characterize the semantic relevance between the corresponding candidate content information and the content query information. The higher the relevance score, the greater the contribution of the corresponding candidate content information to the answer content of the content query information. For example, when the relevance score is 1, it means that the candidate content information can be directly used to generate the answer content of the content query information. The lower the relevance score, the lower the contribution of the corresponding candidate content information to the answer content of the content query information, that is, the lower the semantic relevance between the candidate content information and the content query information. For example, when the relevance is 0, it means that the candidate content information is of no help in generating the answer content of the content query information.
[0099] The relevance model is trained based on pre-acquired sample pairs, which include query information samples, plot information samples, and relevance scores between query information samples and plot information samples.
[0100] The query information samples can include at least one of the following: basic plot query information samples, plot comparison query information samples, and plot location query information samples. Basic plot query information samples are used to query plot information such as video synopsis, characters, and reviews. Plot comparison query information samples are used to query comparisons between a target video and other videos. Plot location query information samples are used to query the video or video segment containing the plot description text.
[0101] Since the relevance model in this embodiment is trained based on query information samples, content information samples, and the relevance score between the two, the relevance score determined by the relevance model can accurately rank multiple candidate content information from the dimension of semantic similarity between candidate content information and content query information.
[0102] S430. Generate model prompts based on the information of the top N candidate contents and the content query information, and input the model prompts into the question answering model. The model prompts drive the question answering model to generate the answer content corresponding to the content query information.
[0103] Where N is a preset natural number. For example, when N is 10, after sorting the matched candidate content information from largest to smallest according to the relevance score, the top 10 candidate content information and the content query information are used to generate prompt words.
[0104] In this embodiment of the disclosure, after determining the relevance score between the content query information and each candidate content information using a relevance model and outputting the relevance score corresponding to each candidate content information, the candidate content information is reordered according to the relevance score. Prompt words are generated using the top 10 candidate content information and the plot query information after sorting, and the question-answering model is driven to generate the answer content corresponding to the content query information.
[0105] As can be seen, since the relevance model in this embodiment is trained based on query information samples, plot information samples, and the relevance score between the two, the relevance score determined by the relevance model can accurately rank multiple candidate content information from the semantic similarity dimension between candidate content information and content query information. This makes the candidate content information input to the question answering model highly relevant to the content query information, thereby improving the response quality of the question answering model.
[0106] Figure 5 This is a schematic diagram of a sample construction apparatus provided in an embodiment of this disclosure. Figure 5 As shown, the sample construction apparatus 500 includes: The first acquisition module 501 is used to acquire a sample of query information; The first determining module 502 is used to determine the content information corresponding to the query information sample by performing semantic similarity matching between the query information sample and the content information in the knowledge base, and use it as the content information sample corresponding to the query information sample. The second determining module 503 is used to generate model prompt words based on the scoring criteria, corresponding query information samples and content information samples, input the model prompt words into the large language model, and use the model prompt words to drive the large language model to determine the relevance score between the query information sample and the content information sample according to the scoring criteria and the semantic similarity between the content information sample and the query information sample; Module 504 is used to construct sample pairs by taking query information samples and content information samples with corresponding relationships as model inputs and the corresponding relevance scores as true values.
[0107] In the sample construction apparatus provided in this embodiment, a more accurate relevance score is generated by using a large language model to construct sample pairs. Compared with the method of manually constructing sample pairs, this method can reduce the sample construction time and improve the accuracy of the relevance score between query information samples and content information samples.
[0108] In some embodiments, the first acquisition module includes: The extraction submodule is used to input the pre-collected user conversation content into the intent recognition model, and the intent recognition model extracts plot-related query information from the user conversation content; A submodule is constructed to build query information samples based on the plot-related query information.
[0109] In some embodiments, the construction submodule is specifically used for: The plot-related query information is matched with entity objects in the target entity library. If the match is successful, the plot-related query information is determined as a query information sample. The entity objects in the target entity library are obtained by entity recognition of the content information in the knowledge base.
[0110] In some embodiments, the query information sample includes at least one of a basic plot query information sample, a plot comparison query information sample, and a plot location query information sample. The basic plot query information sample is used to query at least one of the plot introduction, plot characters, and plot evaluation of a video. The plot comparison query information sample is used to query the comparison between a target video and other videos. The plot location query information sample is used to query the video or video segment where the plot description text is located.
[0111] In some embodiments, the knowledge base includes a plot knowledge base, which includes plot information, and the first acquisition module is further configured to: Extract plot descriptions from the plot information in the plot knowledge base based on the plot synopsis; The plot description is filled into a preset question template to construct a plot location query information sample.
[0112] Figure 6 This is a schematic diagram of a relevance model training device for ranking, provided in an embodiment of this disclosure. Figure 6 As shown, the relevance model training device 600 includes: The second acquisition module 601 is used to acquire multiple sample pairs; wherein, the sample pair includes a query information sample, a content information sample and a relevance score that have a corresponding relationship, the relevance score is used to characterize the semantic relevance between the content information sample and the query information sample, and the sample pair is constructed using the above-mentioned sample construction method; The training module 602 is used to input the query information sample and the content information sample in the sample pair into the model, and the model outputs the predicted relevance score of the query information sample and the content information sample. The predicted relevance score is verified using the relevance score in the sample pair. The parameters of the model are adjusted according to the verification result until the model reaches the preset convergence condition, and the model is used as the relevance model.
[0113] Since the relevance model in this embodiment is trained based on the corresponding query information samples and content information samples, as well as the relevance score between the query information samples and content information samples, in subsequent applications, it is possible to re-rank each candidate content information according to the semantic similarity between the query information and the candidate content information. Compared with the method of re-ranking candidate content information by semantic similarity, this embodiment can improve the recall rate of useful knowledge in knowledge question answering scenarios.
[0114] Figure 7 This is a schematic diagram of a question-answering enhancement processing device provided in an embodiment of this disclosure. Figure 7 As shown, the question-and-answer enhancement processing device 700 includes: The receiving module 701 is used to receive content query information and match multiple candidate content information from the knowledge base based on semantic similarity for the content query information. The sorting module 702 is used to input the content query information and the plurality of candidate content information into a relevance model, whereby the relevance model determines the relevance score between the content query information and each candidate content information, and sorts the candidate content information from largest to smallest according to the relevance score; wherein, the relevance score is used to characterize the semantic relevance between the corresponding candidate content information and the content query information, and the relevance model is trained using the relevance model training method for sorting described in claim 6 above; The generation module 703 is used to generate model prompt words based on the information of the top N candidate contents and the content query information, and input the model prompt words into the question answering model, so that the question answering model can generate the answer content corresponding to the content query information; N is a preset natural number.
[0115] In the question-answering enhancement processing device provided in this embodiment, since the relevance model is trained based on query information samples, plot information samples, and the relevance score between the two, the relevance score determined by the relevance model can accurately rank multiple candidate content information from the semantic similarity dimension between candidate content information and content query information. This makes the candidate content information input to the question-answering model highly relevant to the content query information, thereby improving the response quality of the question-answering model.
[0116] In some embodiments, the content query information includes plot query information, and the knowledge base is a plot knowledge base, which includes plot information.
[0117] The sample construction apparatus, the relevance model training apparatus for ranking, and the question-answering enhancement processing apparatus provided in the embodiments of this disclosure can execute the sample construction method, the relevance model training method for ranking, and the question-answering enhancement processing method provided in any embodiment of this disclosure, and have the corresponding functional modules and beneficial effects of executing the method.
[0118] It is worth noting that in the embodiments of the above-mentioned sample construction device, relevance model training device for ranking, and question-answering enhancement processing device, the various sub-modules are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional sub-module are only for easy differentiation and are not used to limit the scope of protection of this disclosure.
[0119] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Figure 8 As shown, the electronic device 800 includes one or more processors 801 and memory 802.
[0120] The processor 801 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 800 to perform desired functions.
[0121] The memory 802 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 801 may execute the program instructions to implement the sample construction method, the relevance model training method for ranking, the question-answering enhancement processing method, and / or other desired functions described above in the embodiments of this disclosure. Various content, such as query information samples and plot information samples, may also be stored in the computer-readable storage medium.
[0122] In one example, the electronic device 800 may further include an input device 803 and an output device 804, these components being interconnected via a bus system and / or other forms of connection mechanisms (not shown). The input device 803 may include, for example, a keyboard, a mouse, etc. The output device 804 may output various information to the outside, including relevance scores between determined candidate plot information and plot query information. The output device 804 may include, for example, a display, speakers, a printer, and a communication network and its connected remote output devices, etc.
[0123] Of course, for the sake of simplicity, Figure 8 Only some of the components of the electronic device 800 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 800 may include any other suitable components depending on the specific application.
[0124] In addition to the methods and devices described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the sample construction method, the relevance model training method for ranking, and the question-answering enhancement processing method provided in the embodiments of this disclosure.
[0125] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of this disclosure. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0126] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the sample construction method, the relevance model training method for ranking, and the question-answering enhancement processing method provided in embodiments of this disclosure.
[0127] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.
[0128] It should be noted that the terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the scope of this disclosure. As shown in this specification and claims, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. The term "and / or" includes any one and all combinations of one or more of the associated listed items. Relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another entity or operation and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or apparatus. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element.
[0129] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A sample construction method, characterized in that, include: Obtain a sample of the query information; By performing semantic similarity matching between the query information sample and the content information in the knowledge base, the content information corresponding to the query information sample is determined and used as the content information sample corresponding to the query information sample. Based on the scoring criteria, corresponding query information samples and content information samples, model prompt words are generated. The model prompt words are input into the large language model. The model prompt words are used to drive the large language model to determine the relevance score between the query information sample and the content information sample according to the scoring criteria and the semantic similarity between the content information sample and the query information sample. The model inputs are query information samples and content information samples with corresponding relationships, and the corresponding relevance scores are used as the true values to construct sample pairs.
2. The method according to claim 1, characterized in that, The process of obtaining the query information sample includes: The pre-collected user conversation content is input into the intent recognition model, which then extracts plot-related query information from the user conversation content. A query information sample is constructed based on the aforementioned plot-related query information.
3. The method according to claim 2, characterized in that, The construction of query information samples based on the plot-related query information includes: The plot-related query information is matched with entity objects in the target entity library. If the match is successful, the plot-related query information is determined as a query information sample. The entity objects in the target entity library are obtained by entity recognition of the content information in the knowledge base.
4. The method according to claim 1, characterized in that, The query information sample includes at least one of the following: basic plot query information sample, plot comparison query information sample, and plot location query information sample. The basic plot query information sample is used to query at least one of the plot introduction, plot characters, and plot evaluation of a video. The plot comparison query information sample is used to query the comparison between a target video and other videos. The plot location query information sample is used to query the video or video segment where the plot description text is located.
5. The method according to claim 4, characterized in that, The knowledge base includes a plot knowledge base, which contains plot information. The step of obtaining a sample query information includes: Extract plot descriptions from the plot information in the plot knowledge base based on the plot synopsis; The plot description is filled into a preset question template to construct a plot location query information sample.
6. A method for training a relevance model for ranking, characterized in that, include: Multiple sample pairs are obtained; wherein, the sample pair includes a query information sample, a content information sample and a relevance score that have a corresponding relationship, the relevance score is used to characterize the semantic relevance between the content information sample and the query information sample, and the sample pair is constructed using the sample construction method described in any one of claims 1-5 above; The query information sample and content information sample from the sample pair are input into the model, and the model outputs the predicted relevance score of the query information sample and the content information sample. The predicted relevance score is verified using the relevance score in the sample pair. The parameters of the model are adjusted according to the verification result until the model reaches the preset convergence condition. The model is then used as the relevance model.
7. A question-answering enhancement processing method, characterized in that, include: Receive content query information, and match multiple candidate content information from the knowledge base based on semantic similarity; The content query information and the multiple candidate content information are input into a relevance model, which determines the relevance score between the content query information and each candidate content information. The candidate content information is then sorted from largest to smallest based on the relevance score. The relevance score represents the semantic relevance between the corresponding candidate content information and the content query information. The relevance model is trained using the relevance model training method for sorting described in claim 6. Based on the information of the top N candidate contents and the content query information, model prompt words are generated and input into the question answering model. The model prompt words drive the question answering model to generate the answer content corresponding to the content query information; N is a preset natural number.
8. The method according to claim 7, characterized in that, The content query information includes plot query information, and the knowledge base is a plot knowledge base, which includes plot information.
9. A sample construction apparatus, characterized in that, include: The first acquisition module is used to acquire a sample of query information. The first determining module is used to determine the content information corresponding to the query information sample by performing semantic similarity matching between the query information sample and the content information in the knowledge base, and use it as the content information sample corresponding to the query information sample. The second determining module is used to generate model prompt words based on the scoring criteria, corresponding query information samples and content information samples, input the model prompt words into the large language model, and use the model prompt words to drive the large language model to determine the relevance score between the query information sample and the content information sample according to the scoring criteria and the semantic similarity between the content information sample and the query information sample; The construction module is used to construct sample pairs by taking corresponding query information samples and content information samples as model inputs and the corresponding relevance scores as real values.
10. A relevance model training device for ranking, characterized in that, include: The second acquisition module is used to acquire multiple sample pairs; wherein, the sample pair includes a query information sample, a content information sample and a relevance score that have a corresponding relationship, the relevance score is used to characterize the semantic relevance between the content information sample and the query information sample, and the sample pair is constructed using the sample construction method described in any one of claims 1-5 above; The training module is used to input the query information samples and content information samples from the sample pairs into the model, and the model outputs the predicted relevance scores of the query information samples and the content information samples. The predicted relevance scores are verified using the relevance scores in the sample pairs. The parameters of the model are adjusted according to the verification results until the model reaches the preset convergence condition, and the model is then used as the relevance model.
11. A question-answering enhancement processing device, characterized in that, include: Receive content query information, and match multiple candidate content information from the knowledge base based on semantic similarity; A receiving module is configured to input the content query information and the plurality of candidate content information into a relevance model, wherein the relevance model determines a relevance score between the content query information and each candidate content information, and sorts the candidate content information in descending order of the relevance scores; wherein the relevance score is used to characterize the semantic relevance between the corresponding candidate content information and the content query information, and the relevance model is trained using the relevance model training method for sorting described in claim 6. The sorting module is used to generate model prompts based on the information of the top N candidate contents and the content query information, and input the model prompts into the question-answering model, which then drives the question-answering model to generate the answer content corresponding to the content query information; N is a preset natural number.
12. An electronic device, characterized in that, The electronic device includes: Processor and memory; The processor, by calling the program or instructions stored in the memory, is used to execute the sample construction method as described in any one of claims 1 to 5, or to execute the relevance model training method for ranking as described in claim 6, and further to execute the question-answering enhancement processing method as described in claim 7.
13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that cause a computer to perform the sample construction method as described in any one of claims 1 to 5, or to perform the relevance model training method for ranking as described in claim 6, and further to perform the question-answering enhancement processing method as described in claim 7.
14. A computer program product, characterized in that, The computer program product is used to implement the sample construction method according to any one of claims 1 to 5, or to implement the relevance model training method for ranking as described in claim 6, and to implement the question-answering enhancement processing method as described in claim 7.