Information generation method, electronic device, storage medium, and program product

Abstract

The application provides an information generation method, an electronic device, a storage medium and a program product, which can be applied to the technical field of information query and language model. The method comprises the following steps: in response to receiving to-be-queried information, obtaining a plurality of initial retrieval results from a candidate information set based on the characteristics of the to-be-queried information, wherein the characteristics comprise at least one of the following: complexity, relevance, density characteristics and type characteristics; obtaining evaluation results corresponding to the plurality of initial retrieval results according to evaluation parameters and evaluation strategies of the initial retrieval results; obtaining target retrieval results based on the evaluation results and the plurality of initial retrieval results, and obtaining an initial generation result based on the to-be-queried information and the target retrieval results, wherein the target retrieval results are retrieval results in the plurality of initial retrieval results, evaluation results of which satisfy a preset evaluation condition; and updating the initial generation result based on risk evaluation information of the initial generation result and an update strategy to obtain a target generation result.

Description

Technical Field

[0001] The present invention relates to the technical field of information query and language model, and in particular to an information generation method, electronic equipment, storage medium and program product. Background Art

[0002] Retrieval-Augmented Generation (RAG) systems effectively alleviate the knowledge timeliness limitations and hallucination problems of large language models by combining external knowledge bases with generative models. However, when dealing with complex queries, multi-hop reasoning, or specialized domain problems, existing RAG systems suffer from retrieval failures, information conflicts, and limited optimization and evaluation mechanisms. These issues lead to low reliability and make them difficult to meet practical retrieval needs. Summary of the Invention

[0003] In view of the above problems, the present invention provides an information generation method, apparatus, electronic device, storage medium and program product.

[0004] According to a first aspect of the present invention, there is provided an information generation method, comprising: in response to receiving information to be queried, obtaining a plurality of initial retrieval results from a candidate information set based on features of the information to be queried, wherein the features include at least one of the following: the complexity of the information to be queried, the correlation between the information to be queried and domain information, the density features of entities in the information to be queried, and the type features of the information to be queried; obtaining evaluation results corresponding to the plurality of initial retrieval results based on evaluation parameters and an evaluation strategy of the initial retrieval results; obtaining a target retrieval result based on the evaluation result and the plurality of initial retrieval results, and obtaining an initial generation result based on the information to be queried and the target retrieval result, wherein the target retrieval result is a retrieval result whose evaluation result satisfies a preset evaluation condition among the plurality of initial retrieval results; updating the initial generation result based on risk assessment information and an update strategy of the initial generation result to obtain a target generation result.

[0005] The second aspect of the present invention provides an information query device, comprising: a retrieval result determination module, for, in response to receiving information to be queried, obtaining multiple initial retrieval results from a candidate information set based on the characteristics of the information to be queried, wherein the characteristics include at least one of the following: the complexity of the information to be queried, the correlation between the information to be queried and the domain information, the density characteristics of the entities in the information to be queried, and the type characteristics of the information to be queried; an evaluation result determination module, for obtaining evaluation results corresponding to multiple initial retrieval results based on the evaluation parameters and evaluation strategy of the initial retrieval results; a generation result determination module, for obtaining a target retrieval result based on the evaluation result and multiple initial retrieval results, and obtaining an initial generation result based on the information to be queried and the target retrieval result, wherein the target retrieval result is a retrieval result whose evaluation result meets a preset evaluation condition among the multiple initial retrieval results; an update module, for updating the initial generation result based on the risk assessment information and update strategy of the initial generation result to obtain a target generation result.

[0006] A third aspect of the present invention provides an electronic device, comprising: a memory; and a processor configured to execute the above-mentioned information query method according to instructions and data stored in the memory.

[0007] The fourth aspect of the present invention further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0008] The fifth aspect of the present invention further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0010] Figure 1 A diagram illustrating an application scenario of an information generation method, apparatus, electronic device, storage medium, and program product according to an embodiment of the present invention is shown;

[0011] Figure 2 A flow chart of an information generating method according to an embodiment of the present invention is shown;

[0012] Figure 3 A flowchart of a method for generating information based on self-verification and error correction of a query system according to an embodiment of the present invention is shown;

[0013] Figure 4 It shows a structural block diagram of an information query device according to an embodiment of the present invention;

[0014] Figure 5 A block diagram of an electronic device according to an embodiment of the present invention is shown.

[0015] Figure 6 A block diagram of a first electronic device suitable for implementing an information generating method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0016] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0017] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0018] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0019] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0020] In some examples, similarity scores of search results are calculated as an indicator of search quality. Based on the search results and similarity scores, the system generates answers. Furthermore, the system uses the output probability distribution of the language model to calculate a confidence score, and determines whether uncertainty exists based on the confidence score and a preset threshold. This method, to a certain extent, satisfies the uncertainty quantification requirements of the generated information.

[0021] However, this approach relies solely on the output probabilities of the language model and the retrieval similarity scores to estimate uncertainty, ignoring sources of error such as retrieval failures and information conflicts. Furthermore, it is limited to identifying potential uncertainties and fails to proactively correct errors or improve retrieval strategies. For partially correct and partially incorrect answers, only an overall confidence score is given, failing to pinpoint the specific error point or exact error type.

[0022] In some examples, information-enhanced retrieval systems are constructed based on multi-path retrieval and re-ranking. For example, multiple retrieval strategies and re-ranking models are applied to comprehensively score and rank retrieval results, and the top-ranked results are selected as contextual input to a language model to generate a final answer. This approach improves the coverage of retrieval strategies to a certain extent.

[0023] However, this method lacks a verification mechanism for the final generated content and cannot ensure the consistency between the generated content and the retrieved information. When multiple sources provide contradictory information, it relies solely on the reranking model to select results, rather than resolving or identifying knowledge conflicts. Moreover, the fusion strategy of the retrieval results is predefined and difficult to dynamically adjust according to the query type or retrieval effect. At the same time, retrieval and generation in this scheme are two relatively independent steps, and potential problems discovered during the generation process cannot be fed back to the retrieval link for optimization.

[0024] Based on the above problems, the present invention provides an information generation method, comprising: in response to receiving information to be queried, obtaining multiple initial retrieval results from a candidate information set based on the characteristics of the information to be queried, wherein the characteristics include at least one of the following: the complexity of the information to be queried, the correlation between the information to be queried and the domain information, the density characteristics of the entities in the information to be queried, and the type characteristics of the information to be queried; according to the evaluation parameters and evaluation strategy of the initial retrieval results, obtaining evaluation results corresponding to the multiple initial retrieval results; based on the evaluation results and the multiple initial retrieval results, obtaining a target retrieval result, and based on the information to be queried and the target retrieval result, obtaining an initial generation result, wherein the target retrieval result is a retrieval result whose evaluation result meets a preset evaluation condition among the multiple initial retrieval results; updating the initial generation result based on the risk assessment information and update strategy of the initial generation result to obtain a target generation result.

[0025] According to an embodiment of the present invention, since the initial retrieval result is determined by comprehensively integrating the multi-dimensional features of the information to be queried, it meets the dynamic determination and adjustment mode of the retrieval result, and thus the evaluation result of the initial retrieval result is flexibly determined according to the evaluation strategy, optimization strategy and evaluation parameters, and combined with the target retrieval result that meets the preset evaluation conditions, the initial generation result after evaluation is obtained. On this basis, possible errors are further identified according to the risk assessment information of the initial generation result to trigger the update strategy to update the initial generation result, and the target generation result that has undergone multi-layer evaluation and update optimization is obtained. By evaluating the retrieval results in the retrieval stage and updating the generation results in the generation stage, even if an error occurs in a certain link, it can be discovered and blocked in time, preventing the accumulation and expansion of errors. Especially for multi-hop reasoning and high-complexity scenarios, the error isolation mechanism significantly improves the overall stability and reliability of the system.

[0026] Figure 1 A diagram illustrating application scenarios of the information generation method, apparatus, electronic device, storage medium, and program product according to an embodiment of the present invention is shown.

[0027] like Figure 1 As shown, the application scenario according to this embodiment may include a terminal device 101, a network 102, and a server 103. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired or wireless communication links or optical fiber cables.

[0028] A user can use a terminal device 101 to interact with a server 103 via a network 102 to receive or send messages, etc. The terminal device 101 can be any electronic device with a display screen and supporting web browsing, including but not limited to a smartphone, a tablet computer, a laptop computer, a desktop computer, and the like.

[0029] For example, an application on the terminal device 101 (such as chat software, smart assistant application, etc.) can receive the query information input by the user, perform basic preprocessing on the content of the query information, and convert it into a format suitable for transmission on the network, such as encapsulating the question into a data packet according to a certain protocol format.

[0030] Taking a medical scenario as an example, the terminal device 101 may be a self-service query device used by medical personnel or patients, etc. For example, the terminal device 101 sends the information to be queried to the server 103 via the network 102 .

[0031] After receiving the request, server 103 searches its vast database of medical literature, clinical case studies, and other resources. Using pre-defined search algorithms, it matches the medical literature with keywords, disease names, treatment method descriptions, and other information. Server 103 returns the relevant search results to terminal device 101 via network 102, where they are displayed on the doctor's or researcher's screen.

[0032] For example, in a scientific research scenario, researchers can use research literature search software on terminal device 101 and enter the information to be searched (e.g., "application of new nanomaterials in energy storage"). The terminal device then sends the information to server 103 via high-speed network 102.

[0033] Server 103 searches a scientific literature database (including academic journals, conference papers, patent documents, etc.) and can find documents related to the topic based on information such as the document's title, abstract, keywords, and author. Furthermore, server 103 can return the retrieved relevant scientific literature (such as the full text or abstracts of several recent, high-impact papers on the application of nanomaterials in energy storage) to terminal device 101 via network 102 for reference by researchers.

[0034] It should be noted that the information generation method or task execution method provided in the embodiments of the present invention can generally be executed by the server 103. Accordingly, the information query device or task execution device provided in the embodiments of the present invention can generally be set in the server 103. The information generation method or task execution method provided in the embodiments of the present invention can also be executed by a server or server cluster that is different from the server 103 and can communicate with the terminal device 101 and / or the server 103. Accordingly, the information query device or task execution device provided in the embodiments of the present invention can also be set in a server or server cluster that is different from the server 103 and can communicate with the terminal device 101 and / or the server 103.

[0035] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0036] Figure 2 A flow chart of an information generating method according to an embodiment of the present invention is shown.

[0037] like Figure 2 As shown, the information generating method of this embodiment includes operations S210 to S240.

[0038] In operation S210, in response to receiving the information to be queried, multiple initial retrieval results are obtained from the candidate information set based on the characteristics of the information to be queried, wherein the characteristics include at least one of the following: the complexity of the information to be queried, the correlation between the information to be queried and the domain information, the density characteristics of the entities in the information to be queried, and the type characteristics of the information to be queried.

[0039] In an embodiment of the present invention, the information to be queried may be information input by a user through an electronic device for retrieval or generation, and its types include but are not limited to text information, image information, audio information, and video information. Complexity may be a complexity index determined based on syntactic analysis and / or entity recognition methods, and entities may be objects with actual physical meaning in the information to be queried. Domain information may include fields such as medicine, law, finance, and scientific research. The density feature of an entity may be determined by the number of entities in the information to be queried and the text length of the information to be queried itself. Type features may include factual queries, reasoning queries, and open queries.

[0040] For example, after receiving the information to be queried input by the user through an electronic device, the query system can determine multiple initial retrieval results from the candidate information set based on the complexity of the current information to be queried, the density characteristics of the entity, and the correlation between the current information to be queried and the medical, legal, financial and scientific research fields.

[0041] In operation S220 , evaluation results corresponding to the plurality of initial search results are obtained according to the evaluation parameters and the evaluation strategy of the initial search results.

[0042] In an embodiment of the present invention, the evaluation parameter may be an initial search evaluation result obtained by performing a multi-dimensional evaluation on the initial search result. The evaluation strategy may be a strategy for the user to perform a weighted evaluation on the initial search evaluation result according to actual application scenarios and requirements.

[0043] For example, after determining the evaluation parameters of the initial search results, the evaluation strategy may be combined to perform weighting and evaluation processing on the multiple initial search results.

[0044] In operation S230, a target search result is obtained based on the evaluation result and the multiple initial search results, and an initial generated result is obtained based on the query information and the target search result. The target search result is a search result among the multiple initial search results whose evaluation result satisfies the preset evaluation conditions. In an embodiment of the present invention, the target search result may be a search result determined after optimizing or updating the initial search results based on an optimization strategy. The optimization strategy may be a strategy for verifying, optimizing, or updating the initial search evaluation results. The initial generated result may be an initial result obtained based on the target search result among the evaluation results that satisfies the preset evaluation conditions.

[0045] For example, when the evaluation results cannot meet the preset evaluation conditions, the optimization strategy is used to verify, optimize or update the initial search results; when the optimized search results are obtained, the optimized search results are further evaluated and verified, and when the verification is passed, the verified search results are used as the target search results; and then the search operation is performed in the target search results to generate the initial generation results.

[0046] In operation S240 , the initial generation result is updated based on the risk assessment information of the initial generation result and the update strategy to obtain a target generation result.

[0047] In embodiments of the present invention, risk assessment information may be an evaluation result obtained by evaluating the match, support, scope, and uncertainty between the initial generation result and the target search result from multiple dimensions. An update strategy may be a method for updating an initial generation result whose risk assessment information does not meet preset evaluation criteria. The target generation result may be a generation result updated using the update strategy, or may be the generation result ultimately output by the query system based on the query information input by the user.

[0048] For example, when the risk assessment information does not meet the preset risk assessment conditions, the initial generation result is updated using the update strategy to obtain a target generation result that meets the preset risk assessment conditions.

[0049] According to an embodiment of the present invention, since the initial retrieval result is determined by comprehensively integrating the multi-dimensional features of the information to be queried, it meets the dynamic determination and adjustment mode of the retrieval result, and thus the evaluation result of the initial retrieval result is flexibly determined according to the evaluation strategy, optimization strategy and evaluation parameters, and combined with the target retrieval result that meets the preset evaluation conditions, the initial generation result after evaluation is obtained. On this basis, possible errors are further identified according to the risk assessment information of the initial generation result to trigger the update strategy to update the initial generation result, and the target generation result that has undergone multi-layer evaluation and update optimization is obtained. By evaluating the retrieval results in the retrieval stage and updating the generation results in the generation stage, even if an error occurs in a certain link, it can be discovered and blocked in time, preventing the accumulation and expansion of errors. Especially for multi-hop reasoning and high-complexity scenarios, the error isolation mechanism significantly improves the overall stability and reliability of the system.

[0050] According to an embodiment of the present invention, based on the characteristics of the information to be queried, multiple initial retrieval results are obtained from the candidate information set, including: determining the characteristics corresponding to the information to be queried; determining the target policy information from the policy information set based on the characteristics; performing a retrieval operation in the candidate information set based on the target policy information to obtain multiple initial retrieval results, wherein the multiple initial retrieval results include an initial text and an initial correlation between the initial text and the information to be queried.

[0051] In the embodiments of the present application, the set of policy information can be a retrieval policy library for retrieval, and can include various retrieval policies and combined policies between different retrieval policies. The target policy information can be retrieval policy information determined according to the characteristics of the current information to be queried. The initial text can be a document obtained after performing a retrieval operation. The initial correlation degree can be an initial relevance score between the initial text and the information to be queried.

[0052] For example, based on the multi-dimensional characteristics of the information to be queried, the matching degree and the weight between the current policy information can be determined based on the multi-dimensional characteristics, and the target policy information can be determined. Then, the retrieval policy can be performed based on the target policy information to obtain an initial retrieval result.

[0053] According to the embodiments of the present application, the characteristics corresponding to the information to be queried are determined, including at least one of the following: determining the complexity based on the analysis result and the recognition result of the information to be queried; determining the relevance based on the field correlation degree between the information to be queried and the field information in the candidate information set; determining the density characteristic as the ratio between the number of entities of the information to be queried and the entity query length value; and combining the multiple type labels of the information to be queried to obtain the type characteristic.

[0054] In the embodiments of the present application, a comprehensive index can be constructed to calculate the complexity of the query in combination with the results of the syntax analysis and the entity recognition. According to the results of the syntax analysis, a syntax complexity index is defined. For example, the number of long sentences in the query, the number of nested clauses, the number of complex phrase structures, etc. can be counted. According to the results of the entity recognition, an entity complexity index is defined. The number of entities, the type diversity, etc. can be counted.

[0055] For example, the information to be queried can be preprocessed, including word segmentation, part-of-speech tagging, syntax analysis and entity recognition; syntax features such as sentence length, clause number, phrase nesting depth, etc. are extracted; entity features such as the number of entities, the number of entity types, etc. are extracted; and then the comprehensive complexity score of the query is calculated according to the extracted features.

[0056] In the embodiments of the present application, the field correlation degree between the information to be queried and the field information in the candidate information set can be determined based on a semantic analysis method or a key word and phrase matching method.

[0057] For example, key words and phrases are extracted from the information to be queried, and the occurrence frequency of these key words in different knowledge field documents is detected, and the field with high occurrence frequency indicates a higher correlation degree with the information to be queried; and the core key word set of different knowledge fields can also be obtained and stored in advance in the query system, and the key words in the query are compared with these sets to calculate the matching degree to evaluate the correlation degree.

[0058] In an embodiment of the present invention, the entity density index can be determined by calculating the ratio of the number of named entities contained in the information to be queried to the query length, so as to evaluate the richness of the entity information contained in the information to be queried based on the entity density index, thereby providing reference information for subsequent query analysis and retrieval strategy optimization.

[0059] In an embodiment of the present invention, the type of information to be queried can be determined through semantic analysis, keyword matching, context analysis, and query structure analysis, and its types may include factual query, inference query, and open query.

[0060] In a feasible embodiment, the query analysis module in the query system can be used to analyze the query information and calculate the feature F of the query information. Q , as shown in the following formula (1):

[0061] (1);

[0062] Among them, C Q Can represent complexity, D Q Can characterize the correlation, E Q Can characterize density characteristics, T Q It can represent type characteristics, and Q can be the information to be queried.

[0063] The query analysis module accurately identifies query complexity and reasoning requirements, adaptively selects a multi-stage retrieval strategy, and tracks reasoning chains and evidence associations during answer generation. This structured information processing approach enables the system to effectively handle complex problems that require connecting multiple information points, demonstrating excellent performance in highly complex scenarios such as medical diagnostic assistance, legal analysis, and scientific research.

[0064] According to embodiments of the present invention, the most appropriate retrieval algorithm and strategy are selected based on the type of information being queried (e.g., factual, inferential, or open-ended). For example, for factual queries, specific factual information can be quickly matched, reducing interference from irrelevant information. Furthermore, by identifying the query domain (e.g., science and technology, healthcare, or history), searches can be performed within datasets in that specific domain, narrowing the search scope and improving search speed and accuracy.

[0065] According to an embodiment of the present invention, a policy information set includes multiple retrieval strategies and a combination strategy between the multiple retrieval strategies; determining target policy information from the policy information set based on features includes: determining the degree of match between the features and the current policy information; determining the target policy information from the policy information set based on the degree of match between the features and the current policy information and the weight of the current policy information.

[0066] In the embodiments of the present application, the set of policy information can be a plurality of retrieval strategies and a combination strategy between different retrieval strategies stored in advance in the query system for subsequent information query.

[0067] For example, the feature F of the information to be queried can be used as a basis Q The optimal strategy combination is selected from the retrieval strategy library, and the retrieval strategy selection is specifically shown in the following formula (2):

[0068] (2);

[0069] Wherein, S opt The optimal strategy combination currently selected can be represented by S, S can represent the set of policy information, and S i The target policy information currently determined can be represented by S The initial correlation degree between the feature and the target policy information can be represented by F. The weight of the initial correlation degree learned through the historical data can be represented by W.

[0070] In a feasible embodiment, an adaptive retrieval module in the query system can be used to perform a retrieval operation, so that a set of retrieval results, denoted as R = r1, r2,..., rk, can be obtained, and each result can include document content and an initial relevance score.

[0071] According to the embodiments of the present application, the method further comprises at least one of the following: determining the information coverage of the initial retrieval result to the information to be queried based on the target sequence information, the semantic similarity information and the respective corresponding weight between the information to be queried and the initial retrieval result, as the coverage in the evaluation parameters, wherein the target sequence information includes the target sequence, the appearance frequency and the distribution information of the target sequence that exist simultaneously in the information to be queried and the initial retrieval result; determining the semantic correlation degree between the information to be queried and the initial retrieval result, and taking the semantic correlation degree as the correlation degree in the evaluation parameters; converting the plurality of initial retrieval results into a plurality of word vectors respectively, so as to determine the vector similarity between the plurality of word vectors as the similarity in the evaluation parameters; determining the evaluation confidence between the information to be queried and the initial retrieval result based on the type information of the information to be queried, the number of initial retrieval results and the evaluation strategy, as the confidence in the evaluation parameters.

[0072] In the embodiments of the present application, the evaluation parameters can include at least one of the coverage, the correlation degree, the similarity and the evaluation confidence. The target sequence can be a keyword or a key word that appears simultaneously in the information to be queried and the initial retrieval result.

[0073] The coverage can be used to evaluate the coverage degree between the information to be queried and the initial retrieval results. The relevance can be determined by cross-encoding the semantic relevance between the information to be queried and the initial retrieval results. The similarity can be used to evaluate the information consistency degree between the plurality of initial retrieval results. The confidence can be a confidence determined based on a type of the information to be queried and a normalized index of the number of initial retrieval results.

[0074] For example, the frequency and the distribution position of the keywords appearing in the information to be queried and the initial retrieval results can be detected and analyzed to obtain a first coverage corresponding to the keywords. The semantic similarity between the respective documents of the information to be queried and the initial retrieval results can be calculated by using a semantic analysis technology, and the respective documents can be converted into semantic vectors, and the cosine similarity value can be calculated to obtain a second coverage in the semantic. The first coverage and the second coverage are respectively weighted and combined to obtain the coverage in the evaluation parameters.

[0075] For example, the semantic relevance between the information to be queried and the initial retrieval results can be determined by combining the cross-encoding technology in deep learning. The feature vectors can be obtained by performing word segmentation processing and feature extraction on the information to be queried and the initial retrieval results, respectively, and then the relevance between the feature vectors is calculated as the relevance in the evaluation parameters.

[0076] For example, the similarity in the evaluation parameters can be evaluated by converting the text in the initial retrieval results into a word vector, and then calculating the similarity between the vectors. The similarity includes: obtaining a word vector after preprocessing the document, thereby merging all word vectors in the document into a document vector by a weighted average method, and then calculating the similarity between different document vectors by using cosine similarity or Euclidean distance.

[0077] For example, the confidence of the retrieval results can be evaluated by considering the number of different types of information to be queried and initial retrieval results, including a confidence evaluation model or a related algorithm (such as fuzzy logic and extreme learning algorithm).

[0078] In a feasible embodiment, the retrieval verification module in the query system can be used to perform multi-dimensional evaluation on the initial retrieval results to obtain the evaluation parameters Q R of the initial retrieval results, which is shown in the following formula (3):

[0079] (3);

[0080] Wherein, Cov R may be the information coverage of the initial retrieval results to the information to be queried, Rel R may be the semantic relevance between the information to be queried and the initial retrieval results, and Con RIt can be the vector similarity between multiple word vectors, Rec R It can be used to evaluate the confidence between the query information and the initial retrieval results.

[0081] According to embodiments of the present invention, single-dimensional evaluation metrics often only reflect one aspect of the quality of search results. Multi-dimensional evaluation can comprehensively consider multiple aspects, such as information coverage, relevance, similarity, and confidence, thereby more comprehensively and accurately measuring the overall quality of search results. This avoids misjudgments of search result quality due to the one-sidedness of a single metric and reduces bias in evaluation results. High information coverage means that search results contain more relevant content, reducing the risk of missing information.

[0082] According to an embodiment of the present invention, based on the type information of the information to be queried, the number of initial retrieval results and the evaluation strategy, the evaluation confidence between the information to be queried and the initial retrieval results is determined, including: performing normalization processing based on the type information and the number of initial retrieval results to obtain the confidence; or fusing the reference confidence of the type information and the number of initial retrieval results, multiple type information at multiple moments and multiple result numbers to obtain the confidence.

[0083] In an embodiment of the present invention, the normalization process can be implemented by a normalization model to calculate the confidence level. The reference confidence level can be a priori confidence level obtained based on experience or experimental statistics. The multiple types of information can be multiple types of information to be queried.

[0084] For example, based on the query type and the number of retrieval results, a mathematical model is constructed to map different types of queries and different numbers of retrieval results to a unified confidence assessment scale; machine learning methods can be used to construct a normalization model; and the pre-trained normalization model obtained by sample information training is used to process the input type information and initial retrieval results to obtain multiple confidence levels, which can be used for sorting or screening.

[0085] For example, based on historical experience or statistics, determine the confidence prior distribution of different types of information to be queried and the number of initial retrieval results; record the actual retrieval effects under different query types and numbers of retrieval results, such as user click-through rate, feedback satisfaction, etc.; thus, the prior probability and observation data can be combined to update the posterior probability, that is, the confidence under the current query and retrieval results.

[0086] According to embodiments of the present invention, the confidence level of initial search results is updated in real time based on new observational data. As more information becomes available, the query system can dynamically adjust its confidence level in the search results, making the confidence level more accurate and reliable. Furthermore, this method can adapt to changes in data distribution and user needs, promptly reflecting these changes in the confidence level calculation, making the query system more flexible and adaptable.

[0087] According to an embodiment of the present invention, a target retrieval result is obtained based on an evaluation result and multiple initial retrieval results, including: when the evaluation result is less than an evaluation threshold, updating the policy information set to obtain updated policy information; using the updated policy information to generate multiple intermediate retrieval results corresponding to the information to be queried; when the intermediate evaluation result of the intermediate retrieval result is greater than or equal to the evaluation threshold, using the intermediate retrieval result as the target retrieval result.

[0088] In embodiments of the present invention, multiple evaluation parameters may be weighted based on an evaluation strategy to obtain an evaluation result. Furthermore, a target search result may be determined from multiple initial search results based on the evaluation result and an evaluation threshold. The evaluation threshold may be determined based on actual conditions or experiments, and is not specifically limited herein. The updated policy information may be information obtained by updating, optimizing, and enhancing the policy information in the policy information set.

[0089] For example, the evaluation result Q score It can be determined by the following formula (4):

[0090] (4);

[0091] Among them, w1, w2, w3 and w4 are the weights of information coverage, semantic relevance, vector similarity and evaluation confidence respectively.

[0092] Furthermore, when Q score Below the threshold τ Q When the intermediate evaluation result of the intermediate retrieval result is greater than or equal to the evaluation threshold, the retrieval result corresponding to the intermediate evaluation result is used as the target retrieval result.

[0093] According to an embodiment of the present invention, updating the policy information set to obtain updated policy information includes: updating the information to be queried, the domain information in the policy information set, and the retrieval strategy respectively to obtain updated information to be queried, updated domain information, and updated retrieval strategy; and combining the updated domain information, updated retrieval strategy, and updated information to be queried to obtain updated policy information.

[0094] In the embodiment of the present invention, re-search can be achieved by reconstructing the information to be queried, updating and expanding the query scope, or switching the search strategy to obtain updated domain information, updated search strategy, and updated information to be queried after re-search.

[0095] For example, reconstructing the query information can include identifying key terms in the query information, expanding them using a synonym dictionary, and adding synonyms to the query information. Alternatively, word embedding can be used to add semantically related words to the query information. Alternatively, the query information can be reconstructed in different combinations by adjusting the order of keywords in the query information. It is understood that the query information can be reconstructed using any combination of the above methods.

[0096] For example, the search scope can be expanded to other fields related to the original field, while the time range, geographical range, or document type of the search can be expanded to achieve an update and expansion of the query scope. Furthermore, if the keyword search results do not meet the requirements, a search method based on semantic understanding can be used.

[0097] According to an embodiment of the present invention, by updating the policy information set through methods such as synonym expansion, semantic expansion and keyword reorganization, the user's needs can be expressed more accurately, the relevance of the retrieval results can be improved, and expanding the retrieval scope can help the system obtain more relevant information and increase the chance of finding documents that match the user's needs. Switching to a more appropriate retrieval strategy can better match the semantics of the query and the user's true intention, and improve the accuracy of the retrieval results; providing more accurate retrieval results can reduce the number of times the user manually adjusts the information to be queried, save user time, better understand user needs, and improve user satisfaction with the retrieval system.

[0098] According to an embodiment of the present invention, the method further includes: updating the evaluation threshold based on the user's preference information and the current correlation between the current information to be queried and the domain information to obtain an updated evaluation threshold, and using the updated evaluation threshold to update the evaluation result.

[0099] In an embodiment of the present invention, the evaluation threshold of the evaluation result may be personalized and updated in consideration of actual application scenarios and user preferences, so as to better adapt to actual scenarios and user needs.

[0100] For example, the query system can dynamically adjust the verification threshold according to the application scenario, the characteristics of the information to be queried, and the user's preferences to strike a balance between specificity (avoiding errors) and sensitivity (providing useful information). The update method of the evaluation threshold is shown in the following formula (5):

[0101] (5);

[0102] Among them, τbase It can be used as the basic threshold, D Q It can be a query domain relevance vector; U P It can be the user’s preference vector, which indicates the user’s preference for the accuracy and completeness of the generated results. Δτ can be an adjustment function that maps the domain relevance vector and user preference to the threshold adjustment amount, τ dynamic The threshold can be evaluated after the update.

[0103] According to embodiments of the present invention, dynamically adjusting the evaluation threshold can more accurately filter out high-quality results relevant to the query based on the user's specific needs and the characteristics of the information being searched, thereby improving the relevance and accuracy of retrieval results. For example, by increasing the verification threshold, irrelevant or low-quality information can be effectively filtered out, reducing noise and enabling users to obtain useful information more quickly. Striking a balance between specificity and sensitivity ensures that users neither miss important information nor are overwhelmed by large amounts of useless information, thereby improving user satisfaction with the retrieval system.

[0104] According to an embodiment of the present invention, an initial generation result is obtained based on the information to be queried and the target retrieval result, including: generating the initial generation result based on the target retrieval result, the type information of the information to be queried, the intent information, and the template information corresponding to the type information and the intent information.

[0105] In an embodiment of the present invention, the target search result may be a result in the initial search result whose corresponding evaluation result is greater than or equal to the evaluation threshold. The template information may be multiple templates stored in the query system corresponding to different types and intents of the information to be queried.

[0106] For example, based on the type and intent of the query, an appropriate answer template can be selected. For example, for a query like "The latest AI trends in 2024," a template can be selected that lists multiple trend points and briefly describes each trend. The selected template is then populated with filtered and integrated information from the target search results, and the language is organized using natural language generation technology to make the answer fluent and natural. The generated answer is then adjusted and polished to ensure the readability and professionalism of the initial generated results.

[0107] In a feasible embodiment, the answer generation module in the query system can be used to generate the initial generation result A through the received target search results and the information to be queried. init For example, the query system answers user questions (the information to be queried) based on the provided context. For each key point in the answer, the specific document source supporting that point can be indicated. If insufficient information is available, this can be clearly stated.

[0108] Furthermore, the query system can record A init The internal state in the generation process can be recorded as state, as shown in the following formula (6):

[0109] (6);

[0110] Among them, se i It can be the i-th statement or sequence in the initial generated result, c i The confidence level of the generated sentence d can be expressed as i The set of document indexes in the target retrieval results that support the i-th sentence can be represented, and n can represent the number of sentences in the initial generated results.

[0111] According to an embodiment of the present invention, the risk assessment information includes a first matching degree between the initial generation result and the initial retrieval result; the method also includes: determining the semantic matching degree and the embedded matching degree between the query sequence of the initial generation result and the result sequence of the initial retrieval result, so as to combine the semantic matching degree and the embedded matching degree to obtain the first matching degree.

[0112] In an embodiment of the present invention, the degree of matching between the query sequence of the initial generated result and the result sequence of the initial retrieval result may be determined based on semantic similarity and embedding similarity algorithms.

[0113] For example, the sentences in the initial generation results and the target retrieval results can be converted into semantic representations, and the similarity between these semantic representations can be calculated using dot product and cosine similarity algorithms to measure the similarity of the sentences.

[0114] For example, each word in the respective sentences of the initial generation result and the target retrieval result can be converted into a word embedding vector, and the word embedding vectors can be combined into a sentence embedding vector through weighted summation to calculate the similarity between the embedding vectors of the sentences in the initial generation result and the target retrieval result.

[0115] On the basis of obtaining the semantic matching degree and embedding matching degree, the first matching degree F between the initial generation result and the initial retrieval result can be obtained. A , as shown in the following formula (7):

[0116] (7);

[0117] Among them, se i It can be the i-th statement or sequence in the initial generated result, d' j It can be the jth sequence in the initial search results, Sim(se i ,d' j) can be the similarity between the i-th statement and the j-th sequence in the initial search results, A can be the set of initial generated results, It can represent the total length of the set of initially generated results, and R' can represent the retrieved results after verification.

[0118] According to an embodiment of the present invention, the initial generation result includes multiple sub-information, and the risk assessment information also includes a second matching degree between the multiple sub-information; the method also includes: determining multiple information matching degrees between the multiple sub-information and a valley matching degree among the multiple information matching degrees, so as to convert the valley matching degree to obtain a second matching degree.

[0119] The sub-information may be a sub-generation result in the initial generation result, the valley matching degree may be the lowest matching degree value among the multiple information matching degrees, and the second matching degree may be obtained based on the logical consistency between the multiple sub-generation results within the initial generation result.

[0120] For example, the second matching degree C between multiple self-information A The determination method is shown in the following formula (8):

[0121] (8);

[0122] Among them, ConsistencyScore(se i ,s j ) is a function that evaluates the logical consistency between two sub-information. Its value range is [0,1]. The larger the value, the more consistent it is. j This can be the jth statement or sequence in the initial generated results. The brackets within min() represent the average consistency score across all initial generated results. min() can indicate the worst case consistency (to prevent "partial consistency" from masking "local serious contradictions"). 1-min() converts the consistency score into an "inconsistency risk score," with higher values ​​indicating more severe internal conflicts.

[0123] According to an embodiment of the present invention, the risk assessment information further includes support of the initial search result to the initial generation result; the method further includes: determining the support based on the length value of the initial generation result and a support threshold.

[0124] In embodiments of the present invention, the length value may be the total length of the initial generated results, and the support value may indicate the extent to which supporting content can be found in the initial search results. A low support value indicates a high risk of hallucination in the current generated results. The support value threshold may be determined based on experiments or actual conditions and is not specifically defined herein.

[0125] For example, by identifying the content that can be supported by the initial search results, the support degree H of the initial search results to the initial generated results is determined. A The method is shown in the following formula (9):

[0126] (9);

[0127] in, It can be an indicator function, when the similarity exceeds the threshold If there is no sufficiently similar supporting evidence for an answer, it is 1, otherwise it is 0. When there is no sufficiently similar supporting evidence for an answer, it can be counted as a "missing" answer. |A| is the total length of the set of initial generated results.

[0128] Through multi-dimensional verification and precise error detection mechanisms, the query system can detect potential hallucinations in real time during the generation of results (answers). When a portion of an answer is detected as lacking support from search results, the system immediately triggers a specialized error correction process, including secondary search, content reconstruction, or explicit annotation of uncertainty. This multi-layered protection ensures that the final output is factually robust, significantly reducing the incidence of hallucinations caused by knowledge gaps. This system demonstrates particularly high factual accuracy in scenarios ranging from open-domain question-answering to specialized consulting.

[0129] According to an embodiment of the present invention, the risk assessment information also includes scope assessment information for the initial generation result based on the information to be queried; the method also includes: determining the scope similarity between the query semantic scope of the information to be queried and the generation semantic scope of the initial generation result, so as to determine the scope assessment information based on the scope similarity and the length value.

[0130] In an embodiment of the present invention, the scope assessment information may be an assessment result obtained after assessing the scope suitability risk of the initial generation result, and is used to assess whether the initial generation result exceeds the scope of the information to be queried.

[0131] For example, the range evaluation information S of the initial generation result A It can be determined by the following formula (10):

[0132] (10);

[0133] Among them, Relevance(A,Q) can be used to evaluate the function of the overall relevance between the initial generated result set A and the query information Q, divided by Can be used to penalize overly long answers.

[0134] According to an embodiment of the present invention, the risk assessment information also includes certainty assessment information for the initial retrieval result based on the initial generation result; the method also includes: weighting the statement confidence and assessment result of the target statement in the initial generation result to obtain certainty assessment information.

[0135] In an embodiment of the present invention, the certainty evaluation information may be determined based on the generation confidence of the sentences in the initial generation result, and used to evaluate the uncertainty risk of the initial generation result with respect to the initial retrieval result.

[0136] For example, the initial generated result is the certainty evaluation information U of the initial retrieval result. A It can be determined by the following formula (11):

[0137] (11);

[0138] Among them, c i It can be the generation confidence of the i-th sentence in the initial sentence generation result, which indicates the confidence of the model in its output of this sentence. score It can be the quality score of the initial search results, reflecting the relevance and quality of the external knowledge supporting the answer. λ can be a weight coefficient that balances the contribution ratio of generation confidence and retrieval quality.

[0139] According to an embodiment of the present invention, the initial generation result is updated based on the risk assessment information and the update strategy of the initial generation result to obtain a target generation result, including: combining the first matching degree, the second matching degree, the support degree, the range assessment information and the certainty assessment information to obtain the risk assessment information; based on the comparison result between the risk assessment information and the risk threshold, determining the target update strategy from the update strategy to update the initial generation result using the target update strategy to obtain the target generation result.

[0140] In an embodiment of the present invention, risk assessment information may be information obtained by combining multi-dimensional assessment results. For the determined risk assessment information, the query system may trigger corresponding error correction and update strategies. The update strategy may include update sub-strategies for different risk assessment information.

[0141] For example, the answer verification module in the query system generates the initial result A init Multi-dimensional verification can be performed to obtain risk assessment information E A , as shown in the following formula (12):

[0142] (12);

[0143] Among them, F A It can be the first matching degree, H ACan be the support, C A Can be the second matching degree, S A Can evaluate information for range, U A Information can be evaluated for certainty.

[0144] For example, when the first matching degree F A Or support H A When the second matching degree C A When the risk assessment information E A When the risk assessment information E exceeds the acceptable range but is lower than the refusal threshold, the answer reconstruction strategy can be triggered to generate a more conservative but more reliable answer. A When the rejection threshold is exceeded, the query system confirms that it cannot provide a reliable answer and explains the reason. Further, after the update, the query system can generate the target generation result A final .

[0145] According to an embodiment of the present invention, the method also includes: combining the updated result obtained by updating the risk information in the current generation result with the non-risk information in the current generation result to obtain an intermediate generation result; if no risk information is identified in the intermediate generation result, the intermediate generation result is determined as the target generation result.

[0146] The risk information may be a high-risk result after evaluation in the currently generated result. It can be understood that the currently generated result may include a high-risk result part and a non-risk part.

[0147] To avoid relying solely on a single confidence level for evaluation and updating, unlike the traditional method of simply providing a single confidence score, the present invention uses a multi-dimensional error risk vector to finely characterize uncertainty and provide users with specific sources of uncertainty and potential impacts.

[0148] It can be understood that, unlike traditional one-time error correction, multi-round progressive error correction can gradually improve the quality of the answer (generated result) through multiple iterations, including: after the initial answer is generated, the system identifies the highest risk part P; the query system can perform error correction only on part P, generating a partially updated answer A'; thereby comparing A' with the initial generated result A init The low-risk parts in the merging are merged to form the updated generated result A update ; and then to A update Perform verification and update again. If the risk after update still exceeds the threshold, repeat the error correction process. When the verification passes or the maximum number of iterations is reached, the target generation result A can be output. finalThe process of iteratively generating answers is specifically shown in equation (13) as follows:

[0149] (13);

[0150] where t can represent the current iteration round, A t may be the answer of the current round (current generation result), A t+1 may be the generation result of the subsequent round after error correction update, may represent the high-risk part identified in the tth round, Correct() can represent the error correction function, and Merge() can represent the merging function.

[0151] For example, the query system can explicitly indicate that a certain part of the current generation result is uncertain due to "insufficient search information", "conflicting information", or "incomplete reasoning chain". This transparent expression of uncertainty not only improves the credibility of the system, but also provides users with more decision-making basis, especially suitable for high-risk decision-making scenarios such as financial analysis and medical diagnosis.

[0152] In a feasible embodiment, through adaptive learning and multi-round error correction functions, the query system can continuously optimize the verification and error correction capabilities from historical interactions.

[0153] For example, based on adaptive confidence calibration, the system can collect user feedback and labeled data to establish a confidence calibration model Cal to estimate uncertainty. The calibration model Cal is shown in equation (14) as follows:

[0154] (14);

[0155] where c can be the original confidence score, σ can be the Sigmoid function, and σ ensures that the output value is between [0, 1]. a and b are parameters learned by minimizing the following loss function, as shown in equation (15) as follows:

[0156] (15);

[0157] where, may be the cross-entropy loss function, c i may be the generation confidence of the ith statement, and yi can be the true label of the confidence yi∈{0,1}. It can be understood that by using the above cross-entropy loss function, the calibration model Cal is trained. The confidence of the model output generation result (after calibration) is compared with the true label, so that the calibration model outputs more "credible" probabilities as much as possible.

[0158] In one feasible embodiment, information query can be implemented based on the query system's self-verification and error correction. The query system may include a query analysis module, an adaptive retrieval module, a retrieval verification module, an answer generation module, and an answer verification and error correction module. The query analysis module can be used to receive the query information input by the user, analyze the query type, domain, and complexity, and provide a basis for subsequent retrieval strategy selection; the adaptive retrieval module can automatically select and configure the most suitable retrieval strategy combination based on the analysis results of the query information to perform the retrieval operation; the retrieval verification module can evaluate the quality, relevance, and completeness of the retrieval results, identifying potential retrieval failures and information gaps. The answer generation module can generate an initial generation result based on the verified target retrieval result and record the key basis and reasoning path during the generation process. The answer verification and error correction module can perform multi-dimensional verification of the generated initial generation result, identify possible errors, and trigger corresponding error correction strategies based on the error type to obtain the target generation result. The query system may also include a candidate information set, which is a retrieval knowledge base covering multiple domains and is used to generate query information.

[0159] Figure 3 A flow chart of a method for generating information based on self-verification and error correction of a query system according to an embodiment of the present invention is shown.

[0160] like Figure 3 As shown, the information generation method based on self-verification and error correction of the query system may include operations S301 to S311.

[0161] In operation S301, a user inputs information to be queried into a query analysis module through an electronic device.

[0162] In operation S302 , the query analysis module processes the information to be queried to obtain a feature vector, and sends the information to be queried and the feature vector to the adaptive retrieval module.

[0163] In operation S303 , the adaptive retrieval module selects an optimal retrieval strategy combination based on the candidate information set and performs a retrieval operation to obtain an initial retrieval result.

[0164] In operation S304, the initial search result is sent to the search verification module.

[0165] In operation S305, the retrieval verification module evaluates the retrieval quality and obtains the evaluation parameter Q R .

[0166] In operation S306, based on the evaluation parameter, the initial search result and the preset threshold, the evaluation parameter Q R Less than the threshold τ Q In this case, the retrieval enhancement strategy is triggered and the adaptive retrieval module is returned.

[0167] In operation S307, the updated and optimized search is performed to obtain an updated search result, which is returned to the search verification module.

[0168] In operation S308 , the search verification module sends the verified search results to the answer generation module.

[0169] In operation S309 , the answer generation module may generate an initial generation result, record status information during the generation process, and send the result to the answer verification and error correction module.

[0170] In operation S310 , the answer verification and error correction module generates target query information by calculating risk assessment information of the initial generation result and selecting a target update strategy from the update strategies based on the risk assessment information.

[0171] In operation S311, target query information is sent to the user.

[0172] Based on the above information generation method, the present invention also provides an information query device. Figure 4 The device is described in detail.

[0173] Figure 4 A structural block diagram of an information query device according to an embodiment of the present invention is shown.

[0174] like Figure 4 As shown, the information query device 400 of this embodiment includes a retrieval result determination module 410 , an evaluation result determination module 420 , a generation result determination module 430 and an update module 440 .

[0175] Retrieval result determination module 410 is configured to, in response to receiving the query information, obtain a plurality of initial retrieval results from the candidate information set based on characteristics of the query information, wherein the characteristics include at least one of the following: the complexity of the query information, the relevance between the query information and the domain information, the density characteristics of entities in the query information, and the type characteristics of the query information. In one embodiment, retrieval result determination module 410 may be configured to perform operation S210 described above, and will not be further described here.

[0176] The evaluation result determination module 420 is configured to obtain evaluation results corresponding to the multiple initial search results based on the evaluation parameters and evaluation strategy of the initial search results. In one embodiment, the evaluation result determination module 420 may be configured to perform the operation S220 described above, which will not be described in detail here.

[0177] Generation result determination module 430 is configured to obtain a target search result based on the evaluation result and the multiple initial search results, and to obtain an initial generation result based on the query information and the target search result. The target search result is a search result in the multiple initial search results whose evaluation result satisfies a preset evaluation condition. In one embodiment, generation result determination module 430 may be configured to perform operation S230 described above, and will not be further described herein.

[0178] The updating module 440 is configured to update the initial generation result based on the risk assessment information of the initial generation result and the updating strategy to obtain the target generation result. In one embodiment, the updating module 440 may be configured to execute the operation S240 described above, which will not be described in detail here.

[0179] According to an embodiment of the present invention, based on the retrieval result determination module 410, the evaluation result determination module 420, the generation result determination module 430 and the update module 440 in the information query device 400, since the initial retrieval result is determined by integrating the multi-dimensional features of the information to be queried, it meets the dynamic determination and adjustment mode of the retrieval result, so that the evaluation result of the initial retrieval result is flexibly determined according to the evaluation strategy, optimization strategy and evaluation parameters, and combined with the target retrieval result that meets the preset evaluation conditions, the initial generation result after evaluation is obtained. On this basis, possible errors are further identified according to the risk assessment information of the initial generation result to trigger the update strategy to update the initial generation result, and the target generation result that has undergone multi-layer evaluation and update optimization is obtained. By evaluating the retrieval results in the retrieval stage and updating the generation results in the generation stage, even if an error occurs in a certain link, it can be discovered and blocked in time, preventing the accumulation and expansion of errors. Especially for multi-hop reasoning and high-complexity scenarios, the error isolation mechanism significantly improves the overall stability and reliability of the system.

[0180] According to an embodiment of the present invention, the retrieval result determination module 410 includes: a feature determination submodule, a policy information determination submodule, and a retrieval submodule. The feature determination submodule is configured to determine features corresponding to the information to be queried; the policy information determination submodule is configured to determine target policy information from the policy information set based on the features; and the retrieval submodule is configured to perform a retrieval operation within the candidate information set based on the target policy information to obtain multiple initial retrieval results, wherein the multiple initial retrieval results include an initial text and an initial degree of association between the initial text and the information to be queried.

[0181] According to an embodiment of the present invention, the feature determination submodule includes at least one of the following: a complexity determination unit, a relevance determination unit, a density feature determination unit, and a combination unit. The complexity determination unit is used to determine the complexity based on the analysis results and recognition results of the information to be queried; the relevance determination unit is used to determine the relevance based on the domain association between the information to be queried and the domain information in the candidate information set; the density feature determination unit is used to determine the ratio between the number of entities in the information to be queried and the entity query length value as the density feature; and the combination unit is used to combine multiple type labels of the information to be queried to obtain the type feature.

[0182] According to an embodiment of the present invention, a policy information set includes multiple search policies and combinations of these policies. The policy information determination submodule includes a matching degree determination unit and a policy information determination unit. The matching degree determination unit is configured to determine the matching degree between a feature and the current policy information. The policy information determination unit is configured to determine target policy information from the policy information set based on the matching degree between the feature and the current policy information and the weight of the current policy information.

[0183] According to an embodiment of the present invention, the device further includes at least one of the following: a coverage determination module, a relevance determination module, a similarity determination module, and a confidence determination module. The coverage determination module is used to determine the information coverage of the initial retrieval result on the information to be queried based on the target sequence information, semantic similarity information, and respective corresponding weights between the information to be queried and the initial retrieval result, as the coverage in the evaluation parameter, wherein the target sequence information includes the target sequence that exists simultaneously in the information to be queried and the initial retrieval result, the frequency of occurrence of the target sequence, and the distribution information; the relevance determination module is used to determine the semantic relevance between the information to be queried and the initial retrieval result, and use the semantic relevance as the relevance in the evaluation parameter; the similarity determination module is used to convert multiple initial retrieval results into multiple word vectors respectively, so as to determine the vector similarity between the multiple word vectors as the similarity in the evaluation parameter; the confidence determination module is used to determine the evaluation confidence between the information to be queried and the initial retrieval result based on the type information of the information to be queried, the number of initial retrieval results, and the evaluation strategy, as the confidence in the evaluation parameter.

[0184] According to an embodiment of the present invention, the confidence determination module includes: a processing submodule for performing normalization processing based on the type information and the number of initial search results to obtain a confidence level; and a fusion submodule for fusing the reference confidence levels of the type information and the number of initial search results, multiple types of information at multiple time points, and multiple numbers of results to obtain a confidence level.

[0185] According to an embodiment of the present application, the result generation determining module 430 comprises: an information set updating submodule, a search result generation submodule, and an evaluation result determining submodule. The information set updating submodule is configured to update the policy information set to obtain updated policy information when the evaluation result is less than the evaluation threshold. The search result generation submodule is configured to generate a plurality of intermediate search results corresponding to the to-be-searched information by using the updated policy information. The evaluation result determining submodule is configured to take the intermediate search result as the target search result when the intermediate evaluation result of the intermediate search result is greater than or equal to the evaluation threshold.

[0186] According to an embodiment of the present application, the information set updating submodule comprises: an updating unit and a combination unit. The updating unit is configured to update the to-be-searched information, the domain information in the policy information set, and the search strategy respectively to obtain updated to-be-searched information, updated domain information, and updated search strategy. The combination unit is configured to combine the updated domain information, the updated search strategy, and the updated to-be-searched information to obtain the updated policy information.

[0187] According to an embodiment of the present application, the device further comprises: a threshold updating module configured to update the evaluation threshold based on the preference information of the user, the current relevance between the current to-be-searched information and the domain information to obtain an updated evaluation threshold, and update the evaluation result by using the updated evaluation threshold.

[0188] According to an embodiment of the present application, the result generation determining module 430 further comprises: a result generation submodule configured to generate an initial generation result based on the target search result, the type information of the to-be-searched information, the intention information, and the template information corresponding to the type information and the intention information respectively.

[0189] According to an embodiment of the present application, the risk evaluation information comprises a first matching degree between the initial generation result and the initial search result; and the device further comprises: a matching degree determining module configured to determine a semantic matching degree and an embedding matching degree between a query sequence of the initial generation result and a result sequence of the initial search result, combine the semantic matching degree and the embedding matching degree to obtain the first matching degree.

[0190] According to an embodiment of the present application, the initial generation result comprises a plurality of sub-information, and the risk evaluation information further comprises a second matching degree between the plurality of sub-information; and the device further comprises: a conversion module configured to determine a plurality of information matching degrees between the plurality of sub-information and a valley value matching degree in the plurality of information matching degrees, and convert the valley value matching degree to obtain the second matching degree.

[0191] According to an embodiment of the present application, the risk evaluation information further comprises a support degree of the initial search result to the initial generation result; and the device further comprises: a support degree determining module configured to determine the support degree based on a length value of the initial generation result and a support threshold.

[0192] According to an embodiment of the present invention, the risk assessment information also includes range assessment information for the initial generation result based on the information to be queried; the device also includes: a similarity determination module, used to determine the range similarity between the query semantic scope of the information to be queried and the generation semantic scope of the initial generation result, so as to determine the range assessment information based on the range similarity and the length value.

[0193] According to an embodiment of the present invention, the risk assessment information also includes certainty assessment information for the initial retrieval result based on the initial generation result; the device also includes: a weighting module, which is used to weight the statement confidence and assessment result of the target statement in the initial generation result to obtain certainty assessment information.

[0194] According to an embodiment of the present invention, update module 440 includes an information combination submodule and a strategy determination submodule. The information combination submodule is configured to combine the first matching degree, the second matching degree, the support degree, the range assessment information, and the certainty assessment information to obtain risk assessment information. The strategy determination submodule is configured to determine a target update strategy from the update strategies based on a comparison result between the risk assessment information and a risk threshold, and to update the initial generation result using the target update strategy to obtain a target generation result.

[0195] According to an embodiment of the present invention, the apparatus further includes: an information combination module and a generation result determination module. The information combination module is configured to combine an updated result obtained by updating risk information in a current generation result with non-risk information in the current generation result to obtain an intermediate generation result; and the generation result determination module is configured to determine the intermediate generation result as a target generation result if no risk information is identified in the intermediate generation result.

[0196] According to an embodiment of the present invention, any multiple modules among the search result determination module 410, the evaluation result determination module 420, the generation result determination module 430, and the update module 440 may be combined into a single module, or any one of these modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in a single module. According to an embodiment of the present invention, at least one of the search result determination module 410, the evaluation result determination module 420, the generation result determination module 430, and the update module 440 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented in hardware or firmware through any other reasonable means of circuit integration or packaging, or may be implemented in any one of the three implementation methods of software, hardware, and firmware, or any appropriate combination of any of these. Alternatively, at least one of the retrieval result determination module 410 , the evaluation result determination module 420 , the generation result determination module 430 and the update module 440 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0197] Figure 5 A block diagram of an electronic device according to an embodiment of the present invention is shown.

[0198] like Figure 5 As shown, the electronic device 500 includes a memory 501 and a processor 502 configured to execute the above information generation method according to the instructions and data stored in the memory.

[0199] The data stored in the memory 501 may include information to be queried, retrieval strategy information, retrieval result information of each stage, and generation result information. The instructions may include retrieval enhancement instructions and strategy update instructions.

[0200] Figure 6 A block diagram of a first electronic device suitable for implementing an information generating method according to an embodiment of the present invention is shown.

[0201] like Figure 6As shown, the first electronic device 600 according to an embodiment of the present application includes a first processor 601 which can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 602 or a program loaded from a storage section 608 into a random access memory (RAM) 603. The first processor 601 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a dedicated microprocessor (e.g., an application specific integrated circuit (ASIC)), and the like. The first processor 601 can also include an on-board memory for cache use. The first processor 601 can include a single processing unit or multiple processing units for executing different actions of the method processes according to embodiments of the present application.

[0202] In the RAM 603, various programs and data required for the operation of the first electronic device 600 are stored. The first processor 601, the ROM 602, and the RAM 603 are connected to each other by a bus 604. The first processor 601 performs various operations of the method processes according to embodiments of the present application by executing the programs in the ROM 602 and / or the RAM 603. Note that the programs can also be stored in one or more memories other than the ROM 602 and the RAM 603. The first processor 601 can also perform various operations of the method processes according to embodiments of the present application by executing the programs stored in the one or more memories.

[0203] According to an embodiment of the present application, the first electronic device 600 can further include an input / output (I / O) interface 605 which is also connected to the bus 604. The first electronic device 600 can further include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, a mouse, and the like; an output section 607 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage section 608 including a hard disk, and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, and the like. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as necessary. A removable recording medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is attached to the drive 610 as necessary, so that a computer program read therefrom is installed into the storage section 608 as necessary.

[0204] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0205] According to an embodiment of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present invention, a computer-readable storage medium may include the ROM 602 and / or RAM 603 described above, and / or one or more memories other than ROM 602 and RAM 603.

[0206] The embodiments of the present invention further include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to cause the computer system to implement the method provided by the embodiments of the present invention.

[0207] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when the first processor 601 executes the computer program. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0208] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0209] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609 and / or installed from a removable medium 611. When the computer program is executed by the first processor 601, the above-described functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0210] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0211] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0212] It will be understood by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or couplings fall within the scope of the present invention.

[0213] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. An information generation method, characterized in that: The method comprises: In response to receiving the information to be queried, obtaining a plurality of initial search results from the candidate information set based on characteristics of the information to be queried, wherein the characteristics include at least one of the following: complexity of the information to be queried, relevance between the information to be queried and domain information, density characteristics of entities in the information to be queried, and type characteristics of the information to be queried; Obtaining evaluation results corresponding to the plurality of initial search results according to the evaluation parameters and evaluation strategy of the initial search results; Obtaining a target search result based on the evaluation result and the multiple initial search results, and obtaining an initial generation result based on the information to be queried and the target search result, wherein the target search result is a search result among the multiple initial search results whose evaluation result satisfies a preset evaluation condition; Update the initial generation result based on the risk assessment information and update strategy of the initial generation result to obtain a target generation result; Wherein, based on the evaluation result and the multiple initial search results, a target search result is obtained, including: When the evaluation result is less than the evaluation threshold, updating the policy information set to obtain updated policy information; generating a plurality of intermediate search results corresponding to the information to be queried using the updated strategy information; When the intermediate evaluation result of the intermediate search result is greater than or equal to the evaluation threshold, the intermediate search result is used as the target search result.

2. The method according to claim 1, characterized in that Based on the characteristics of the information to be queried, multiple initial search results are obtained from the candidate information set, including: Determining a feature corresponding to the information to be queried; determining target policy information from the policy information set based on the characteristics; A search operation is performed in the candidate information set based on the target policy information to obtain the multiple initial search results, wherein the multiple initial search results include initial text and initial association between the initial text and the information to be queried.

3. The method according to claim 2, characterized in that Determining a feature corresponding to the information to be queried includes at least one of the following: Determining the complexity based on the analysis result and the recognition result of the information to be queried; Determining the relevance based on the domain association between the information to be queried and the domain information in the candidate information set; Determine the ratio between the number of entities in the information to be queried and the entity query length value as the density feature; The multiple type tags of the information to be queried are combined to obtain the type feature.

4. The method according to claim 2, characterized in that The strategy information set includes a plurality of search strategies and a combination strategy between the plurality of search strategies; Determining target policy information from the policy information set based on the feature includes: determining a degree of match between the feature and current policy information; The target policy information is determined from the policy information set based on a matching degree between the feature and the current policy information and a weight of the current policy information.

5. The method according to claim 1, wherein The method further comprises at least one of the following: Determining information coverage of the information to be queried by the initial search result based on target sequence information, semantic similarity information between the information to be queried and the initial search result, and weights corresponding to the target sequence information and the semantic similarity information, as the coverage in the evaluation parameter, wherein the target sequence information includes target sequences that exist simultaneously in the information to be queried and the initial search result, and occurrence frequency and distribution information of the target sequences; Determining the semantic relevance between the information to be queried and the initial search results, and using the semantic relevance as the relevance in the evaluation parameter; Converting the plurality of initial search results into a plurality of word vectors respectively, and determining the vector similarity between the plurality of word vectors as the similarity in the evaluation parameter; Based on the type information of the information to be queried, the number of the initial search results and the evaluation strategy, an evaluation confidence between the information to be queried and the initial search results is determined as the confidence in the evaluation parameter.

6. The method according to claim 5, characterized in that Determining, based on the type information of the information to be queried, the number of the initial search results, and an evaluation strategy, an evaluation confidence level between the information to be queried and the initial search results, including: performing normalization processing based on the type information and the number of the initial search results to obtain the confidence level; or The confidence is obtained by fusing the reference confidences of the type information and the number of the initial search results, multiple types of information at multiple moments, and multiple numbers of results.

7. The method according to claim 1, characterized in that Updating the policy information set to obtain updated policy information includes: The information to be queried, the domain information in the policy information set, and the retrieval strategy are updated respectively to obtain updated information to be queried, updated domain information, and updated retrieval strategy; The updated domain information, the updated search strategy and the updated to-be-queried information are combined to obtain the updated strategy information.

8. The method according to claim 1, characterized in that The method further comprises: The evaluation threshold is updated based on the user's preference information and the current correlation between the current information to be queried and the domain information to obtain an updated evaluation threshold, so as to update the evaluation result using the updated evaluation threshold.

9. The method according to claim 1, characterized in that Based on the information to be queried and the target search results, an initial generation result is obtained, including: The initial generation result is generated based on the target retrieval result, the type information of the information to be queried, the intent information, and the template information corresponding to the type information and the intent information respectively.

10. The method according to claim 1, characterized in that The risk assessment information includes a first matching degree between the initial generation result and the initial search result; the method further includes: Determine a semantic match and an embedding match between the query sequence of the initial generated result and the result sequence of the initial search result, and combine the semantic match and the embedding match to obtain the first match.

11. The method according to claim 10, characterized in that The initial generation result includes a plurality of sub-information, and the risk assessment information further includes a second matching degree between the plurality of sub-information; the method further includes: A plurality of information matching degrees between the plurality of sub-information and a valley matching degree among the plurality of information matching degrees are determined, so as to convert the valley matching degrees to obtain the second matching degree.

12. The method according to claim 11, characterized in that The risk assessment information also includes the support degree of the initial search result to the initial generated result; the method further includes: The support is determined based on the length value of the initial generation result and a support threshold.

13. The method according to claim 12, characterized in that The risk assessment information further includes range assessment information for the initial generation result based on the information to be queried; and the method further includes: The scope similarity between the query semantic scope of the information to be queried and the generated semantic scope of the initial generated result is determined, so as to determine the scope evaluation information based on the scope similarity and the length value.

14. The method according to claim 13, characterized in that The risk assessment information further includes certainty assessment information for the initial search result based on the initial generation result; the method further includes: The sentence confidence of the target sentence in the initial generation result and the evaluation result are weighted respectively to obtain the deterministic evaluation information.

15. The method according to claim 14, characterized in that Updating the initial generation result based on the risk assessment information and the update strategy of the initial generation result to obtain a target generation result includes: combining the first matching degree, the second matching degree, the support degree, the range assessment information, and the certainty assessment information to obtain the risk assessment information; Based on the comparison result between the risk assessment information and the risk threshold, a target update strategy is determined from the update strategies, so as to update the initial generation result by using the target update strategy to obtain the target generation result.

16. The method according to claim 9, characterized in that The method further comprises: combining an updated result obtained by updating the risk information in the current generation result with the non-risk information in the current generation result to obtain an intermediate generation result; In a case where no risk information is identified in the intermediate generation result, the intermediate generation result is determined as the target generation result.

17. An electronic device, characterized in that: include: Memory; A processor configured to execute the method according to any one of claims 1 to 16 according to the instructions and data stored in the memory.

18. A non-volatile computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instructions are executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

19. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instructions are executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

Images