Content matching method, device and computer equipment generated by large language model

Abstract

The application relates to a large language model content generation matching method and device and a computer device. The method comprises the following steps: obtaining model response content generated by a large language model and a candidate document segment set corresponding to the model response content; determining similarity characteristic values between the model response content and the candidate document segments in multiple dimensions; determining weights corresponding to vector semantic characteristic values, keyword retrieval characteristic values and lexical overlap characteristic values according to quantity characteristics and distribution characteristics; processing the vector semantic characteristic values, the keyword retrieval characteristic values and the lexical overlap characteristic values according to the weights to obtain matching scores corresponding to the candidate document segments; determining a target threshold value for screening the candidate document segments based on statistical characteristics; and labeling candidate document segments satisfying the target threshold value as reference sources of the model response content. The method can improve the reference accuracy of the large language model content.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and information retrieval technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for matching content generated by a large language model. Background Technology

[0002] In the application scenarios of large language models, automatically labeling the sources of generated text fragments is a key step in improving the credibility of the content. Specifically, citation annotation aims to provide a highly credible basis for the content generated by large language models. It can force the cited content to have an objective literal source, which can effectively reduce and suppress the risk of large language models generating "illusions".

[0003] Currently, when matching source citations to text generated by large language models, a single similarity calculation method is usually used. For example, the query text and the document library text are converted into vectors, and the cosine similarity between them is calculated. If the similarity score is higher than a preset fixed threshold, it is considered a successful match. This single-dimensional matching strategy is difficult to guarantee high-precision matching and is prone to mismatches or missed matches. This causes the large language model to cite irrelevant document fragments when outputting content, affecting the reference value of the final presented citation information and resulting in the problem of low citation accuracy of content generated by large language models. Summary of the Invention

[0004] Therefore, it is necessary to provide a matching method, apparatus, computer device, computer-readable storage medium, and computer program product for large language model-generated content that can improve the citation accuracy of large language model-generated content in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a matching method for content generated by a large language model, including:

[0006] Obtain the model response content generated by the large language model, and the candidate document fragment set corresponding to the model response content; the candidate document fragment set includes multiple candidate document fragments;

[0007] Determine the similarity feature values ​​between the model response content and the candidate document fragment in multiple dimensions; the similarity feature values ​​in multiple dimensions include at least vector semantic feature values, keyword retrieval feature values, and word overlap feature values;

[0008] Obtain the quantity features of the multiple candidate document fragments and the distribution features of each of the similarity feature values, and determine the weights corresponding to the vector semantic feature value, the keyword retrieval feature value and the word overlap feature value based on the quantity features and the distribution features;

[0009] Based on the weights, the vector semantic feature values, the keyword retrieval feature values, and the word overlap feature values ​​are processed to obtain the matching scores corresponding to the candidate document segments, and statistical features of the matching scores corresponding to each candidate document segment are extracted;

[0010] Based on the statistical features, a target threshold is determined for filtering the candidate document fragments, and candidate document fragments whose matching scores meet the target threshold are labeled as reference sources of the model response content.

[0011] In one embodiment, determining the similarity feature values ​​between the model response content and the candidate document fragment across multiple dimensions includes:

[0012] The semantic feature value of the vector is determined based on the vector similarity between the feature vectors corresponding to the model response content and the candidate document fragments.

[0013] Extract corresponding keywords from the model response content, and determine the keyword retrieval feature value based on the word frequency and inverse document frequency of the keywords in the candidate document fragment;

[0014] The word overlap feature value is determined based on the degree of literal overlap between the model response content and the candidate document fragment;

[0015] Based on the vector semantic feature value, the keyword retrieval feature value, and the word overlap feature value, the similarity feature values ​​of the multiple dimensions are determined.

[0016] In one embodiment, determining the weights corresponding to the vector semantic feature value, the keyword retrieval feature value, and the word overlap feature value based on the quantitative features and the distribution features includes:

[0017] The preset weights are determined as the weights corresponding to the word overlap feature values;

[0018] Based on the quantitative characteristics, determine the basic weights corresponding to the vector semantic feature values ​​and the keyword retrieval feature values, respectively;

[0019] Based on the distribution characteristics, the basic weights are adjusted to obtain the weights corresponding to the vector semantic feature values ​​and the keyword retrieval feature values.

[0020] In one embodiment, determining the basic weights corresponding to the vector semantic feature value and the keyword retrieval feature value based on the quantitative features includes:

[0021] If the number of candidate document fragments indicated by the quantity feature is less than the first quantity threshold, the basic weight corresponding to the vector semantic feature value is determined as the first weight value, and the basic weight corresponding to the keyword retrieval feature value is determined as the second weight value.

[0022] When the number of candidate document fragments is greater than or equal to the second quantity threshold, the basic weight corresponding to the vector semantic feature value is determined as the third weight value, and the basic weight corresponding to the keyword retrieval feature value is determined as the fourth weight value; wherein, the second quantity threshold is greater than the first quantity threshold, the third weight value is less than the first weight value, and the fourth weight value is greater than the second weight value;

[0023] When the number of candidate document fragments is greater than or equal to the first quantity threshold and less than the second quantity threshold, interpolation is performed between the first weight value and the third weight value to determine the basic weight corresponding to the vector semantic feature value, and interpolation is performed between the second weight value and the fourth weight value to determine the basic weight corresponding to the keyword retrieval feature value.

[0024] In one embodiment, the distribution characteristics include standard deviation and mean; adjusting the basic weights according to the distribution characteristics includes:

[0025] If the standard deviation corresponding to the vector semantic feature value is greater than the standard deviation corresponding to the keyword retrieval feature value, increase the base weight of the vector semantic feature value and decrease the base weight of the keyword retrieval feature value.

[0026] If the standard deviation corresponding to the keyword retrieval feature value is greater than the standard deviation corresponding to the vector semantic feature value, increase the base weight of the keyword retrieval feature value and decrease the base weight of the vector semantic feature value.

[0027] If the mean value of the vector semantic feature value is greater than a preset mean threshold, and the mean value of the keyword retrieval feature value is less than or equal to the preset mean threshold, increase the base weight of the vector semantic feature value and decrease the base weight of the keyword retrieval feature value.

[0028] If the mean value of the keyword retrieval feature value is greater than the preset mean threshold, and the mean value of the vector semantic feature value is less than or equal to the preset mean threshold, the basic weight of the keyword retrieval feature value is increased, and the basic weight of the vector semantic feature value is decreased.

[0029] In one embodiment, the statistical features include the highest score, the lowest score, the standard deviation, and the mean; determining the target threshold for filtering the candidate document fragments based on the statistical features includes:

[0030] If the highest score is less than the first extreme threshold, the initial threshold for filtering the candidate document fragments is lowered; if the lowest score is greater than the second extreme threshold, the initial threshold is raised; the second extreme threshold is greater than the first extreme threshold.

[0031] If the standard deviation is greater than a preset discrete threshold, the initial threshold is decreased; if the standard deviation is less than a preset clustered threshold, the initial threshold is increased; the preset clustered threshold is less than the preset discrete threshold.

[0032] If the mean is greater than a first mean threshold, the initial threshold is increased; if the mean is less than a second mean threshold, the initial threshold is decreased; the second mean threshold is less than the first mean threshold.

[0033] The target threshold is determined based on the adjusted initial threshold.

[0034] Secondly, this application also provides a matching device for content generated by a large language model, comprising:

[0035] The acquisition module is used to acquire the model response content generated by the large language model, and the candidate document fragment set corresponding to the model response content; the candidate document fragment set includes multiple candidate document fragments.

[0036] The determination module is used to determine the similarity feature values ​​between the model response content and the candidate document fragment in multiple dimensions; the similarity feature values ​​in multiple dimensions include at least vector semantic feature values, keyword retrieval feature values, and word overlap feature values;

[0037] The determining module is further configured to obtain the quantity features of the plurality of candidate document fragments and the distribution features of each of the similarity feature values, and, based on the quantity features and the distribution features, determine the weights corresponding to the vector semantic feature value, the keyword retrieval feature value and the word overlap feature value, respectively.

[0038] The calculation module is used to process the vector semantic feature value, the keyword retrieval feature value and the word overlap feature value according to the weight to obtain the matching score corresponding to the candidate document fragment, and to extract the statistical features of the matching score corresponding to each candidate document fragment;

[0039] The annotation module is used to determine a target threshold for filtering the candidate document fragments based on the statistical features, and to annotate the candidate document fragments whose matching scores meet the target threshold as the reference sources of the model response content.

[0040] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0041] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0042] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0043] The aforementioned matching method, apparatus, computer device, computer-readable storage medium, and computer program product for content generated by the large language model acquire the model response content generated by the large language model, and a set of candidate document fragments corresponding to the model response content, the set of candidate document fragments including multiple candidate document fragments; determine the similarity feature values ​​between the model response content and the candidate document fragments in multiple dimensions, the multiple similarity feature values ​​including at least vector semantic feature values, keyword retrieval feature values, and word overlap feature values; acquire the quantity features of multiple candidate document fragments and the distribution features of each similarity feature value, and, based on the quantity features and distribution features, determine the weights corresponding to the vector semantic feature values, keyword retrieval feature values, and word overlap feature values ​​respectively; process the vector semantic feature values, keyword retrieval feature values, and word overlap feature values ​​according to the weights to obtain the matching scores corresponding to the candidate document fragments, and extract the statistical features of the matching scores corresponding to each candidate document fragment; determine the target threshold for filtering candidate document fragments based on the statistical features, and mark the candidate document fragments whose matching scores meet the target threshold as the reference sources of the model response content. In this way, by using similarity feature values ​​from multiple dimensions such as vector semantics, keyword retrieval, and word overlap, the candidate document fragments are weighted and scored. The weights of each feature value are dynamically determined based on the quantity characteristics of the candidate document fragment set and the distribution characteristics of each similarity feature value. The screening threshold is adaptively determined based on the statistical characteristics of the matching scores of each candidate document fragment. This allows the matching process to take into account semantic understanding, keyword differentiation, and literal verification, avoiding the mismatch and missed match problems caused by single-dimensional matching and fixed thresholds. This achieves high-precision and high-robustness content matching, thereby improving the citation accuracy of the content generated by the large language model and reducing the risk of "illusion" caused by citation errors. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is an application environment diagram of a matching method for content generated by a large language model in one embodiment;

[0046] Figure 2 This is a flowchart illustrating a content matching method generated by a large language model in one embodiment.

[0047] Figure 3 This is a flowchart illustrating the overall processing flow of a content matching method generated by a large language model in one embodiment.

[0048] Figure 4 This is a logic diagram for dynamically adjusting the weights of similarity feature values ​​in one embodiment;

[0049] Figure 5 This is a structural block diagram of a matching device for content generated by a large language model in one embodiment;

[0050] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] The matching method for content generated by a large language model provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.

[0053] Terminal 102 acquires the model response content generated by the large language model, and a set of candidate document fragments corresponding to the model response content; the set of candidate document fragments includes multiple candidate document fragments; terminal 102 determines the similarity feature values ​​between the model response content and the candidate document fragments in multiple dimensions; the similarity feature values ​​in multiple dimensions include at least vector semantic feature values, keyword retrieval feature values, and word overlap feature values; terminal 102 acquires the quantity features of multiple candidate document fragments and the distribution features of each similarity feature value, and, based on the quantity features and distribution features, determines the weights corresponding to the vector semantic feature values, keyword retrieval feature values, and word overlap feature values ​​respectively; terminal 102 processes the vector semantic feature values, keyword retrieval feature values, and word overlap feature values ​​according to the weights to obtain the matching scores corresponding to the candidate document fragments, and extracts the statistical features of the matching scores corresponding to each candidate document fragment; terminal 102 determines the target threshold for filtering candidate document fragments based on the statistical features, and marks the candidate document fragments whose matching scores meet the target threshold as the reference sources of the model response content.

[0054] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. The server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0055] In one exemplary embodiment, such as Figure 2 As shown, a matching method for content generated by a large language model is provided, and this method is applied to... Figure 1 Taking terminal 102 as an example, the explanation includes:

[0056] Step S202: Obtain the model response content generated by the large language model, and the set of candidate document fragments corresponding to the model response content.

[0057] Among them, the Large Language Model (LLM) refers to a deep learning model with powerful text generation capabilities. It is trained on massive text corpora and can generate coherent and semantically complete text content based on the input information.

[0058] The model response content refers to the text content generated by the large language model that requires citation source annotation. The model response content can be either the complete answer text generated by the large language model in response to the user's question, or text units such as sentences and paragraphs divided from the answer text; this application does not limit this.

[0059] In this context, a candidate document chunk refers to a basic text segment unit obtained by pre-segmenting the original document for retrieval and matching. The original document can be various types of documents, such as documents, web pages, and reports, collected in a knowledge base. When segmenting the original document, it can be divided into multiple document chunks according to rules such as sentences, paragraphs, or preset length windows.

[0060] In practice, after obtaining the model response content, a preliminary search can be performed on the model response content to recall multiple document fragments that are semantically related to the model response content from the document library, forming a candidate document fragment set. This preliminary search can be implemented, for example, based on vector retrieval, that is, calculating the vector similarity between the model response content and each document fragment in the document library, and recalling several document fragments with the highest vector similarity ranking as candidates.

[0061] The candidate document fragment set includes multiple candidate document fragments. The source, length, and number of candidate document fragments in the candidate document fragment set can be set according to the actual retrieval requirements.

[0062] For example, when a large language model generates an answer text containing multiple arguments in response to a user's question, the terminal can use the answer text as the model's response content, and determine several potentially related document fragments from the document library through preliminary retrieval to form a candidate document fragment set, so that the document fragments that can actually be used as citation sources can be further filtered from these candidate document fragments.

[0063] Step S204: Determine the similarity feature values ​​between the model response content and the candidate document fragments in multiple dimensions.

[0064] In practice, the similarity between the model's response content and candidate document fragments can be measured from multiple dimensions, rather than relying solely on a single-dimensional similarity calculation. These multiple-dimensional similarity features include at least vector semantic features, keyword retrieval features, and lexical overlap features.

[0065] Among them, the vector semantic feature value is used to measure the proximity between the model response content and the candidate document fragment from a semantic perspective. That is, it is determined by mapping the text to a vector space and then determining the similarity between the corresponding vectors of the text.

[0066] Among them, the keyword retrieval feature value is used to measure the hit rate of keywords in the model response content in candidate document fragments from the keyword level.

[0067] Among them, the word overlap feature value is used to measure the degree of overlap between the model response content and the candidate document fragments at the literal level.

[0068] The similarity feature values ​​of the above three dimensions describe the degree of matching between the two from the perspectives of semantics, keywords and literal meaning, respectively. They can complement each other and thus more comprehensively reflect the true similarity relationship between the model response content and the candidate document fragments.

[0069] By making comprehensive judgments from multiple dimensions, even if the model's response content and the candidate document fragment are semantically similar but have different key entities (such as time, location, amount, etc.), or are literally similar but have different meanings, they can still be identified through mutual verification between multiple dimensions, reducing the risk of mismatch or missed match caused by single-dimensional matching.

[0070] Step S206: Obtain the quantity features of multiple candidate document fragments and the distribution features of each similarity feature value; and, based on the quantity features and distribution features, determine the weights corresponding to the vector semantic feature value, keyword retrieval feature value, and word overlap feature value.

[0071] Among them, the quantitative feature refers to the feature presented by the number of candidate document fragments in the candidate document fragment set, such as the number of candidate document fragments.

[0072] Among them, distribution characteristics refer to the characteristics presented by the distribution of the values ​​of each similarity feature across multiple candidate document segments, such as the mean, standard deviation, and other statistical measures of each similarity feature value.

[0073] In practice, after obtaining the similarity feature values ​​for each dimension, a fixed weight is not used to simply weight each dimension. Instead, the weights for each dimension are dynamically determined based on the number of candidate document fragments and the distribution of each similarity feature value. This is because when the number of candidate document fragments or the distribution of each similarity feature value changes, the distinguishing ability of each dimension in the current matching scenario will also change. Dynamically determined weights can give higher weights to dimensions with stronger distinguishing abilities, thereby improving the accuracy of the matching results.

[0074] Step S208: Based on the weights, process the vector semantic feature values, keyword retrieval feature values, and word overlap feature values ​​to obtain the matching scores corresponding to the candidate document fragments, and extract the statistical features of the matching scores corresponding to each candidate document fragment.

[0075] In practice, the vector semantic feature value, keyword retrieval feature value, and word overlap feature value are processed according to the weights. This can be done by multiplying the vector semantic feature value, keyword retrieval feature value, and word overlap feature value of the candidate document fragment by their respective weights and then summing them to obtain the matching score corresponding to the candidate document fragment.

[0076] The matching score is used to characterize the overall matching degree between the candidate document fragment and the model response content. The higher the matching score, the better the candidate document fragment matches the model response content.

[0077] Among them, statistical features refer to the statistics of the score set consisting of the matching scores of all candidate document fragments, such as the highest score, lowest score, standard deviation, and mean of the score set.

[0078] Step S210: Determine the target threshold for filtering candidate document fragments based on statistical features, and mark the candidate document fragments whose matching scores meet the target threshold as the reference sources of the model response content.

[0079] The target threshold is a filtering condition used to select the final citation sources from candidate document fragments. This target threshold is not a pre-set fixed value, but is dynamically determined based on the statistical characteristics of the matching scores corresponding to each candidate document fragment. In this way, the target threshold can adapt to the actual distribution of matching scores in different matching scenarios, avoiding misselection caused by a fixed threshold when the overall matching scores are too high or too low. For example, when the overall matching scores are low, using a high fixed threshold might filter out all actually relevant candidate document fragments; conversely, when the overall matching scores are high, using a low fixed threshold might misidentify a large number of actually irrelevant candidate document fragments as citation sources.

[0080] After determining the target threshold, candidate document fragments whose matching scores meet the target threshold can be labeled as sources cited in the model's response content. Meeting the target threshold means the matching score is greater than or equal to the target threshold. These labeled candidate document fragments serve as the basis for the model's response content and can be output along with the model's response content for users to verify the source of the model's response content, thereby improving the credibility of the generated content.

[0081] In an exemplary embodiment, to improve the processing efficiency of the matching process, the above steps can be executed in parallel. Specifically, the content and metadata of multiple candidate document fragments can be acquired in parallel, as can the model response content and the embedding vectors corresponding to each candidate document fragment. Vector semantic similarity and keyword retrieval scores can also be calculated in parallel. Here, embedding vectors refer to the technique of mapping high-dimensional discrete data such as text to a low-dimensional continuous vector space, making semantically similar content close in distance within the vector space. Simultaneously, the acquired embedding vectors and document content can be cached in a caching device (e.g., Redis) for direct reading in subsequent matching processes, reducing redundant computation. Through the above parallelization and caching mechanisms, a high processing speed can be maintained even in scenarios with a large number of candidate document fragments, meeting the application requirements of high concurrency and low latency.

[0082] In the above-mentioned matching method for content generated by the large language model, the candidate document fragments are weighted and scored using similarity feature values ​​from multiple dimensions such as vector semantics, keyword retrieval, and word overlap. The weights of each feature value are dynamically determined based on the quantity characteristics of the candidate document fragment set and the distribution characteristics of each similarity feature value. The screening threshold is adaptively determined based on the statistical characteristics of the matching scores of each candidate document fragment. This allows the matching process to take into account semantic understanding, keyword differentiation, and literal verification, avoiding the mismatch and missed match problems caused by single-dimensional matching and fixed thresholds. This achieves high-precision and high-robustness content matching, thereby improving the citation accuracy of the content generated by the large language model and reducing the risk of "illusion" caused by citation errors.

[0083] In another embodiment, determining the similarity feature values ​​between the model response content and the candidate document fragments in multiple dimensions includes: determining vector semantic feature values ​​based on the vector similarity between the feature vectors corresponding to the model response content and the candidate document fragments; extracting corresponding keywords from the model response content and determining keyword retrieval feature values ​​based on the word frequency and inverse document frequency of the keywords in the candidate document fragments; determining lexical overlap feature values ​​based on the degree of literal overlap between the model response content and the candidate document fragments; and determining similarity feature values ​​in multiple dimensions based on the vector semantic feature values, keyword retrieval feature values, and lexical overlap feature values.

[0084] In practical implementation, the semantic feature values ​​of vectors can be determined based on the vector similarity between the feature vectors corresponding to the model response content and the candidate document fragments. Specifically, a pre-trained semantic embedding model can be used to convert the model response content and each candidate document fragment into high-dimensional feature vectors. The semantic embedding model can map semantically similar text to positions with close proximity in the vector space, thus ensuring that semantically similar content is close in the vector space. For example, the semantic embedding model can be the BGE-M3 model. After obtaining the feature vectors corresponding to the model response content and the candidate document fragments, the vector similarity between them is calculated to obtain the vector semantic feature values. Here, the vector similarity can be the cosine similarity between two feature vectors. To improve computational efficiency, the feature vectors of the model response content and all candidate document fragments can be organized into a matrix, and batch matrix operations can be used to calculate the cosine similarity between the model response content and each candidate document fragment at once, resulting in a set of vector scores.

[0085] In practice, keyword retrieval feature values ​​can be obtained by extracting corresponding keywords from the model's response content and determining them based on the term frequency (TF) and inverse document frequency (IDF) of the keywords in candidate document fragments. TF measures the frequency of a keyword's occurrence in candidate document fragments, while IDF measures the general importance of the keyword; the more frequently a keyword appears in a few document fragments, the higher its IDF, and the more valuable it is for distinguishing document fragments. By combining TF and IDF, the matching degree between the keywords in the model's response content and candidate document fragments can be evaluated, resulting in keyword retrieval feature values. For example, keyword retrieval feature values ​​can be calculated using the BM25 (Best Match 25) ranking function. BM25 is a widely used ranking function in information retrieval to evaluate the relevance of a document to a given search query, calculated based on TF and IDF. After calculating the original BM25 score, the original score can be normalized, for example by using Min-Max normalization or Sigmoid normalization, to map the original score to a uniform range of values, thus obtaining a set of keyword scores.

[0086] In practice, the lexical overlap feature value can be determined based on the degree of literal overlap between the model's response content and the candidate document fragments. Literal overlap measures the degree of overlap between the model's response content and the candidate document fragments in terms of literal expression. For example, the ROUGE-L metric, based on the Longest Common Subsequence (LCS), can be preferentially used for calculation. ROUGE-L is a commonly used metric for evaluating the quality of automatic summarization, calculating the F1 score based on the longest common subsequence between the generated text and the reference text, thus obtaining a set of lexical overlap scores. If ROUGE-L calculation is unavailable, Jaccard similarity can be used as a fallback to calculate the literal overlap between the model's response content and the candidate document fragments. Literal overlap reflects whether the model's response content and the candidate document fragments have a direct literal correspondence. By introducing this dimension, it is possible to enforce that the cited content has an objective literal source, thereby suppressing the "illusion" generated by large language models.

[0087] After obtaining the vector semantic feature value, keyword retrieval feature value, and word overlap feature value, similarity feature values ​​in multiple dimensions can be determined based on these three.

[0088] Optionally, word overlap features can also be calculated using edit distance, which measures the minimum number of editing operations required to transform one piece of text into another; keyword retrieval features can also be calculated using TF-IDF. All of these alternative calculation methods can determine the similarity feature values ​​for the corresponding dimensions.

[0089] For example, the model response content is "Artificial intelligence is increasingly widely used in the medical field," and the candidate document fragment is "Artificial intelligence technology is becoming more and more common in the medical field." While not exactly identical in wording, they are semantically similar, resulting in high vector semantic feature values. Keywords such as "artificial intelligence," "medical," and "application" all appear in the candidate document fragment, leading to high keyword retrieval feature values. Furthermore, there is significant word overlap, resulting in high lexical overlap feature values. By comprehensively judging these three dimensions, the relevance of the candidate document fragment to the model response content can be identified relatively accurately.

[0090] The technical solution of this embodiment calculates similarity feature values ​​from three dimensions: vector semantics, keyword retrieval, and word overlap. This enables the matching process to capture deep semantic information while also taking into account keyword hits and literal verification, thus solving the problem that single-dimensional matching cannot guarantee accuracy.

[0091] In another embodiment, the weights corresponding to the vector semantic feature value, the keyword retrieval feature value, and the word overlap feature value are determined based on the quantitative and distribution characteristics. This includes: determining a preset weight as the weight corresponding to the word overlap feature value; determining the basic weights corresponding to the vector semantic feature value and the keyword retrieval feature value based on the quantitative characteristics; and adjusting each basic weight based on the distribution characteristics to obtain the weights corresponding to the vector semantic feature value and the keyword retrieval feature value.

[0092] In practical implementation, the preset weights can be determined as the weights corresponding to the word overlap feature values. That is, the weights of the word overlap feature values ​​are fixed values ​​set in advance and are not adjusted according to changes in the number and distribution of candidate document fragments. For example, to prevent the large language model from generating "illusions," it can be enforced that the cited content has an objective literal source, and the weights of the word overlap feature values ​​can be fixed at a preset weight, for example, 0.1. Since the word overlap feature values ​​reflect the direct literal correspondence between the model's response content and the candidate document fragments, fixing their weights to a relatively stable value ensures that the matching results are always based on objective literal sources.

[0093] In practice, the basic weights of vector semantic feature values ​​and keyword retrieval feature values ​​can be determined based on the quantitative characteristics. These basic weights are the initial weight values ​​before subsequent adjustments. When the number of candidate document fragments is less than a preset threshold, the available candidate information for matching is limited, requiring greater reliance on semantic understanding to identify relevant fragments; therefore, the basic weights of vector semantic feature values ​​are relatively high. When the number of candidate document fragments reaches or exceeds the preset threshold, keywords can more effectively distinguish candidate fragments; therefore, the basic weights of keyword retrieval feature values ​​are relatively high.

[0094] In practice, the basic weights can be adjusted based on distribution characteristics to obtain the weights corresponding to the vector semantic feature values ​​and the keyword retrieval feature values. Distribution characteristics reflect the dispersion and overall level of each similarity feature value across the candidate document fragment set. If a certain similarity feature value has stronger discriminative power, for example, a larger standard deviation, its basic weight can be appropriately increased; conversely, it can be appropriately decreased. Through these adjustments, the final weights corresponding to the vector semantic feature values ​​and the keyword retrieval feature values ​​are obtained.

[0095] In an exemplary embodiment, quality protection can also be implemented for keyword retrieval feature values. Specifically, if the keyword retrieval feature value of a candidate document fragment is lower than a preset invalidity threshold (e.g., lower than 0.1), the keyword match of the candidate document fragment is determined to be invalid. When calculating the matching score of the candidate document fragment, its keyword retrieval feature value is ignored, and this part of the weight is redistributed to the vector semantic feature value and the word overlap feature value (e.g., allocated to 90% and 10% respectively). In this way, interference with the matching results due to keyword retrieval failure can be avoided.

[0096] The technical solution of this embodiment fixes the weights of word overlap feature values, determines the basic weights based on quantity features, and adjusts the basic weights based on distribution features, so that the weights of each dimension can be dynamically adapted to changes in the matching scenario, thereby enhancing the adaptability of the matching method to changes in the number of candidates and the distribution of scores.

[0097] For the convenience of those skilled in the art, Figure 3 An exemplary flowchart of the overall processing flow of a matching method for content generated by a large language model is provided. Figure 3As shown, firstly, the text to be matched and a list of candidate documents are input; then, in step S1 (multi-dimensional feature extraction), vector semantic feature values, keyword retrieval feature values, and word overlap feature values ​​are calculated for the text to be matched and each candidate document fragment, respectively, to obtain a vector score set S_vec, a keyword score set S_bm25, and a word overlap score set S_token; the scores of the three dimensions output in step S1 are used as input for step S2; then, in step S2 (dynamic weight fusion), the weights are adjusted according to the number of candidate document fragments, the standard deviation of the vector score set S_vec, and the mean of the keyword score set S_bm25; where, BM25 low score protection refers to, when a certain When the BM25 score of a candidate document fragment is lower than the preset invalid threshold, the keyword match of the candidate document fragment is deemed invalid. When calculating the mixed score, its keyword retrieval feature value is ignored, and its weight is redistributed to the vector semantic feature value and the word overlap feature value. The mixed score S_final output in step S2 is used as the input for step S3. Then, step S3 (adaptive threshold filtering) is entered, the distribution of the final matching score set S_final is statistically analyzed, and a dynamic threshold is calculated based on the highest score, lowest score, mean, and standard deviation to obtain the filtering results. The filtering results enter the judgment stage to determine whether they pass the threshold. If they pass the threshold, the best matching ranking list is output; if they do not pass the threshold, an empty list is output.

[0098] In another embodiment, determining the basic weights corresponding to the vector semantic feature value and the keyword retrieval feature value based on the quantitative characteristics includes: when the number of candidate document fragments indicated by the quantitative characteristics is less than a first quantitative threshold, determining the basic weight corresponding to the vector semantic feature value as a first weight value and the basic weight corresponding to the keyword retrieval feature value as a second weight value; when the number of candidate document fragments is greater than or equal to the second quantitative threshold, determining the basic weight corresponding to the vector semantic feature value as a third weight value and the basic weight corresponding to the keyword retrieval feature value as a fourth weight value; wherein the second quantitative threshold is greater than the first quantitative threshold, the third weight value is less than the first weight value, and the fourth weight value is greater than the second weight value; when the number of candidate document fragments is greater than or equal to the first quantitative threshold and less than the second quantitative threshold, interpolation calculation is performed between the first weight value and the third weight value to determine the basic weight corresponding to the vector semantic feature value, and interpolation calculation is performed between the second weight value and the fourth weight value to determine the basic weight corresponding to the keyword retrieval feature value.

[0099] In a specific implementation, a first quantity threshold and a second quantity threshold can be preset, with the second quantity threshold being greater than the first quantity threshold. For example, the first quantity threshold can be 5, and the second quantity threshold can be 10.

[0100] When the number of candidate document fragments is less than a first threshold (i.e., fewer than 5), there is limited candidate information available for matching, and semantic understanding should be emphasized. The basic weight corresponding to the vector semantic feature value is determined as the first weight value, and the basic weight corresponding to the keyword retrieval feature value is determined as the second weight value, with the first weight value being greater than the second weight value. For example, the first weight value is set to 0.7, and the second weight value is set to 0.3.

[0101] When the number of candidate document fragments is greater than or equal to the second threshold (i.e., when the number of candidate document fragments reaches or exceeds 10), a stronger ability to distinguish between candidate fragments is needed. In this case, keyword differentiation should be emphasized. The basic weight corresponding to the vector semantic feature value is determined as the third weight value, and the basic weight corresponding to the keyword retrieval feature value is determined as the fourth weight value. The third weight value is less than the first weight value, and the fourth weight value is greater than the second weight value. For example, the third weight value is 0.6, and the fourth weight value is 0.4.

[0102] When the number of candidate document fragments is greater than or equal to the first threshold and less than the second threshold, i.e., when the number of candidate document fragments is between 5 and 10, interpolation can be performed between the first weight value and the third weight value to determine the basic weight corresponding to the vector semantic feature value, and interpolation can be performed between the second weight value and the fourth weight value to determine the basic weight corresponding to the keyword retrieval feature value.

[0103] Among them, linear interpolation can be used for interpolation calculation, so that the basic weights smoothly transition from the first weight value and the second weight value to the third weight value and the fourth weight value as the number of candidate document fragments changes, avoiding abrupt changes in weight values ​​at critical numbers.

[0104] The technical solution of this embodiment switches the weight strategy according to the number of candidate document fragments, so that the matching process focuses on semantic understanding when the number of candidates is small and on keyword differentiation when the number of candidates is large, thereby maintaining a good matching effect under candidate sets of different sizes.

[0105] In another embodiment, the distribution characteristics include standard deviation and mean. Based on the distribution characteristics, the basic weights are adjusted, including: increasing the basic weight of the vector semantic feature value and decreasing the basic weight of the keyword retrieval feature value when the standard deviation corresponding to the vector semantic feature value is greater than the standard deviation corresponding to the keyword retrieval feature value; increasing the basic weight of the keyword retrieval feature value and decreasing the basic weight of the vector semantic feature value when the standard deviation corresponding to the keyword retrieval feature value is greater than the standard deviation corresponding to the vector semantic feature value; increasing the basic weight of the vector semantic feature value and decreasing the basic weight of the keyword retrieval feature value when the mean corresponding to the vector semantic feature value is greater than a preset mean threshold and the mean corresponding to the keyword retrieval feature value is less than or equal to a preset mean threshold; and increasing the basic weight of the keyword retrieval feature value and decreasing the basic weight of the vector semantic feature value when the mean corresponding to the keyword retrieval feature value is greater than a preset mean threshold and the mean corresponding to the vector semantic feature value is less than or equal to a preset mean threshold.

[0106] The standard deviation measures the dispersion of each similarity feature value across the candidate document fragment set. A larger standard deviation indicates a more significant difference between the similarity feature value and different candidate document fragments, and thus a stronger discriminative ability. The mean measures the overall level of each similarity feature value across the candidate document fragment set.

[0107] In practice, if the standard deviation of the vector semantic feature value is greater than the standard deviation of the keyword retrieval feature value, it indicates that the vector semantic feature value has a stronger discriminative ability in the current candidate set. This can increase the basic weight of the vector semantic feature value and decrease the basic weight of the keyword retrieval feature value. Conversely, if the standard deviation of the keyword retrieval feature value is greater than the standard deviation of the vector semantic feature value, the basic weight of the keyword retrieval feature value should be increased and the basic weight of the vector semantic feature value should be decreased.

[0108] In specific implementation, if the mean of the vector semantic feature values ​​is greater than a preset mean threshold, and the mean of the keyword retrieval feature values ​​is less than or equal to the preset mean threshold, it indicates that the overall level of the vector semantic feature values ​​in the current candidate set is high, while the overall level of the keyword retrieval feature values ​​is average. In this case, the base weight of the vector semantic feature values ​​can be increased, and the base weight of the keyword retrieval feature values ​​can be decreased. Conversely, if the mean of the keyword retrieval feature values ​​is greater than the preset mean threshold, and the mean of the vector semantic feature values ​​is less than or equal to the preset mean threshold, the base weight of the keyword retrieval feature values ​​can be increased, and the base weight of the vector semantic feature values ​​can be decreased. For example, the preset mean threshold can be 0.7.

[0109] The adjustment range of the basic weights can be controlled within a certain range, such as within a preset ratio (e.g., ±10%), to avoid excessive weight adjustment that could lead to unstable matching results.

[0110] The technical solution of this embodiment judges the discrimination ability and overall level of each similarity feature value based on the standard deviation and mean of each similarity feature value, and fine-tunes the basic weights accordingly, so that the weight allocation can better fit the actual distribution of the current candidate set, thereby improving the accuracy of matching.

[0111] For the convenience of those skilled in the art, Figure 4 An example is provided: a logic diagram for dynamically adjusting the weights of similarity feature values. For instance... Figure 4As shown, firstly, the candidate set and scores are obtained; the number of candidates N is determined: when N < 5, the basic weight of the vector semantic feature value (Vec) is 0.7 (referred to as vector 0.7 in the figure), and the basic weight of the keyword retrieval feature value (BM25) is 0.3 (referred to as BM25 0.3 in the figure), focusing on trust semantics; when N ≥ 10, the basic weight of the vector semantic feature value is 0.6, and the basic weight of the keyword retrieval feature value is 0.4, focusing on balanced differentiation; when 5 ≤ N < 10, the basic weights are calculated through linear interpolation; then the distribution characteristics are calculated, such as the standard deviation Std and mean Mean of the vector semantic feature value (referred to as vector Std / Mean in the figure) and the standard deviation Std and mean Mean of the keyword retrieval feature value (referred to as BM25 Std / Mean in the figure); then the standard deviation difference is judged (discrimination analysis): when vector Std > BM25 Std, the vector weight is increased by 10%, and the BM25 weight is decreased by 10%; when BM25 When Std > vector Std, BM25 weight +10% and vector weight -10%; when standard deviations are close, maintain the current weights; then perform mean difference judgment (overall quality analysis): when vector Mean > 0.7 and BM25Mean ≤ 0.7, vector weight +5% and BM25 weight -5%; when BM25 Mean > 0.7 and vector Mean ≤ 0.7, BM25 weight +5% and vector weight -5%; when means are close, maintain the current weights; then perform fixed allocation of token weights: Token = 10%, the sum of Vec and BM25 weights is reduced to 90%; then perform weight normalization: ensure Vec + BM25 + Token = 1; then output the policy weight coefficients (Vec_W, BM25_W, ... Token_W); then apply the rules to determine whether to trigger BM25 low score protection: when the candidate BM25 score < 0.1, trigger the protection mechanism, ignore the BM25 weight, and reset Vec=0.9 and Token=0.1; when the candidate BM25 score ≥ 0.1, apply the standard strategy weight and calculate the mixed score S_final; finally output the final score.

[0112] In another embodiment, the statistical features include the highest score, the lowest score, the standard deviation, and the mean. Determining a target threshold for filtering candidate document fragments based on these statistical features includes: lowering the initial threshold for filtering candidate document fragments when the highest score is less than a first extreme threshold; increasing the initial threshold when the lowest score is greater than a second extreme threshold; the second extreme threshold being greater than the first extreme threshold; lowering the initial threshold when the standard deviation is greater than a preset discrete threshold; increasing the initial threshold when the standard deviation is less than a preset clustered threshold; the preset clustered threshold being less than a preset discrete threshold; increasing the initial threshold when the mean is greater than a first mean threshold; lowering the initial threshold when the mean is less than a second mean threshold; the second mean threshold being less than the first mean threshold; and determining the target threshold based on the adjusted initial threshold.

[0113] In practice, an initial threshold can be preset for filtering candidate document fragments, and then the initial threshold can be adjusted according to statistical characteristics to obtain the target threshold.

[0114] If the highest score is lower than the first extreme threshold, it indicates that the overall matching scores of all candidate document fragments in the current candidate set are relatively low, meaning the overall quality is low. In this case, the initial threshold can be lowered to retain candidate fragments of relatively better quality. If the lowest score is higher than the second extreme threshold, it indicates that the overall matching scores of all candidate document fragments are relatively high, meaning the overall quality is high. In this case, the initial threshold can be increased to filter out candidate fragments of even higher quality. The second extreme threshold is greater than the first extreme threshold. For example, the first extreme threshold can be set to 0.4, and the second extreme threshold can be set to 0.6.

[0115] If the standard deviation is greater than the preset discrete threshold, it indicates that the distribution of matching scores is relatively discrete, and the differences between candidate segments are obvious. In this case, the initial threshold can be lowered to avoid mistakenly deleting candidate segments with slightly lower scores but actual relevance. If the standard deviation is less than the preset clustering threshold, it indicates that the distribution of matching scores is relatively concentrated, and the candidate segments are difficult to distinguish. In this case, the initial threshold can be increased to filter out mediocre candidate segments. The preset clustering threshold is less than the preset discrete threshold. For example, the preset discrete threshold can be 0.15, and the preset clustering threshold can be 0.05.

[0116] If the mean is greater than the first mean threshold, it indicates that the overall matching score is at a high level, and the initial threshold can be increased. If the mean is less than the second mean threshold, it indicates that the overall matching score is at a low level, and the initial threshold can be decreased. The second mean threshold is less than the first mean threshold. For example, the first mean threshold can be set to 0.65, and the second mean threshold can be set to 0.45.

[0117] Based on the adjusted initial threshold, a target threshold is determined. Furthermore, the target threshold can be limited to a preset safety range, such as [0.40, 0.70], to prevent it from deviating from a reasonable range due to over-adjustment. After determining the target threshold, candidate document fragments whose matching scores meet the target threshold are labeled as sources cited in the model's response content, and the sorted results can be output in descending order of matching scores for user verification.

[0118] Optionally, the target threshold can also be determined based on quantiles. For example, candidate document fragments at a preset ranking percentage (Top-K percentile) can be selected as citation sources according to the matching scores from high to low, instead of adjusting the threshold item by item based on the above statistical features. This alternative method can also achieve the goal of dynamically filtering candidate document fragments based on the actual distribution of matching scores.

[0119] This embodiment dynamically determines the target threshold based on statistical characteristics such as the highest score, lowest score, standard deviation, and mean of the matching scores. This allows the threshold to adapt to the actual distribution of matching scores in different scenarios, avoiding the problem of fixed thresholds failing when the overall matching scores are too high or too low. This achieves a balance between precision and recall.

[0120] The matching method for content generated by a large language model provided in this application combines multi-dimensional hybrid semantic matching, dynamic weight adjustment, and adaptive threshold filtering. This enables the citation annotation process of content generated by the large language model to simultaneously take into account deep semantic understanding, keyword differentiation, and literal verification. This not only improves the accuracy of citation annotation but also enhances the adaptability of the matching method to different scenarios. At the same time, parallel computing and caching optimization ensure the processing efficiency under a large candidate set, making it highly practical.

[0121] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0122] Based on the same inventive concept, this application also provides a matching device for large language model generated content to implement the matching method for large language model generated content described above. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations of one or more embodiments of the matching device for large language model generated content provided below can be found in the limitations of the matching method for large language model generated content described above, and will not be repeated here.

[0123] In one exemplary embodiment, such as Figure 5 As shown, a matching device for content generated by a large language model is provided, comprising:

[0124] The acquisition module 510 is used to acquire the model response content generated by the large language model, and the candidate document fragment set corresponding to the model response content; the candidate document fragment set includes multiple candidate document fragments.

[0125] The determining module 520 is used to determine the similarity feature values ​​between the model response content and the candidate document fragment in multiple dimensions; the similarity feature values ​​in multiple dimensions include at least vector semantic feature values, keyword retrieval feature values ​​and word overlap feature values;

[0126] The determining module 520 is further configured to obtain the quantity features of the plurality of candidate document fragments and the distribution features of each of the similarity feature values, and, based on the quantity features and the distribution features, determine the weights corresponding to the vector semantic feature value, the keyword retrieval feature value and the word overlap feature value respectively;

[0127] The calculation module 530 is used to process the vector semantic feature value, the keyword retrieval feature value and the word overlap feature value according to the weight to obtain the matching score corresponding to the candidate document fragment, and to extract the statistical features of the matching score corresponding to each candidate document fragment;

[0128] The annotation module 540 is used to determine a target threshold for filtering the candidate document fragments based on the statistical features, and to annotate the candidate document fragments whose matching scores meet the target threshold as the reference sources of the model response content.

[0129] In one embodiment, the determining module 520 is specifically configured to: determine the vector semantic feature value based on the vector similarity between the feature vectors corresponding to the model response content and the candidate document fragment; extract corresponding keywords from the model response content; determine the keyword retrieval feature value based on the word frequency and inverse document frequency of the keywords in the candidate document fragment; determine the lexical overlap feature value based on the literal overlap between the model response content and the candidate document fragment; and determine the similarity feature values ​​of the multiple dimensions based on the vector semantic feature value, the keyword retrieval feature value, and the lexical overlap feature value.

[0130] In one embodiment, the determining module 520 is specifically used to determine the preset weight as the weight corresponding to the word overlap feature value; determine the basic weight corresponding to the vector semantic feature value and the keyword retrieval feature value respectively according to the quantitative feature; and adjust each of the basic weights according to the distribution feature to obtain the weight corresponding to the vector semantic feature value and the keyword retrieval feature value.

[0131] In one embodiment, the determining module 520 is specifically configured to: when the number of candidate document fragments indicated by the quantity feature is less than a first quantity threshold, determine the basic weight corresponding to the vector semantic feature value as a first weight value, and determine the basic weight corresponding to the keyword retrieval feature value as a second weight value; when the number of candidate document fragments is greater than or equal to the second quantity threshold, determine the basic weight corresponding to the vector semantic feature value as a third weight value, and determine the basic weight corresponding to the keyword retrieval feature value as a fourth weight value; wherein the second quantity threshold is greater than the first quantity threshold, the third weight value is less than the first weight value, and the fourth weight value is greater than the second weight value; when the number of candidate document fragments is greater than or equal to the first quantity threshold and less than the second quantity threshold, perform interpolation calculation between the first weight value and the third weight value to determine the basic weight corresponding to the vector semantic feature value, and perform interpolation calculation between the second weight value and the fourth weight value to determine the basic weight corresponding to the keyword retrieval feature value.

[0132] In one embodiment, the distribution features include standard deviation and mean; the determining module 520 is specifically configured to: increase the base weight of the vector semantic feature value and decrease the base weight of the keyword retrieval feature value when the standard deviation corresponding to the vector semantic feature value is greater than the standard deviation corresponding to the keyword retrieval feature value; increase the base weight of the keyword retrieval feature value and decrease the base weight of the vector semantic feature value when the standard deviation corresponding to the keyword retrieval feature value is greater than the standard deviation corresponding to the vector semantic feature value; increase the base weight of the vector semantic feature value and decrease the base weight of the keyword retrieval feature value when the mean corresponding to the vector semantic feature value is greater than a preset mean threshold and the mean corresponding to the keyword retrieval feature value is less than or equal to the preset mean threshold; increase the base weight of the keyword retrieval feature value and decrease the base weight of the vector semantic feature value when the mean corresponding to the keyword retrieval feature value is greater than the preset mean threshold and the mean corresponding to the vector semantic feature value is less than or equal to the preset mean threshold.

[0133] In one embodiment, the statistical features include the highest score, the lowest score, the standard deviation, and the mean; the annotation module 540 is specifically configured to: lower the initial threshold for filtering the candidate document fragments when the highest score is less than a first extreme threshold; raise the initial threshold when the lowest score is greater than a second extreme threshold; the second extreme threshold is greater than the first extreme threshold; lower the initial threshold when the standard deviation is greater than a preset discrete threshold; raise the initial threshold when the standard deviation is less than a preset clustered threshold; the preset clustered threshold is less than the preset discrete threshold; raise the initial threshold when the mean is greater than a first mean threshold; lower the initial threshold when the mean is less than a second mean threshold; the second mean threshold is less than the first mean threshold; and determine the target threshold based on the adjusted initial threshold.

[0134] Each module in the matching device for the content generated by the aforementioned large language model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0135] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a matching method for generating content from a large language model.

[0136] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0137] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0138] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0139] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0140] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0141] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0142] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0143] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A matching method for content generated by a large language model, characterized in that, The method includes: Obtain the model response content generated by the large language model, and the candidate document fragment set corresponding to the model response content; the candidate document fragment set includes multiple candidate document fragments; Determine the similarity feature values ​​between the model response content and the candidate document fragment in multiple dimensions; the similarity feature values ​​in multiple dimensions include at least vector semantic feature values, keyword retrieval feature values, and word overlap feature values; Obtain the quantity features of the multiple candidate document fragments and the distribution features of each of the similarity feature values, and determine the weights corresponding to the vector semantic feature value, the keyword retrieval feature value and the word overlap feature value based on the quantity features and the distribution features; Based on the weights, the vector semantic feature values, the keyword retrieval feature values, and the word overlap feature values ​​are processed to obtain the matching scores corresponding to the candidate document segments, and statistical features of the matching scores corresponding to each candidate document segment are extracted; Based on the statistical features, a target threshold is determined for filtering the candidate document fragments, and candidate document fragments whose matching scores meet the target threshold are labeled as reference sources of the model response content.

2. The method according to claim 1, characterized in that, Determining the similarity feature values ​​between the model response content and the candidate document fragment in multiple dimensions includes: The semantic feature value of the vector is determined based on the vector similarity between the feature vectors corresponding to the model response content and the candidate document fragments. Extract corresponding keywords from the model response content, and determine the keyword retrieval feature value based on the word frequency and inverse document frequency of the keywords in the candidate document fragment; The word overlap feature value is determined based on the degree of literal overlap between the model response content and the candidate document fragment; Based on the vector semantic feature value, the keyword retrieval feature value, and the word overlap feature value, the similarity feature values ​​of the multiple dimensions are determined.

3. The method according to claim 1, characterized in that, The step of determining the weights corresponding to the vector semantic feature value, the keyword retrieval feature value, and the word overlap feature value based on the quantitative features and the distribution features includes: The preset weights are determined as the weights corresponding to the word overlap feature values; Based on the quantitative characteristics, determine the basic weights corresponding to the vector semantic feature values ​​and the keyword retrieval feature values, respectively; Based on the distribution characteristics, the basic weights are adjusted to obtain the weights corresponding to the vector semantic feature values ​​and the keyword retrieval feature values.

4. The method according to claim 3, characterized in that, The step of determining the basic weights corresponding to the vector semantic feature value and the keyword retrieval feature value based on the quantitative features includes: If the number of candidate document fragments indicated by the quantity feature is less than the first quantity threshold, the basic weight corresponding to the vector semantic feature value is determined as the first weight value, and the basic weight corresponding to the keyword retrieval feature value is determined as the second weight value. When the number of candidate document fragments is greater than or equal to the second quantity threshold, the basic weight corresponding to the vector semantic feature value is determined as the third weight value, and the basic weight corresponding to the keyword retrieval feature value is determined as the fourth weight value; wherein, the second quantity threshold is greater than the first quantity threshold, the third weight value is less than the first weight value, and the fourth weight value is greater than the second weight value; When the number of candidate document fragments is greater than or equal to the first quantity threshold and less than the second quantity threshold, interpolation is performed between the first weight value and the third weight value to determine the basic weight corresponding to the vector semantic feature value, and interpolation is performed between the second weight value and the fourth weight value to determine the basic weight corresponding to the keyword retrieval feature value.

5. The method according to claim 3, characterized in that, The distribution characteristics include standard deviation and mean; adjusting the basic weights according to the distribution characteristics includes: If the standard deviation corresponding to the vector semantic feature value is greater than the standard deviation corresponding to the keyword retrieval feature value, increase the base weight of the vector semantic feature value and decrease the base weight of the keyword retrieval feature value. If the standard deviation corresponding to the keyword retrieval feature value is greater than the standard deviation corresponding to the vector semantic feature value, increase the base weight of the keyword retrieval feature value and decrease the base weight of the vector semantic feature value. If the mean value of the vector semantic feature value is greater than a preset mean threshold, and the mean value of the keyword retrieval feature value is less than or equal to the preset mean threshold, increase the base weight of the vector semantic feature value and decrease the base weight of the keyword retrieval feature value. If the mean value of the keyword retrieval feature value is greater than the preset mean threshold, and the mean value of the vector semantic feature value is less than or equal to the preset mean threshold, the basic weight of the keyword retrieval feature value is increased, and the basic weight of the vector semantic feature value is decreased.

6. The method according to claim 1, characterized in that, The statistical features include the highest score, lowest score, standard deviation, and mean; determining the target threshold for filtering the candidate document fragments based on the statistical features includes: If the highest score is less than the first extreme threshold, the initial threshold for filtering the candidate document fragments is lowered; if the lowest score is greater than the second extreme threshold, the initial threshold is raised; the second extreme threshold is greater than the first extreme threshold. If the standard deviation is greater than a preset discrete threshold, the initial threshold is decreased; if the standard deviation is less than a preset clustered threshold, the initial threshold is increased; the preset clustered threshold is less than the preset discrete threshold. If the mean is greater than a first mean threshold, the initial threshold is increased; if the mean is less than a second mean threshold, the initial threshold is decreased; the second mean threshold is less than the first mean threshold. The target threshold is determined based on the adjusted initial threshold.

7. A matching device for content generated by a large language model, characterized in that, The device includes: The acquisition module is used to acquire the model response content generated by the large language model, and the candidate document fragment set corresponding to the model response content; the candidate document fragment set includes multiple candidate document fragments. The determination module is used to determine the similarity feature values ​​between the model response content and the candidate document fragment in multiple dimensions; the similarity feature values ​​in multiple dimensions include at least vector semantic feature values, keyword retrieval feature values, and word overlap feature values; The determining module is further configured to obtain the quantity features of the plurality of candidate document fragments and the distribution features of each of the similarity feature values, and, based on the quantity features and the distribution features, determine the weights corresponding to the vector semantic feature value, the keyword retrieval feature value and the word overlap feature value, respectively. The calculation module is used to process the vector semantic feature value, the keyword retrieval feature value and the word overlap feature value according to the weight to obtain the matching score corresponding to the candidate document fragment, and to extract the statistical features of the matching score corresponding to each candidate document fragment; The annotation module is used to determine a target threshold for filtering the candidate document fragments based on the statistical features, and to annotate the candidate document fragments whose matching scores meet the target threshold as the reference sources of the model response content.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images